Weeding is never a waste of energy. The more you do, the less you will have to do. Weed now and you will have less to do in the spring.
But your top job should be cutting down and throwing out any mildewed plants. Do not compost plants which are mildewed as the spores could spread. Everyone had problems with mildew this year. But one thing to consider is, if you want to prevent mildew, the plants in your garden need enough space around them to get good air circulation. This helps to prevent mildew. Phlox are particularly sensitive to mildew and we put up with it for the late blooming beauty of the plant. But once they have bloomed be ruthless! Cut down the stocks at the first sign of mildew, bag them and put them out in your green trash.