What used to happen was I'd download, say, an Ubuntu image, use it, and then say "... I might need this again" and leave it there. Then a week goes by, I've forgotten its there. Sorting by Date Added its now piled under 10 other items. The empire of dirt grows...
The trick? A Trash that automatically empties items older than 30 days. MacOS has this, at least. (Note, this is a rolling empty; not an "empty everything every month" kind of deal). Now I just delete those "I might need this..." files. I know I can grab it from the Trash if I need it again, which I've done a few times. Most of the time, I didn't need it, and so it gets thrown out as it should.
The effect is two fold. Obviously there's less junk in ~/Downloads in general. But that has the knock-on effect that it is now less daunting to go through and sweep up ~/Downloads.
Empirically, my ~/Downloads, ~/old.Downloads, etc. used to be filled with easily 400 files and folders each. Now? My current ~/Downloads is at 76 items after two years. And that's currently at my peak; I'm about to go through and clear out about 20 of those things. Which is easy to do, compared to trying to face a mound of 400 files with names like "TEST.txt"...