Still, it's much more complex than a simple webapp (so the audience willing to do it will be smaller). If there are ~12 million events, we need 10 000 people downloading 1200 pages...
(On the leaderboard, the number of "to do" items goes up instead of down: what does it mean?)
Over time you're going to start seeing duplicates (ie. things you've already archived/spidered), so the number will go down. But while you're still in the 'discovery' stage it's going to go up and up.
But here's what the OP said: "All of Upcoming's events and venues use autoincremented ids, making it dead simple to generate a list of URLs to scrape."
So if you go from 1 to {current_id} you get them all, and you have the list at the start.
I'm not sure about the details for this project, but sometimes it's not easy to get a dump of a list of the items you want all upfront. So the warrior process catalogs new items it encounters and updates the list on the tracker.
A webapp would simply need to receive a list of ids and download one at a time (with a comfortable delay between each call so as to not trigger Yahoo's security).