The Death of Upcoming.org
waxy.org
waxy.org
Internet Properties: Don't be a dick. If you're closing up shop, bundle the site up (public portion, not personal info), and ship it off to the Internet Archive.
It's really easy to set up and I would encourage anyone remotely interested to give it a try. More info & download: http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri...
Regardless, thanks to the people who built Upcoming and sorry that Yahoo ruined it.
This is a once in a lifetime opportunity for your project :)
At minimum, get email addresses.
A way to bypass security is to distribute scraping over many, many clients.
For example, one could write a browser extension and have many people use it at once; this would have the added benefit of being able to scrape information generated by JavaScript (just do the scraping after the page is completely rendered).
If needed, I can help write such an extension for Chrome (I had written one for del.icio.us some time ago that did exactly that, and another one for IMDb).
(There would be a public server that would serve the ids of events yet to scrape, and that would receive the data that has been scraped).
But for this strategy to work, you need a big community of people willing to help (people who will download the extension and let it run).
Also, 11 days is a very short time...
Still, it's much more complex than a simple webapp (so the audience willing to do it will be smaller). If there are ~12 million events, we need 10 000 people downloading 1200 pages...
(On the leaderboard, the number of "to do" items goes up instead of down: what does it mean?)
Over time you're going to start seeing duplicates (ie. things you've already archived/spidered), so the number will go down. But while you're still in the 'discovery' stage it's going to go up and up.
But here's what the OP said: "All of Upcoming's events and venues use autoincremented ids, making it dead simple to generate a list of URLs to scrape."
So if you go from 1 to {current_id} you get them all, and you have the list at the start.
I'm not sure about the details for this project, but sometimes it's not easy to get a dump of a list of the items you want all upfront. So the warrior process catalogs new items it encounters and updates the list on the tracker.
A webapp would simply need to receive a list of ids and download one at a time (with a comfortable delay between each call so as to not trigger Yahoo's security).
However, I've thought about this space a bit (the idea-space, not the solution-space), and I've always come to the conclusion it's too fragmented. Too many things to too many different players.
But if the field is wide open again, better to invite someone into the opportunity than dwell on the past.
Neither Eventful nor Zvents plugs into a social network to tell you stuff your friends are going to, or thinking of going to. They are just sites that list events.
Facebook's events app is 100% social - it tells you what your friends are doing, and you can follow artists and venues. There are comments and such on events. It is also good for non-public, informal events, something Upcoming didn't do very well. But Facebook is useless for searching for events in your area, or for following a particular genre of music or type of event.
Upcoming was sort of in the middle of all of this. Which was part of the problem. It was hard to know what to focus on. I'm not sure the story is entirely that Yahoo was solely to blame for fucking it up. We also made some incorrect decisions. But unquestionably, Yahoo didn't give us the resources to succeed, and prioritized things that had nothing to do with success.
Your elucidation of Upcoming's differences helps explain why I probably quit using it, though. While I can see the value in having your social network plugged into a facet of (if not your entire) calendar, I personally choose not to participate in third party social networks. If Upcoming required a Facebook (or just about any other social network) login, I would have stopped using it. And yes, I realize I'm in the minority :)
It was a loss of time for me, maybe, but I was well compensated. The real losers were all the community members who built it into something.
It's a bit homely and needs some love but I'd like to upgrade and open-source it. Anyone interested? It's called kpasa and is at http://lax.kpasa.co/ .
It only works in Oregon, USA, but let me know if it it's interesting to you guys and I'll add more places.
Also check out the list I made at http://dontstartaneventaggregator.1fiesta.com/
Uh.