Hacker News Reader

Comment by winddude | Hacker News Reader

Well I'm winning than. I've written so much useless software, and projects, and starts of projects.

It's all fun and joy until you come back to it 1 or 1.5 years later, and resources are now throttled because they're so over used because of the explosion in AI. The fun seems to go away when you come back to an old project, and things have changed.

View on HN

> It's all fun and joy until you come back to it 1 or 1.5 years later, and resources are now throttled because they're so over used because of the explosion in AI.

What does this mean?

No yesterday, I just tried to spin up an old project using common crawl, which is hosted on AWS open data, seems like it's gotten quite popular since I lasted worked on it, and they're experiencing throttling from over usage, maybe even ddos (or just a really bad developer).

But seems to be a pretty common thing with stuff related to data and crawling. Everyone is cracking down on crawlers, adding paywalls, removing previously free APIs, so coming back to old projects in that domain seems to be impossible.

What's the lesson to be learned from that experience?

Finish something and exploit (I mean market) the shit out of it. Or find a lot more money somewhere and build a lot more from scratch.

What kinds of resources?

commoncrawl at the moment.

But also used to have some stuff using the old reddit (through pushshift) and twitter apis. Plus some old classified crawlers, where sites have taken a more proactive approach.

Reply on news.ycombinator.com