Hacker News Reader

Comment by bobsomers | Hacker News Reader

When I worked at iFixit ~12-13 years ago most of the site was already aggressively cached whenever possible, and I’m sure it’s only gotten better since.

They are no strangers to bursty traffic. We would get insane bursts of traffic many, many times greater than typical wherever a teardown of a new iDevice was published shortly after release.

If Kyle and the devops folks there are noticing it, it’s definitely disruptive behavior.

View on HN

I believe you. But at the same time it's insane to me we have these standards that aren't upheld on one end or the other. I'm fulling willing to acknowledge that Anthropic is the problem here. But at some level, and at some point, don't you lose the ability to say the site is down because it's too popular? I'm totally open to the arguments that the traffic overloaded the Cloudflare limits or whatever. But the frequency of outages caused by not just traffic at Anthropic levels, but at random HN post levels are enough to take down websites. I guess this is just my cognitive dissonance between being told on HN "you don't need to scale! You're not Facebook!" and "OMG our went down because we have a few thousand concurrent visitors!"

I don't think the site itself went down, so much as the scraping itself was excessive and beyond this, the presumed use of said scraping is against the TOS and copyrights of the site itself.

On a smaller site, it can absolutely be brutal. I worked on a site where literally 2/3 of requests were bots and a lot of those were dynamically constructed pages that are/were harder to cache. In the end, switching to a supplemental search database helped significantly, but it doesn't mean that an aggressive bot is an okay thing.

I don't see where iFixit said this caused an outage.

Reply on news.ycombinator.com