Hacker News Reader

Comment by bethekidyouwant | Hacker News Reader

Does it suck if you offer a product and someone’s looking for it and the agent can’t look on Amazon but they can look on your small side maybe that’s good

View on HN

Not it the site can't stay up because it's constantly being hammered by bots.

Dude, I had a 1Gb of nginx logs daily then I blocked them and it dropped to 10mb weakly.

Legit what one of our clients problem was. Normal web traffic wasn’t an issue and the logs were just a few megs. Yeah there was a problem where they were not rotating but once meta started crawling them 24/7 ignoring the robots.txt it kept filling their vps up. An annoying problem to fix once a server is completely full and you can’t even ssh in.

There are also some pathological bad use cases out there that are caused by the combination of trying to scrape all the content, a randomized param, and the scrapers trying to be anonymous. The one I've encountered is pbpBB.

phpBB will assign you a randomized session if if you visit it anonymously (without logging in), and will serve the page to you with that sid param added to all links. Using that link will load a page without the sid param appended to links, because the system assumes you have a session I'd saved in a cookie since it saw you have a sid, even I'd you don't have/didn't send a cookie with it. Following any of those links starts the chain over if you aren't sending a cookie along with a sid, which the AI scrapers never do so they can't easily be blocked by that, and a new unique sid is generated, and all the links have the new sid appended.

Every page with a unique set of params and values needs to be investigated, because it can be an entirely different page, so AI scrapers will scrap forever, always finding more and more URLs to harvest, even though they're all duplicates of what they already have. It's easy to have multiple scrapers stuck going through a very small bulletin board continuously.

Maybe they've gotten smart enough to deal with this situation in the last few months, or maybe they put in a specific workaround. I ended up just throwing the site behind cloudflare because spending time trying to outsmart AI agents in a battle of escalation doesn't seem like an effective use of my time.

git hosts get the same effect - ddosers request a diff of every file between every combination of first and last commits.

30GB a month isn't too terrible? I guess it depends - for example my provider includes 10TB egress per month, so 30gb would be negligible if it's static content. I'm not sure what others do, but I often click on the links that the agents pull up as sources so the websites do get more discoverable.

What really sucks is e.g. Gemini that pulls images without being able to go to the source (unless it changed recently.)

Edit: I didn't read "logs"

The issue is that a dynamic site that can serve the traffic that generates 10MB of logs a week might be very performant on the smallest and cheapest of virtual servers, but one that generated a GB daily might find itself very slow and sometimes crashing because of load depending on spikes, unless they opt for a server with more resources, for more money.

Any answer to that along the lines of "do something more complex" or "optimize your site for load" to be able to deal with the rampant bad behavior of multi-billion dollar companies is unpalatable to most.

In other way I must use LLM to fix issue created by LLM in the first place

If you think your only tool is a hammer, everything indeed starts to look like a nail.

This is one of those things that might be best solved through policy and law. There probably will be some changes to policy and law to help, eventually. That stuff usually takes a while, and that's always been a problem for those suffering in the interim.

1GB of logs, not content

Average log size is about 300 bytes. 1GB of logs per day is approximately 40 requests per second average.

Not every request is meant to be served below 500ms and this is server which supposed to be called couple of times a minute not milliseconds

We started with caching but this created a new kind of problems of its own (veterans know what circus of man eating from outside out clowns this create) but client was mad manual refresh is causing "freezes" in the system.

So we ban LLM and problem solved until client starts asking why ChatGPT, Claude, whatever he prefer can't access data and release him/her from pain of using brain

Reply on news.ycombinator.com