Hacker News Reader

Comment by sva_ | Hacker News Reader

30GB a month isn't too terrible? I guess it depends - for example my provider includes 10TB egress per month, so 30gb would be negligible if it's static content. I'm not sure what others do, but I often click on the links that the agents pull up as sources so the websites do get more discoverable.

What really sucks is e.g. Gemini that pulls images without being able to go to the source (unless it changed recently.)

Edit: I didn't read "logs"

View on HN

The issue is that a dynamic site that can serve the traffic that generates 10MB of logs a week might be very performant on the smallest and cheapest of virtual servers, but one that generated a GB daily might find itself very slow and sometimes crashing because of load depending on spikes, unless they opt for a server with more resources, for more money.

Any answer to that along the lines of "do something more complex" or "optimize your site for load" to be able to deal with the rampant bad behavior of multi-billion dollar companies is unpalatable to most.

In other way I must use LLM to fix issue created by LLM in the first place

If you think your only tool is a hammer, everything indeed starts to look like a nail.

This is one of those things that might be best solved through policy and law. There probably will be some changes to policy and law to help, eventually. That stuff usually takes a while, and that's always been a problem for those suffering in the interim.

1GB of logs, not content

Average log size is about 300 bytes. 1GB of logs per day is approximately 40 requests per second average.

Not every request is meant to be served below 500ms and this is server which supposed to be called couple of times a minute not milliseconds

We started with caching but this created a new kind of problems of its own (veterans know what circus of man eating from outside out clowns this create) but client was mad manual refresh is causing "freezes" in the system.

So we ban LLM and problem solved until client starts asking why ChatGPT, Claude, whatever he prefer can't access data and release him/her from pain of using brain

Reply on news.ycombinator.com