Hacker News Reader

Why create the markov text server side? If the bots are running javascript just have their client generate it.

1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup.

2. You need to send the data for the Markov chain generator to the client, along with the code. This is probably bigger than the response you'd be sending anyway. (And good luck getting a bot to cache JavaScript)

3. As the author said, each request uses microseconds of CPU and just over a megabyte of RAM. This isn't taxing for anyone.

> 1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup.

Anyone crawling at scale would try to limit the per-request memory and CPU bounds, no? Surely you'd try to minimize resource contention at least a little bit?

Then why generate text at all? Just run a script that enters an infinite loop. But the bots would have to protect against this or the scrapers wouldn't make it very far on the larger internet, would they? Spending a few microseconds on the server costs essentially nothing, and guarantees the scraper's most precious resource (bandwidth) is wasted.

Reply on news.ycombinator.com