Hacker News Reader

My bet is that it is for technical reasons. Whenever you see a reply link (among other things) the server has to persist a continuation for it. Getting all of those pages crawled must generate a massive number of continuations to persist.

I know that this is one of the reasons that vote-links look the way they do.

If that's not the reason, then my #2 guess is that getting the whole site crawled is moving too many comments into the cache. If I remember correctly, whenever an uncached comment gets accessed, it gets cached (basically adding it to a hash), and thus added to the in-memory ram. That gradually raises the memory requirements for running the app, until pg does a restart, which resets the cache as well.

And the regular restarts invalidate every single continuation-based URL

Probably to stop linkspam?

It's unfortunate because I often use site: to find old threads on ycombinator. especially since news.ycombinator.com doesn't have its own search facility, external search engines are the old way to comb through old posts here.

http://www.searchyc.com/ still exists, but it's up in the air if they will continue.

Not unless pg replies to this with "Please stop".

OK, thanks.

That doesn't make sense. Why not just add rel=nofollow to all links?

maybe to see what firms ignore robot.txt files?