[0] https://news.ycombinator.com/item?id=13817557
One of you will now point out that it would be better not to have to do this (or, as someone gently put it downthread, "maybe it is time they actually considered fixing HN"). You're right!
I mean what exactly is the load that causes this?
If you are executing some algorithm for each request - stop. Just run it every minute to rebuild a cache. HN discourages rapid fire replies anyway.
How hard is it to simply build an html fragment from a tree and cache it? You just see what nodes need other nodes to be appended.
OK, I can imagine MAYBE the most intensive thing is the votes and re-ordering of replies.
First of all I hope you are using this algorithm: https://medium.com/hacking-and-gonzo/how-reddit-ranking-algo...
Secondly, seriously once again you can batch the upvotes by thread and then process them during the recalculation phase. You only need to recalculate weight totals for the threads where there were upvotes - and this can be parallelized.
It can't be any harder than not coming off blithely condescending.
That's because the solution is straightforward, the people who built it are smart, had years to do it.
If I can build the basic scalable submission and comment system in one 8 hour day I am sure they can fix it in less than that.
Then you need to iterate of course and get the bugs out. But come on. I am incredulous that HN is buckling under a load of 20 stories with on average 100 comments each. Seriously?
...
[unless this is deliberate parody in which case, you got me]
And btw I said 8 hours, not a weekend.
Look up Poe's Law.
The problem isn't straightforward because the software does a ton of things to try to keep the site good. We do a lot of manual moderation too, but anything we can write code for, we of course do.
For one thing, other forums would use a database, whereas (from what I understand, but don't necessarily comprehend) Hacker News uses Lisp objects in memory, and flat files. We know how simple it might be to refactor in, say, PHP or Node but who knows what kind of eldritch Lisp weirdness this thing has going on under the hood?
You have two option with the web. Either you put your logic in the backend or in the frontend.
This addition is obliviously to reduce the load on the server. So either you add some logic on the frontend to make the user experience good or you do like they do right now and have a bad user experience.
The JS in the page should attach an event to the link to append the new page at the bottom. Better: it should do it automatically when the scroll position is close enough to the end.
Bonus: the script should add a search box to search even in the content not yet in the page.
I googled if it's possible to intercept CTRL-F and handle it programmatically. It seems it is [1] but I don't know if I would do it: the user could be surprised to find a search box in a non standard position with a non standard behaviour. It reminds me of scrollbar hijacking.
[1] http://stackoverflow.com/questions/9159322/overriding-the-br...
What I would do is add a dialog the first time the user press CTRL+F. Asks if the user wants the default or the override. Save this in a cookie or in the database.
A simple (non-JS) "Search these comments" form would suffice, but I've never seen it implemented.
My understanding is it wasn't because HN depends on S3 but because everyone was on HN at the same time talking about it!
Edit: my google/HN Search fu is failing to find a reference, but they can't easily horizontally scale HN because it is running on a single arc process. Comments/Submissions are stored in text files, and the "More" links are actually stored in memory in closures. When the closures get garbage collected you get the "Unknown or expired link." error.
It was the volume of comments in a single thread.
HN may do that, but I've only seen it paginate by the number of subthreads. The bulk of new traffic in a thread is likely to be comments to existing threads. This would also make thread folding faster, because there would be a hard limit to the depth of any thread on a single page.