Slashdot effect
en.wikipedia.org
en.wikipedia.org
They were limited by tools and resources. Now you'll just see some async "API endpoint error -32768 timeout" and 8,000 spinning pinwheels on the page instead of it simply hanging. Because now you need to allocate 650MB RAM so you can output Hello World in some buggy "stack" with "memory safety".
The smart ones were using Java 25 years ago and what needed some 80lb Solaris box could run off a mobile phone now.
There is no hope for large segments of the industry. We actively choose to make bad decisions all the time, and then people rationalize them.
What's going on that's taking down these sites, really, when its not a quota limit?
The kinds of sites that go down when receiving unexpected traffic are usually built in such a way where they're making multiple DB requests to render each page. Or they have a dozen poorly configured perf-expensive WordPress plugins running. Or likely all of the above
At that time, it was quite common to start a new thread or even fork a new process per request/connection. Apache was prone to this:
> https://www.digitalocean.com/community/tutorials/apache-vs-n...
"mpm_prefork: This processing module spawns processes with a single thread each to handle requests. Each child can handle a single connection at a time. [...]
mpm_worker: This module spawns processes that can each manage multiple threads. Each of these threads can handle a single connection. [...]"
(this website also mentions the mpm_event MPM).
So, if you have ten thousands of processes or threads on some web server, this can easily overstrain the resources of the web server. Additionally keep in mind that at that time servers were a lot less beefy than today.
Most people don't do performance tuning of their blogs because the average traffic is miniscule. This means that any configuration issues rear their heads when load happens. For example, having your max connections to the web server be far more than the database can support. Perhaps the machine could handle the load if properly configured, but no one did so because there was no need.
Also, when slashdotting was common, networks weren’t always the most robust. It wasn’t uncommon in the late 90’s for companies to have a single a T1 line for access to the internet. So, unless you had a good, well peered network provider, that was another potential bottleneck.
We worked with what we had. Thankfully, everything is more robust now.
There are actually a lot of little things to tune if you can't just auto-scale out of a bottleneck.
The other reason is, not every web server, web application, database, etc is optimized for large number of connections. Sometimes they take up too much CPU and memory. Sometimes each page request triggers a connection to a database, so the database connection limit is hit. Sometimes the queries are inefficient and slow the database to a crawl. Sometimes the sheer number of packets causes so many interrupts that it sucks CPU away from the application/database. And sometimes the network connection of the webserver was just slow.
Most people hit by the /. effect were not just a simple web server serving static content. Often it was a php website with a mysql database, on a server less powerful than a 2015 smartphone.
You're grossly overstating the capacity of a server to meet peak load demands, specially when hit with unexpected load spikes. The slashdot effect was a phenomenon observed two decades ago, in a time when the bulk of the web was served as Apache running CGI scripts. The rate of requests any random server could process was around ~300requests per second, and some of the links being hugged to death were literally served from a box under some dude's desk.
Or because once you get the max requests per sec out of a machine, you can often get a second machine to double rps for much cheaper than a machine that can do double the rps.
In the 20-30 year old time frame, easy to obtain servers are one to two cores, often one core per socket. SMP scalability was not a given; you could certainly run on SMP and get benefits, but depending on your load you might not get anywhere close to 2x the throughput.
A single core webserver serving static files isn't going to be much faster than 1 ms per file, so you're around 1k rps. And that's assuming you files are all cached. If you do dynamic pages, they're more likely to start closer to 10 ms, and can easily be much longer, so 100 rps is a good target.
But, if your pages are sizable, you can easily run out of bandwidth and then everything gets slow and you can hit too much concurrency and things get worse. A T3 was a very good connection back then, and it was 45 mbps. 1000 rps of 50k images is going to fill up your T3.
Load balancing, done right, is very scalable for typical http loads where client sends little and server sends lots. Direct Server Return means the load balancer only sees the traffic from the client, which significantly reduces the effort for the load balancer. Of course, you can't do a lot of fancy things with DSR, but that means your balancers aren't overloaded with fancy things. If your DSR load balancer fails, you can fail over to DNS round robin.
What “is” is this? Already in 2005, I was serving images at >3k qps on a single core of my server, without any tuning at all. Now we're 20 years later, with much faster TCP stacks and much faster cores.
If you were lucky you could reduce load by caching the URLs taking up the most resources and generate a webpage copy in the filesystem, then add some URL rewrite logic to Apache to skip going through your application logic and bypass the DB.
Then you discovered there is a limit of open file descriptors in Linux. After updating this and also shutting down all log files you ran out of things to change pretty quick and started pricing a beefier server. For next time.
- lack of scalability (you can’t dynamically spin up a bare metal server if you don’t already have one in your rack)
- slower memory models (CGI, and forking meant it took longer for the OS kernel to allocate system resources)
- improvements to TCP and fork performances in the Linux kernel over the years. Such as changes to how TCP negotiation happens
- improvements to HTTP transit such as improved compression performance, HTTP/3, etc.
- CDNs weren’t common place, and were super expensive.
- caching was in its infancy. Tools like varnish (I think it was called) were complicated and setting up memcached, varnish, etc all required additional hardware, that added to hosting costs.
- HTTP servers weren’t as mature. Plus weren’t included in the language runtime (eg you’d run httpd that would fork an instance of PHP rather than run the HTTP server as part of the Python, et al)
- serving static content of NFSs mounted SAN was slow (the POSIX overhead, plus NFS protocol, plus network roundtrip, plus disk access times)
- mechanical drives were slow
And so on and so forth.
The biggest difference I think was a combination of affordable or free CDNs, reducing pressure on the backend, and the easy of scaling (VM, Docker, etc) meaning you don’t need to find a happy middle ground between costs and high availability.
it'd be fine if it's one site on a dedicated machine, but these shared webhosts would routinely cram 500-1000 hosting accounts onto something like a single-core, HT-less 2.4GHz Pentium 4 with two gigs of RAM, running mpm_prefork on Apache 2.0, with single-node MySQL 5.x (maybe even 4.x!) on the same box... which was not terribly efficient as others observed
you carry about 20x the compute power of that machine in your pocket, even a Raspberry Pi 4 could triple the workload
Near to the end of (real) Slashdot's life, project 10k was happening. People were rewriting even the Linux process management to make servers perform close to their theoretic capacities.
Muscle memory still takes me there several times a day...
We couldn't afford the new Sun Ultra 2's that would have given us a spacious 2GB of RAM, so we ended up coding a front-end to shard requests across the 2 web servers and used a Sparcstation 5 to run that front-end.
Eventually we rearchitected the entire site to move the static HTML websites into an Oracle database. (by then we had upgraded to a Sun Ultra Enterprise 3000, which could handle up to 6GB of memory, more than we could ever use)
Perhaps the best benchmark for the maximum potential traffic of the community at that time comes from a Slashdot post by Rob Malda (CmdrTaco) on September 14, 2001 (https://news.slashdot.org/story/01/09/13/154222/handling-the...), detailing the site's traffic following the 9/11 attacks. This load on Slashdot itself represents the full attention of the userbase and was higher than the site normally experienced:
Normal Load: 18–20 dynamic page views per second.
Peak Load: 60–70 page views per second.
Daily Volume: 3 million page views (up from a daily average of 1.4 million).
To handle this higher load, Slashdot itself had to disable dynamic content and serve static HTML to survive this load. Their database could not handle the query volume.
A typical "Slashdotting" pushed 5 to 10 Mbps of traffic. Since many sites were hosted on T1 lines (1.5 Mbps), the pipe was instantly clogged, resulting in 100% packet loss for legitimate users.
I remember when Slashdot broke because they used an Unsigned Mediumint counter for a primary key and finally overflowed it with 16 million comments over the life of the site on Nov 6, 2006.
It was a different time back then when we still measured our installed RAM in megabytes.
Back then it was Wordpress/PHP with no caching.
But: you're right.
Once upon a time named Maxim made a web page that pointed out that a Pentium III should be able to handle 10K simultaneous requests, then said Apache was shit, and that he had made his own webserver, and because he was Russian he filled the webpage with animated hammers and sickles.
This was how NGINX was created.
I definitely missed out :(
What was it like?
It had its own in-jokes like: This is finally "the year of the Linux Desktop" and Jonathan "CowboyNeal" Pater, the site's moderator who often posted polls and commented.
In the late 2000s, most migrated their Slashdot reflex to Digg - and then eventually the YC-backed Reddit when there was a disastrous rollout of Digg v4 (in 2010)
Reddit then became less of a technology-focused site as it gained popularity, and HN became the defacto "tech news" aggregator with a well rounded comment section that resembled the early days of Slashdot, Digg, and Reddit.
It was also the year of the Linux on the desktop.
Please don't out me as a ghost parent
They have their infancy and grow into something popular (relatively speaking) then your older members either get bored or jaded, often both. The culture of the platform starts to change while more people leave disliking the change than new people discover the platform. Resulting in the platform steadily shrinking in popularity.
Every social network and forum has experienced the same phenomenon.
Once CmdrTaco sold the site to Dice, they tried to turn it into a business intelligence/job board, which turned off a lot of long-time users (they also tried a site redesign that was functionally useless and had to abandon it after a lot of complaining). Then Dice got tired of it and BizX bought it to add to their trophy case, but haven't done anything meaningful with it. The site has been on autopilot since the acquisition. The 'editors' are faceless interchangeable usernames who just load up a bunch of stories from the queue and let them auto-post throughout the day and almost never show up in the comment threads. The 'community' is barely there, most discussions get less than 50 comments. They turned off anonymous coward posting (unless you're logged in). The user poll hasn't been updated since April. And so on. It also didn't help that Hacker News came along and ate most of their lunch.
Basically, slashdot was hollowed out and the shell is all that's left. But the new owner can put 'COO of Slashdot Media' on his LinkedIn page, which is probably the most important thing.