The average irc server probably handles more than twitter - certainly email software does.
There's a ton of software that needs to scale, and be able to handle a large volume of data.
Yes, but there's vastly more that doesn't, you need to stop thinking that most software runs on the Internet and has tons of users, it doesn't. The vast majority of software is written for businesses and runs either on an intranet or is available on the internet to the users of that business.
For every public app you know about that needs to scale there are a hundred you don't know about that don't need to scale. Most programmers work on systems that will never need to scale, EVER. It is a tiny tiny minority of programmers that actually work on systems that become publicly popular on the Internet and actually face scaling issues.
Secondly, we're talking about programmers here, not the software, most email systems use existing mail software written by that tiny tiny minority of programmers who write such systems. The vast majority of programmers will NEVER write a mail system, a chat system, or anything that handles large volumes of data or users. Most of them of doing one thing, biz apps for in house projects at big and small companies exposing relational data to in house users.
How many IM networks are there? Now how many programmers are there? I'll say it again, most software does not need to scale. Because a lot of software does need to scale does not make that statement in any way false because a shit load more doesn't need to scale.
That doesn't mean a thing in terms of scalability. One user can max out a fast machine with OLAP cubes or other DSS. Or any sort of simulation or modeling. Think in terms of datasets, not in terms of users (all many users are is a large dataset with a requirement for a quick turnaround of each computation).
scalable - How well a hardware or software system can adapt to increased demands.
scalable - the ability of a product or network to accommodate growth
The ability to scale hardware and software to support larger or smaller volumes of data and more or less users.
the ability of a computer system to shrink or grow as requirements change
http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi...
(Many thanks to Eran Hammer-Lahav for writing this so that I don't have to.)
The problem is that where once it would have been written in c as a server, it's now written with big databases and 'popular' languages - so it's harder to scale.
But a message server in Ruby with Eventmachine wouldn't have been that hard, would it?
There are better ways: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protoc...
For a good introduction check out http://google-ukdev.blogspot.com/2008/09/rabbitmq-tech-talk-...