Great point about Twitter never explaining why it didn't just use one of the many awesome open source message queues already in existence.
http://www.rabbitmq.com/ - first version 8th February 2007, but the first version not to have "alpha" or "beta" status was 1.5.1 released 21st January 2009
http://memcachedb.org/memcacheq/ - version 0.1.1 released 26th November 2008
http://www.ejabberd.im/ - not really a message queue
http://xph.us/software/beanstalkd/ - first public release 11th December 2007, hit 1.0 28th May 2008
http://activemq.apache.org/ - not sure when it was first released but the mailing list goes back to December 2005
The first public release of Starling (Twitter's first custom message queue) was 10th January 2008. Presumably they had it running internally for a while before they released it.
From this, we can see that when they built their own pretty much the only realistic open source option was ActiveMQ, which can hardly be described as a light-weight solution (not to mention it still doesn't have a stellar reputation under high loads). When the alternatives aren't rock solid yet, rolling your own (where at least you understand all of the code and how it works) seems like a perfectly practical alternative.
Secondly, after running a custom message queue for well over a year they know EXACTLY what they need from one, so writing their own still makes sense.
Thirdly, if they're going to start moving other core bits of Twitter infrastructure to Scala it makes sense to try it out with an important piece of the puzzle that they thoroughly understand first.
And finally, Twitter's core competency is delivering messages. As such, it's really not so extreme to use their own software to do that - they work at a high enough scale that they need to be experts in whatever solution they are using.
As Douglas Crockford once said, "The good thing about reinventing the wheel is that you can get a round one.". The fact that Twitter's reliability over the past 6-12 months has been a huge improvement, despite the enormous growth the service has seen (it's mentioned in the mainstream media all the time) would suggest that their decision to roll their own message queue paid off.
I think hi-fi devs make lots of stupid decisions in the name of performance. In the few cases where their actual outcomes match up to their posturing, it's because their code is obsessively cobbled around one specific use case they've been working on since 1989.
Have you ever read an order management system, or looked at Tibco Rendezvous on the wire?
Most of the hi-fi companies adopting MQ are built around straight AMQ, and bare-metal performance was out the window long before they bolted their crappy WebSphere app onto it. What these companies are looking for is predictability, not performance, and their problem sets are much simpler and most stable than Twitter's.
But it sounds like we're once again enmeshed in the never-ending process of second-guessing Twitter. Let me offer a pertinent quote, from the Hugo-winning novel The Vor Game:
[You did] a right thing. Perhaps not the best of all possible right things. Three days from now you may think of a cleverer tactic, but you were the man on the ground at the time. I try not to second-guess my field commanders.
Twitter solved their problem. It is unlikely that they did so as elegantly as possible, and it's quite possible that their logic at the time will be of no use to the rest of us at all, because it was based on incomplete information about a rapidly-changing software universe.
We ended up using memcacheq but it's hardly what you would call full featured, so we'll have to do prioritization, etc in our code.