How MySpace Tested Their Live Site with 1 Million Concurrent Users

I've read this story a few times and I always have a few problems with it.

1. The total population of New Zealand is 4 million. Expecting 1 million concurrent users on the service is very optimistic. Also Myspace wasn't that popular in New Zealand, these days facebook dominates but previous Bebo was common among young people.

2. Testing from EC2 instances against Akamai will not simulate load from NZers. The December 2009 the closest EC2 servers to NZ were in the US so traffic would have been pushing completely different servers.

3. A transfer rate of 16Gb/s almost certainly exceeds any spare capacity in the New Zealand Internet. Also there are less than a dozen Akamai clusters in NZ, distributed unevenly and probably most don't have more than 1GB/s connections.

So while it's a nice story, if they were really testing just for NZ they spent a lot of time and money testing the wrong thing. Perhaps it could be a story about how inflated marketing expectations cause cost to be incurred by engineering?

I used to do this at Napster, albeit with far less computing resources. I would record traffic to a machine and then play it back at 10-20x realtime to see where my performance limits were using a few machines. Of course, it turned out that most of the limits we ran into were hardware related (the max packets per second Cisco switches advertised was... optimistic).

Of course, I also just tested in production with real users. I'd redirect more or less users to a particular server to stress it and see where it broke. Since we controlled the protocol and the client, I could hide any problems pretty easily. Users didn't seem to care if a for a couple minutes the service was a little wonky. As Twitter has demonstrated, if people love your service, they will forgive some minor disruptions from time to time.

At our peak, we were at 2.3 million simultaneous users, but we had nowhere near the bandwidth requirements of MySpace. It wasn't until recently with the rise of continuous deployment did I think that testing in production was an ok thing to do. I did it because I had to.