Scale Something: How Draw Something Rode Its Rocket Ship Of Growth | Hacker News Reader

Scale Something: How Draw Something Rode Its Rocket Ship Of Growth

gamasutra.com

81 pointsangersock22 comments

Wow, I had forgotten what it's like to work in a world where I have to manage this drudgery myself. This is exactly why I build on App Engine.

Before all the haters reach for the downvote button: I have apps that routinely go from 2-3 requests/sec to bursts of 500+ requests/sec for hours at a stretch and the only difference is that my bill is a few tens of dollars higher than usual. And there's no reason it couldn't handle 100X that load as long as I keep paying the bill.

no reason it couldn't handle 100X that load as long as I keep paying the bill

You might be surprised what kind of limits you hit and what your invoice looks like when you actually hit 50k reqs/sec for any length of time.

Cloud platforms have bottlenecks, too. The vendors usually handle that by making it comically expensive for you to exceed a certain threshold ("financial back pressure") and/or by throttling you outright.

You might be surprised

I really would be surprised. I'm pretty familiar with GAE's limits and pricing structure. Granted, 50k requests/sec is severe, but they would have to actually go out and write special billing code just to charge me more if I hit that rate. I don't see that happening.

https://developers.google.com/appengine/docs/quotas#Safety_Q...

Google lifts these limits when you ask. It has come up numerous times on the mailing list. I think it fair to take their explanation in that document literally.

You have a point, but App Engine has the following issues: - lock-in (migrating your apps to another platform will be extremely hard) - Less control over your backend

IMO these two issues are enough to keep most of the app developers away from App Engine.

This is foolishness. Your biggest risk is that your product will fail in the marketplace, not that for mysterious-and-unnamed-reasons you will suddenly have to migrate off the platform. As far as "control" goes - there is nothing about appengine that prevents you from running specialized application code in other parts of the cloud. But it's a pretty exotic need.

I know a thing or two about App Engine too, and I disagree with you.

"Things got really bad around 1 a.m. one night, which is when we realized the main issue -- our cloud data store was throwing errors on 90 percent of our requests. Shortly after, we received an email from our vendor telling us we were "too hot" and causing issues, so they would have to start rate limiting us."

Was this Amazon's S3?

This is a great article and I think what they've done is great. But, it reminds me why I don't think I could work for a startup (unless it's my own).

Working these kinds of hours (unless I own a large share in the company) just sounds insane to me.

It wouldn't be indefinitely. And the rush of making something that was being appreciated worldwide combined with the knowledge that this is your one shot, might just get me up at night. It becomes a story you can tell your grandkids.

I suppose. But it's not my one shot. It's the company owner's. I would much rather tell my grandkids about what it takes to be a successful business owner and show them that it is possible to make it on your own.

I've been a part of a few multi-million dollar companies. My code made it possible. But, if I wasn't there, someone else would have made it happen.

I used to have to pull these kind of random hours from time to time whilst working for a global investment bank with hundreds of thousands of employees. I sure as heck didn't get a large share in the company!

But the challenge of solving the problems and finding creative ways to overcome an issue seem like crack to me at that time of the night. As long as it is not a consistent (every night) thing, then I don't mind it at all.

Amazing Insights!

I really wonder how many people are capable of making services scale in such a short time frame and such massive pressure. I hope I can learn it one day

What if working these hours guarantees a great return on your shares of the company?

I am curious to hear whether the engineers working at this knew at the time that this game was going to be so valuable for the company.

Honestly at some point it becomes less about the money and more about the fact that you are pissing of, uh, 30 million people simultaneously. It kind of puts the fire under your ass to fix things.

I wonder about the cost of all this, does anyone know if they were using the enterprise versions of Couchbase?

The lesson I learned from reading that article is don't name your game "Draw My Thing". Hilarious.

There is nothing like the rush of solving Good Problems (tm) in the middle of the night with a lit firecracker down your pants.

However I doubt all these problems are as unpredictable as the article makes them out to be. There's a whole field of science devoted solely to modeling large systems of interconnected stochastic processes and identifying bottlenecks.

This science is so old a mandatory semester-long class was devoted to it in 2nd year of my comp sci undergrad.

I do wonder how many developers who aren't working on government-backed systems actually use that science though ...

There is a big difference between theory and practice. :)

While it is possible to model the large system, usually as a startup you don't have the time to make the model, nor the data to feed the model.

Part of the problem is that you don't know what the inputs to the system will be and which inputs will grow exponentially.

There's a whole field of science devoted solely to modeling large systems of interconnected stochastic processes and identifying bottlenecks.

These models need to be fed known inputs and parameters such as component limits.

Neither can be specified in advance for your average, public-facing, constantly changing webapp.

Exactly. And when you're pumping out half a dozen titles, each with wildly different performance charactistics and no idea which will bomb and which will be a huge hit, it's not really worthwhile to be overly thorough with any single title until the shit hits the fan...

However, I do think the case could be made that trying to roll your own infrastructure in this kind of scenario is perhaps not the best use of your time, when solutions to scalabilty problems already exist and are so well tested.

(Edit: what I mean by that is, if you use something like App Engine, you really don't need to be concerned with scalability all that much -- as long as you're working sensibly and within the constraints of the platform)

Reply on news.ycombinator.com