Hacker News Reader

Comment by richardlblair | Hacker News Reader

> Not everything was smooth sailing at launch! When issues emerged around the game’s stability, Niantic and Google engineers braved each problem in sequence, working quickly to create and deploy solutions. Google CRE worked hand-in-hand with Niantic to review every part of their architecture, tapping the expertise of core Google Cloud engineers and product managers — all against a backdrop of millions of new players pouring into the game.

IMO This is the most valuable thing in this article. It essentially says what others are pointing out. You can't just press a button and have scale. It's not that easy. You have to tackle many layers. Considering they 50x'd their worst case scenario it would have only taken a few bad queries to fuck shit up.

View on HN

At 50x designed max load bad queries would be one of the easier problems to solve. We design systems all the time that have hard scaling limits, and it's only a problem when you operate the system past those limits. e.g. I wrote a service with ACID assumptions from the underlying database but now the largest ACID db box I can buy isn't big enough. Oops. There's a bunch of possible ways around that but they usually involve a nontrivial amount of engineering effort.

I wouldn't expect any service to survive that much unplanned load. Maybe they could've estimated better, but how likely was it for the game to go viral? Worth sinking lots of dev time into and delaying launch? That's a hard question to answer without the benefit of hindsight.

Honestly I was surprised they were able to scale as elastically as they did and we now know the reason why: they used Datastore instead of something like Postgres with sharding. That decision alone is most of the reason they made it. By contrast, my team doesn't use managed NoSQL services for our projects because they're simply too expensive to deploy before seeing considerable growth.

Datastore seems really cheap to me, am I missing something?

They also have a system that's almost trivial to shard in any way they want, which would help even with postgres all around. Since there's no real interaction between users (only user-environment really), they can split databases by location, user groups, or some hashes as they want. Also the data they deal with is extremely cachable - as long as you play on one device.

There is some, when users battle gyms they are battling with other users Pokemon which would be from anywhere in the world. When the Pokemon on the Gym loses it needs to be removed in a transaction with an update on the gym. This takes the easiest form of sharding off the table, so it's not as simple as you make it seem.

It doesn't really need to be a transaction since it is okay if it is eventually consistent (e.g. they have a cleanup job if a transaction is lost), that would be fine.

This: "but now the largest ACID db box I can buy isn't big enough."

The hard part about scaling is knowing when to invest in scale. The binomial choice is to build and throw away, or build so as to be scalable. So hard to predict.

> Considering they 50x'd their worst case scenario

Nitpick: they 10x'd their worst case scenario

> You can't just press a button and have scale

Actually you can... if the system was designed for this purpose.

Technically true, but it doesn't make sense from a business perspective. Anticipating and accounting for every scaling issue costs a lot of money. Most businesses would decide that satisfying 90% of scenarios is sufficient.

Business is irrelevant once you're sure that the service will go into production AND will have some amount of users.

It's known that some things will break from time to time (e.g. servers, hard drives, networking). If you've designed the application poorly so that a single minor failure can snowball to the entire system, it's guaranteed that your site will fall apart periodically, costing you lots of money each time.

At this point, the best technical AND business converge to "pick an option that can sustain [minor] breakages". Either you've done it like that from the start, or you're in the process of rewriting/redesigning your site.

> Business is irrelevant once...

Not really. In the case described they had accounted for some maximum number of users and instead got 50x that number. It is reasonable to design a system which handles just some planned number as long as there is a strategy in place to manage a situation beyond those limits. In this case the strategy was to apply engineering effort. That seems like a valid risk management plan to me.

All you need to do is follow the waterfall development process properly! Think hard, establish a perfect design in enough meetings, write a flawless spec, then implement it correctly and the button will work first time!

Rrrrrright?

Experience suggests that things you didn't anticipate will rain on that plan.

Queues, buffers, locks all have ways of interacting at high volume that defy prediction. And if you think your system has no queues or locks then you just haven't looked deeply enough.

Reply on news.ycombinator.com