itcmcgrath

It does not use CRDTs as they cannot be used to handle all the update requirements of this system.

It uses Paxos as the consensus algorithm, along with internal systems like the TrueTime API to enable us synchronously replicate across multiple data centers.

Hard to explain externally, but it relates to RAM size allocation + throughput in the underlying distributed database system.

Thanks for the feedback, this request is definitely been a frequent one. :)

We've also tried to be transparent and document our current limits here: https://firebase.google.com/docs/firestore/quotas

Nothing we can share at this time. Really looking forward to the feedback we get now we're in beta as this will feed into our GA plans.

Quorum based (Paxos)

Sure, it's strongly consistent on the server-side part (strict serializability), whereas for the mobile/web clients we make sure we move you through consistency snapshots with the real-time sync functionality.

Great question! I'm the PM for both Cloud Datastore and Cloud Firestore.

The biggest difference is the integration with Firebase, so you have access to Mobile (iOS/Android) and Web SDKs along with a native offline mode. This comes along with the real-time synchronization feature that makes serverless app development a breeze.

Cloud Datastore is great for large scale server-side development where you manage your own connection to your app, such as running your own website on App Engine or via Compute/Container Engine.

Except, given that both models ended up #1, and #2 respectively, doesn't it imply that both models can work and other factors are more important in success? Following that, doesn't it imply that shareholders should/will care more about those other factors?

FYI, Java 8 is now in beta for App Engine Standard: https://cloud.google.com/appengine/docs/standard/java/quicks...

It is refreshing to see an article from a distributed database vendor that gives a reasonably good description of the trade-offs they make and why - without the all the nonsense hyperbole claiming they're the best for everything without any trade-offs*

* I've reviewed ~400 databases over the last month and it's surprising (?) how many of them are all the best of every use case and are the [fastest|first|only|best]

It's the verb you use when you want to receive a 400 with "BAD Request - Invalid verb"

One thing I like about Lovefield is the ease of choosing a storage layer: lf.schema.DataStoreType.FIREBASE

You can use it with in-memory (no persistence, .MEMORY), IndexDB (local persistence, .INDEX_DB), or Firebase Realtime Database (cloud persistence, .FIREBASE).

Our younger cat has learnt to use mirrors to watch us around the house, so I'm pretty confident it has worked out what it does: https://goo.gl/photos/mZkiFcBUNKSUFwKx5

As an example, if I get food out in the kitchen while it is watching in the mirror, it comes running to check it out. Non food things - just keeps on watching.

The counter arguments include Google, Apple, Microsoft, etc.

I think the big vs small comparison is flawed. I've seen some atrocious code produced by small/medium sized outfits. My fondest memory including auditing code from a 3 person outfit who's code quite literally setup an RPC on the server that executed any string it was sent, verbatim, against a database that handled money.

I thank you for the compliment, but I am a lowly PM rather than an engineer, that doesn't find Cloud Databases [1] any more esoteric than the Cloud in general. :)

You'll probably find there are a lot of distributed database geeks at Google, me included, so we all get attracted to exciting posts on new distributed systems.

[1]: http://db-engines.com/en/blog_post/68

To add extra color, for about $3M/month @ list prices of Cloud Datastore [1], you can, in a Multi-Region active-active synchronous replication configuration, run a workload with the following profile:

Reads: >1.1M entities/second Write: >380K entities/second Delete: >190K/second Storage: 100TB

And that's if you don't use any of the nearly free optimizations like Projection queries & keys-only queries, which any large scale customer does.

That's not pre-provisioned usage, it's actual pay-as-you-go usage - so if you have no traffic, you have no costs (except for what's already stored). It's been that way for 8 years too.

[1]: https://cloud.google.com/products/calculator/#id=e21b61d5-4a...

(PM for Cloud Datastore - if you'll looking at 1M+ QPS workloads feel free to message me)

Take a look at Cloud Datastore's pricing, which is probably a leader in this area (I work on it).

Awesome!

We have a series of locations that determine replication topology, but you cannot do à la carte region selection at this time. Under the hood we run on at least 3 independent private super fast network fabrics between regions, so we have a pretty high bar infrastructure wise for deployments. We bet on having reliable high-grade deployments and Google's world wide network to get requests to users faster rather than stretching out arbitrary deployments. Seems to work so far for customers like Snap & Niantic Labs (Pokemon GO), but YMMV.

Thanks for the feedback!

I happy to hear you gave it a shot and I've heard several times about it being used successfully from Lambda (hopefully people can look at GCP's Cloud Functions now, but I'm biased). Sorry to hear it didn't work out.

Search is something we're looking at, and we have a high quality bar on reliability and scalability aspects as a service that handles 10's of trillions of requests per month.

I'll also pass on your feedback about the JavaScript SDK to its team.

It's a lot more than a thin layer, but like most systems at Google is a specialized layer that utilizes built on top of more fundamental building blocks that are designed and proven to do a particular job really well.

In this case Megastore provides the underly multi-region/datacenter K-V replication services.

All the database features like secondary & composite indexes, query language, multi-tenancy support, PAYG model, etc, etc, are built in the Cloud Datastore layer.

PM for Cloud Datastore here.

We also have multi-document transactions, self-joins, active-active multi-region replication by default, secondary indexes, composite indexes, a SQL-like query language, PAYG model with no capacity planning, etc, so I think it's a bit disingenuous to say it's "similar to datastore, but with full-blown database features".

Otherwise, congrats on your launch!

It doesn't explicitly call out the E-3 in the leaked document, however IANAL and couldn't tell if any of the clauses implicitly affect it. [I'm on an E-3]

There is some, when users battle gyms they are battling with other users Pokemon which would be from anywhere in the world. When the Pokemon on the Gym loses it needs to be removed in a transaction with an update on the gym. This takes the easiest form of sharding off the table, so it's not as simple as you make it seem.

How about this - Pokemon search trend for the last 5 years: https://www.google.com/trends/explore?q=pokemon

If this doesn't clearly show how crazy this growth was compared to what anyone was expecting, I'm not sure we're operating in the same plane.

No arguments there. There's a balance that needs to be struck between technical detail and marketing appeal. Not everyone is going to agree on where that balance is, less so when you're trying to share sensitive information without giving too much away.

"For instance on launch day in America and Asia I would have expected insane numbers (many hundreds of million)."

Given no app had ever done this in history over the first few weeks, yet alone on launch day with no marketing, I doubt it.

They ran a closed beta.