Hacker News Reader

Comment by sgarland | Hacker News Reader

I have operated a BDR Postgres cluster spread across five global regions with hundreds of TB of data, with multiple reader nodes attached to each primary, each on instances with node-local NVMe drives; the entire cluster saw over 1 million QPS. I will grant you that there are much larger examples of scale, but they typically exist on DBs designed for that, like Spanner, Vitess, etc.

I am not asking for hand-holding, I’m asking for one single publicly-available example of this being done, because I cannot fathom it being necessary or beneficial, and you have not provided concrete proof.

View on HN

Usually first principles architectures and systems engineering doesn't require you to be validated by big players. You can always spin up and do your own measurements.

Yet since you asked, assuming good faith, here is what you are looking for.

Following is an excerpt from https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...

< The indexes are eventually consistent; whenever we write a cell we also update the index entries, but it does not happen in the same transaction. The cells and the index entries typically do not belong to the same shard. So if we were to offer consistent indexes, we would need to introduce 2PC in the writes, which would incur significant overhead. With eventually consistent indexes we avoid the overhead, but Schemaless users may see stale data in the indexes. Most of the time the lag is well below 20ms between cell changes and the corresponding index changes >

Disclaimer: I have never worked at Uber. This is all armchair speculation, and I could of course be wrong.

IMO, if you read the later post on Docstore [0], you can see that they spent an enormous amount of time building back everything that they gave up with Schemaless - the enforcement of a schema (the irony...), transactions, serializability, etc.

They needed to create secondary indices as tables because they were storing everything in JSON blobs, which is difficult to directly index now, let alone in the 2010s. You can of course argue that they needed to use JSON blobs because they kept needing schema changes, and online DDL back then was difficult. I would counter that if you keep needing schema changes, you haven't done a good job modeling.

Everywhere I have worked that has DBs they claim are too large, I have been able to prove (typically by cloning one and remodeling it) that the reason they're suffering from performance issues is because they've made a series of decisions which were fighting against their RDBMS: UUIDv4 PKs, strings for low-cardinality data, excessive secondary indices, poor query plans from the ORM, massive JSON blobs that's duplicating already-present data, etc. At my last job, I showed a 30% speedup purely from changing to a PK that made sense for the table's usage.

0: https://www.uber.com/us/en/blog/schemaless-sql-database/

> Everywhere I have worked that has DBs they claim are too large, I have been able to prove (typically by cloning one and remodeling it) that the reason they're suffering from performance issues is because they've made a series of decisions which were fighting against their RDBMS

Some are just physics. Say if you want SSD write saturation and multiple parallell write transactions, the traditional RDBMS model itself may not suffice. Its not just data size but concurrent write transactions, lock contention, etc. Additionally, think resource constrained devices like mobile phones. Or an aggressive cost reduction mentality where each cpu cycle shouldn't be unnecessarily wasted without feeling guilty. If you come from a systems engineering background like that, you wouldn't be much impressed with many traditional databases at all. I am generally not much interested in SQL databases anyway.

> They needed to create secondary indices as tables because they were storing everything in JSON blobs, which is difficult to directly index

Our database Dip stores raw binary as well as json as first class citizens. Can't speak for Docstore, but as far as I see, it is just an evolved form of its earlier self with different tradeoff according to their workload. Note the lack of mention of the removal of secondary indexes as separate data.

> I would counter that if you keep needing schema changes, you haven't done a good job modeling.

Disagree. Schema stabilizes - yes. But doesn't mean they don't go through cycles of change occasionally.

Postgres can absolutely saturate an NVMe drive on its own.

> Additionally, think resource constrained devices like mobile phones.

SQLite will saturate the phone’s I/O before you saturate SQLite.

> Or an aggressive cost reduction mentality where each cpu cycle shouldn't be unnecessarily wasted without feeling guilty.

> Our database Dip stores raw binary as well as json as first class citizens.

These two statements are in conflict. If you want to minimize CPU cycles, parsing JSON is a terrible idea, and you should be pushing everything into bitfields, operating on it with bitwise manipulations.

> If you come from a systems engineering background like that, you wouldn't be much impressed with many traditional databases at all. I am generally not much interested in SQL databases anyway.

I would love to see a Jepsen analysis of your DB. Then a benchmark of it vs. Postgres, or MySQL, or SQLite.

I love how you first confused application managed indexes with B+trees in code. Then you moved to how you could not see how "this could be faster", asserting there is an additional indirection layer. When pointed out that the indirection also exists in database managed indexes, you wanted examples of usage at scale with a blog post. You also mentioned you have operated 1M QPS, yet you couldn't fathom while being oblivious to lock contention. You were persistent in demanding one public link.

When the Uber blog link was provided with exact scenario in the excerpt, you pivoted to how their new version has schema enforcement. You claimed you were able to prove all inefficiencies you encountered were due to bad data modeling or bad decisions which went against RDBMS.

When pointed out that the newer blog post doesn't mention removing the secondary indexes which was the whole issue, you digressed to the grand finale of absurdly claiming Sqlite will saturate phone's I/O before one saturates Sqlite, which is so false that it is not even funny. And finally demanding our Dip's benchmarks (which will come in due time, if? we ever release it to the public).

All of this when more than enough hand-holding was made to help you understand. Even when the underlying physics was easy to understand. Enough pivoting.

Let our conversation stand and be of help to anyone curious enough to learn the tradeoffs discussed. Best wishes.

Reply on news.ycombinator.com