Hacker News Reader

Comment by jjirsa | Hacker News Reader

The reason people don’t start with a relational DB has nothing to do with the data model or query language - people choose databases like Cassandra because scaling to 500 Postgres or MySQL instances holding a combined petabyte of data is horrific, but it’s dirt simple in things like Cassandra

The query language is a side effect of the underlying storage engine - you don’t choose it because you want a key value store, you choose it because you want horizontal scalability and cross wan HA

View on HN

You make it sound like many people are using petabyte scale databases. This is not even close to true. ESPECIALLY when you're just starting out. Most new projects will start with almost no data at all and the vast majority of projects will never ever get to the petabyte scales.

Many people ARE using petabyte scale databases.

They don't start out that way, but they will be if they're successful, and if they're not they probably shouldn't be running their own databases anyway (go RDS it up or something).

Unless the amount of data you are collecting is increasing at a vastly unsustainable rate, you will be making megabucks and be able to afford a team of engineers to undertake a major redesign for version 2. But picking technologies based on scale at the onset is going to make getting to that point all the more unlikely.

What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes.

If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.

> What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

This get large rather fast when processing time series from multiple sensors.

But I agree with you otherwise, and it's not like you're storing satellite products in a database.

Now I'm curious what NSA is using :).

https://accumulo.apache.org/

See (History): https://en.wikipedia.org/wiki/Apache_Accumulo

Thanks, I wasn't aware of it.

You could just check their Github? :)

https://github.com/nationalsecurityagency

Yeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.

Exactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly

Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases

Postgres (as an example) has no upper limit for database size, and the upper limit for a single table was 64TB a while ago, perhaps higher now. And you can buy stock hardware with 6 TB RAM and 76TB SSD e.g. from Dell.

If you max out that single server, you could easily hire one database engineer to care about horizontal scaling.

Show me a single person who runs 64TB on a single table on a single node, and I'll show you someone who will cry because of a prod failure sooner than later.

It doesn't help when the nbr of queries and updates increases a lot. For many systems having TB of data also means having thousands or millions of users.

You also probably want backups and possibly slaves that are updated inside transactions to avoid data loss. These things can lower your throughput even more.

The vast majority of hipster NoSQL developers I have debated with are using a single machine.

Those people are objectively wrong.

You try and explain that to them, but whatever database they are using "is fast". Not that I have ever had a problem with relational databases being slow. Certain queries, yes, but a bit of rewriting them and proper indexing usually solves that.

Reply on news.ycombinator.com