Hacker News Reader

Comment by happymellon | Hacker News Reader

What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes.

If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.

View on HN

Yeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.

Exactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly

Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases

> What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

This get large rather fast when processing time series from multiple sensors.

But I agree with you otherwise, and it's not like you're storing satellite products in a database.

Now I'm curious what NSA is using :).

https://accumulo.apache.org/

See (History): https://en.wikipedia.org/wiki/Apache_Accumulo

Thanks, I wasn't aware of it.

You could just check their Github? :)

https://github.com/nationalsecurityagency

Reply on news.ycombinator.com