Hacker News Reader

3. Hive is Facebook's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 million map-reduce jobs per day.

So 4 PB per day, but only 300 PB total?

View on HN

Was wondering the same thing. My guess is that some also gets removed each day as well, but it seems unlikely.

Think of it like monitoring data. You may collect one-second data on 500 counters per system over 1000 systems, but then you will do a weekly or monthly roll-up where you dump some of the granularity to save space, and after a year you have aggregates that are basically daily trend lines. The more you collect smaller percentage you actually keep.

Reply on news.ycombinator.com