Hacker News Reader

Comment by SEJeff | Hacker News Reader

Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory.

If it fits into 1 machine or a few, it isn't "big data".

To be honest 6 TB of data isn't that much. I'd say big data starts at around one petabyte because one petabyte requires 3 servers with 45 [1] 8TB hdds. At this point your data is so large it's impossible to store it on a single physical machine.

"big data" usually doesn't require transactions though or at least no transactions across servers so you could probably get away with sharding postgresql via citus.

[1] http://www.45drives.com/products/storage/

I agree with this response. Those aren't very big either though. The Dell DSS7000 series servers[1] support 90 x 8TB drives (We run a ceph cluster on 4 of the DSS7500 storage nodes).

[1] http://www.storagereview.com/dell_announces_availability_of_...

Except that it's just stupid to do that, you don't want to store that much data in one host, it goes against any sane architecture design, good luck when you're mb / cpu dies. Splitting data on smaller / cheap hosts is the way to go for ops / scaling / backups.

Maybe. I recently updated some of my company's internal training material, and discovered there still isn't a single (good) definition of "Big Data". Most definitions talk evaluating data on four dimensions: volume, variety, velocity and veracity. That's still a vague, but it's more helpful than looking at volume alone.

Besides, if we picked machine size as _the_ definition, it would change every year. And that feels like a bad way to talk about big data.

(BTW, I'm not a huge fan of the term in general: I've seen a lot of companies sold a bill of good because some sales guy convinced them they needed a Big Data solution instead of something like Postgres)

I've also struggled with the definition of Big Data. I typically see "size" as being the line of demarcation people throw around when thinking about Big Data, but I think it's it's actually size + latency that's the real definition.

If I have a 12TB dataset, and I need to be able to find a single record in that dataset, and return the results to a process in less than one second, I don't believe that needs to be considered "big data". The dataset is "large", but the latency / access pattern surrounding that data is low enough that you can solve that with a handful of hard drives and your data sorted on disk.

If I have a 12TB dataset and I need < 10ms latency on random lookups, that is now probably big data. To guarantee those sort of lookup times for random IO, I either need very expensive hardware, or a software based solution to distribute my data to multiple machines.

If I have 10GB of census data, and I need to run an analysis that shows what are the 100 most similar cities (non approximation) to Los Angeles in terms of demographic make up, and I need an answer in less than 1 second, that is also probably big data. The analysis is easily parallelized, but if done serially on a single machine is a "slow" process.

I realize this is a moderately unconventional way of approaching Big Data, but it's how I've started to structure some of my thinking and explanations around why it's not just size, but size + latency (which is inherently a question of what you're trying to do with the data).

I've heard this 6TB number thrown around a bit recently, but no one can actually provide me with links to hardware I can buy - has anyone successfully setup a machine with this much ram who can post links to what they used?

http://www3.lenovo.com/us/en/data-center/servers/mission-cri...

Don't know about hardware you can buy, but AWS will rent you a machine with 3TB Ram by the hour :) https://aws.amazon.com/ec2/instance-types/

http://www.thinkmate.com/systems/servers/rax#max-ram:min-6-t...

And the postgres maximum table size is 32T.

You can partition your table if it got anywhere near that large (with the shiny native partitioning introduced in PostgreSQL 10).

You really don't want to have the whole DB be much above 30TB. You'd either want to shard or use some alternative solution.

Reply on news.ycombinator.com