Hacker News Reader

Comment by ojhughes | Hacker News Reader

I think it’s quite rare that a dataset is too large to be processed on a single volume. EBS volumes max out at 64Tb

If you're using EBS for shuffle/source data you've already behind.

If you can afford it you should be using the host NVME drives or FSX and relying on Spark to handle outages. The difference in performance will be orders of magnitude different.

And in this case you won't have the ability to store 64TB. The average max is 2TB.

It's really the only reason to use Spark in the first place, because you're doing non-local level processing - when I was throwing 100TB at the problem it made more sense than most of the data science tasks I see it used for.

This magnitude of data fascinates me. Can you elaborate on how that much data came to be and what kind of processing was needed for all that data? And if not, maybe point to some podcast, blog that goes into the nitty gritty of those types of real big data challenges.

Click/traffic data for a top 100 website. We weren't doing a ton, but basic recommendation processing, search improvement, pattern matching in user behavior, etc

We normally still would only need to process say, the last 10 days of user data to get decent recommendations, but occasionally it would make sense for processes running over the entire dataset.

Also this isn't that large when you consider binary artifacts (say, healthcare imaging) being stored in a database, which pretty sure that's what a lot of electronic healthcare record systems do.

A random company I bumped into has a 40TB OLTP database to this effect.

Financial market data (especially FX data) can have thousands of ticks per second. It's not unheard of for data engineers in that space to sometimes handle 1TB per day.

It's not just disk capacity though. Disk bandwidth, CPU and memory capacity/bandwidth etc can also become bottlenecks on a single machine.

Reply on news.ycombinator.com