Hacker News Reader

Comment by nerdponx | Hacker News Reader

Data in the 1-15 GB range is more common than you'd think. Spark is overkill and comparatively a nightmare to configure. You can fit 15 GB in memory nowadays on a single core, but you probably want things to move a little quicker.

View on HN

I work at a University HPC centre and our new 72 core Icelake nodes have 512gb of RAM, and even our older 40 core Cascade Lake nodes had 256gb as standard, so I always scratch my head at who these libraries serve. That's pretty standard hardware for a HPC cluster, which is where people doing this stuff should be working anyway.

I wasn't thinking about HPC at all. I more had in mind "single-node" use cases like a workstation PC.

But even in a small cluster type of use case, I think the main users are scientists or other practitioners who, for whatever reason, don't have the support of a platform team or data engineering team that can keep a Spark (or other) cluster fed and cared for.

Off the top of my head, I can't remember what size nodes we use. I think it's m5x large AWS nodes with 16 cores and 128 GB RAM. Depending on the size of the data I use anywhere between 20 to 80 of these nodes. Sometimes even higher. That's why I've never had a need for these libraries.

I have seen european companies (claiming) doing big data on 16gb, 4-cpu, 15 node clusters and claiming that their spark jobd were not failing due to 90s hardware. I will send them your configuration. :)

You're definitely correct about Spark being a nightmare to configure. Luckily we've got data platform teams who take care of that. Analysts like I still do configure parameters depending on the scenario. You might be correct about data in the 1-15 GB being more common. I'm an analyst so most of the time, I've to aggregate data before I do any analysis. But data scientists in my team typically work on smaller datasets and use Sagemaker instances.

Reply on news.ycombinator.com