Hacker News Reader

Comment by threeseed | Hacker News Reader

So I deal with hundreds of Data Scientists and none of this is true for us.

Hadoop vendors are simply getting killed by the cloud. AWS EMR, Azure ML, Google Dataproc. And many people are simply swapping HDFS for S3 or similar object stores. Never seen anyone switching to very expensive, more HPC style NVME setups. That seems strange given the data sizes we are working with.

And 95% of what Data Scientists do is ETL, Data Curation, Feature Engineering etc and which Spark is still the overwhelming dominant player. And Spark uses Hadoop filesystem API underneath which is trivial to replace hdfs:// with s3://

View on HN

It really depends on what the workload is and if performance or scale are priorities. Our team and the products we build come from the HPC side of the fence and we've been engaged with all sorts of large businesses that know their performance requirements (think 100s of GB/s read bandwidth), and it leads them down the path of either HPC filesystems or newer NVMe-oriented filesystems. Does 'AI' or 'data science' always mean 'high performance'? No, but it definitely can.

Everyone's data volume is different as well...working a proposal for several petabytes of NVMe storage right now. It's not everyone, but use cases are out there for large volumes of high performance storage.

Fair point on the ability to swap the hdfs interface trivially, I hope it's that simple everywhere. Another issue we run into is that these companies have invested heavily in these Hadoop clusters, and would prefer to continue to get some use out of the mountains of HDDs that are captive in these environments. So tiering/HSM functionality is another facet of the issue that these environments will anchor for quite some time.

The problem you have with Weko and those newer filesystems is that the tooling hasn't caught up. They are trying to get their APIs upstreamed into TensorFlow. But what about Spark, Pandas, Arrow, etc? It's not enough to have just the training part of the pipeline - you need to be the filesyste for the whole pipeline. That's what we provide with HopsFS.

Wow... how do you achieve petabytes of NVMe? like, what is the biggest drive that you can get today?

And then you just plug a ton of them into what kind of "motherboard" to interconnect them? I guess Infiniband based bus? or I am totally off-base here?

Any more details are appreciated, if only to satisfy my curiosity :)

Dell sells poweredges with 3.84 TB nvme drives, some of the 2U servers can pack 24 IIRC. If you do RAID10, 1PB is “only” 100U roughly.

Reply on news.ycombinator.com