The problem with using technology X “because Google does it” or “this is the best open source version of what Google uses, so let’s use it because Google does it” is that companies neglect that Google does not just use the technology out of the box they

1. created the software for their own needs 2. maintain a developer team to improve it and address requirements/pain points/integration 3. have an internal pool of experts in the form of the developer team and “customers”/early adopters 4. most likely have other proprietary systems like Borg or Colossus which integrate with the software very well, which OSS like Hadoop may not (another example: OSS Bazel vs Blaze+Forge+Piper+Monorepo structure).

Something like HDFS was hugely painful for many teams because they had no idea how it worked or how to debug it, had no idea how to fix it or extend it, and didn’t have any good tooling to understand why something was slow. All they could do was try to configure it, integrate with it, and find answers for their problems online. That’s because HDFS was “free” but a team capable of properly maintaining, supporting/operations, and developing HDFS was extremely expensive.

About 10 years ago, I worked at a place where we had a "big data" project. Some big wig wanted to use Hadoop. Turns out, the "big data" isn't big. It's not even a gigabyte, spread over less than a dozen files. Everything worked great, but the job orchestration took longer than the actual processing.

Despite this misuse of Hadoop, another guy really loves it, and decides to start a project rewriting everything to use MapReduce. A new guy started, got assigned to the "MapReduce project"... he worked on this for over a year. It never made it to production.

I recommend reading the GFS paper and consider that there were/are use cases for horizontally scalable, fault-tolerant object storage, with the nuance of understanding that some of your storage/data nodes may also be compute nodes and there can be a benefit or preference for assigning applications to run where underlying data is stored.

In the HDFS case with Hadoop's ecosystem, consider Hive, BigTable, Drill, and even Spark when running on YARN.

In the peak days of Hadoop, many organizations were primarily on-prem, and S3 or S3 compatible object stores were mostly reserved for people using AWS.

Hmm, I experienced the opposite - HDFS was one of the main reasons that one of the large companies where I worked was able to work at all.

> can’t imagine HDFS being a “Good fit” for anything.

what is the better fit if you need to store 1PB of data for cheap?..