Is that the reality of today?
Personally I too feel that distributed computing is an overkill for most 'big' data problems.
But even if you "collect everything and sort it out later", in my own personal experience and in what I've read here on HN, you can go a long long way before you need to reach for the power tools. What most companies call "big data" is typically not that much (in quantity and in velocity). Most companies don't have tens or hundreds of terabytes of data. For example, I'm currently processing timeseries data in postgres using the timescaledb extension, which makes it perform very well. Still too early to state numbers, but its looking promising so far and if their claims are true, then I won't need anything else. We will see :)
And it really only makes sense to have permanent infrastructure for distributed computing if you're constantly using it. Like if you're constantly rebuilding an index of the internet. Which most people aren't.
For occasional reindexing jobs, I've personally had success with Kubernetes. Our customer facing services were deployed with it, so it was trivial to bump up the number of nodes, and then schedule a bunch of workers containers, then roll everything down when the job is complete. No need to learn the ins-and-outs of Hadoop.
In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data.
And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.
Note I'm not saying all. Just 99%, like the parents comment referred to. That leftover 1% are the companies you can name off the top of your head. ExxonMobil, Target, Chase, Visa, etc.
If you have data that is actually big (= doesn't fit onto a machine), then Hadoop is a reasonable candidate. Otherwise you are fine with a lot simpler tools.
This has probably been true for quite some time, and companies are just slowly realizing now that their relevant data isn't actually that big. On the other hand computing power and storage has still grown in the last years, and less resource intensive ETL has become more acessible, so the bar for "big" data has been raised quite a bit.
The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did.
Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.
It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.
We don't know about the specifications because we don't work there and since the OP said he wasn't there he might not know either. And the fact is that requirements regularly change over time.
The point is that it's really easy to judge when you aren't there and are privy to all the facts.
And you're damn right I'm going to be judgemental of someone not only promised the moon, but abandoned the work when the rocket exploded on the pad and someone else had to clean up.
> Implemented big data / real-time Hadoop streaming ETL service processing billions of requests.
I've become very skeptical of anyone who puts a combination of buzzwords and pseudo-numbers in their resume.
I know this is circumstantial, but on average there were 2 developers. The company needed something to just get the bills out. It also needed to be correct, stable, easily expandable for new datasets (requiring all kinds of conversions), testable from unit to e2e, integrate with other APIs to retrieve metadata, and .. I forgot a few, probably.
Hitting a performance or stability limit with MySQL with an unsophisticated schema/architecture does not mean you have big data. But that's the scenario that's common.