Hacker News Reader

Comment by flakiness | Hacker News Reader

For a reference:

Hadoop on Google Trends peeked at 2015: https://trends.google.com/trends/explore?date=all&geo=US&q=h...

"Hadoop is dead" seems to be a popular topic in past few years. https://www.google.com/search?q=hadoop+is+dead

Good points of evidence, however you can't read too much into the google trends. A lot of technologies peak when they are new and then fall to a steady state.

For example "computers" peaked pre-2004: https://trends.google.com/trends/explore?date=all&geo=US&q=c...

Javascript: https://trends.google.com/trends/explore?date=all&geo=US&q=j...

Machine learning: https://trends.google.com/trends/explore?date=all&geo=US&q=m...

In this case it's accurate. Hadoop is largely dead.

YARN, Hive, HDFS, MapReduce have been replaced by Kubernetes, Snowflake, S3, Spark.

And even that will continue to change.

Kubernetes is overused right now, it has its place but it's not nearly universally the right tool for the job.

Snowflake will eventually fall to something else due it's poor economics.

S3 and Spark though I anticipate to be around for a good few years and if they lose out it will be to imitators or evolutionary equivalents.

> Snowflake will eventually fall to something else due it's poor economics.

Can you elaborate on this point? What’s wrong with their model?

Entry level Snowflake is $2 per hour. Or $48 per day if you transfer the metric.

Entry level DO compute instance (so boring, I know) is $5 per month.

There is a large gulf of pricing ranges that can undercut them in the coming years. It doesn't matter now because a lot of analysis projects are disconnected from market forces due to their projects mostly being darling child green field projects or new revenue streams. The moment the next AI winter comes along, a lot of projects by then will start to look like legacy code and the original thought process turns into worrying about cost centers.

And my understanding is they jacked up the prices to boost the earnings to boost the stock price leading up to IPO. They can be disrupted much faster than they will decide to let off that pedal.

Kubernetes works very well for SaaS. The big problem is management of Kubernetes itself, but so far our company has had good experiences using Amazon EKS. I would not say it is perfect. However, it does allow devs to focus (mostly) on problems related to actual applications.

Unless you plan to throw out everything else and only ever use Snowflake, Hive Metastore is still as important as ever.

Almost every Big Data tools only works with Hive Metastore (and Amazon Glue Catalog, but the compatibilities is not 100%)

Yeah, very few companies are running Hadoop clusters on premise these days the way many were at least trying to 5 years ago.

Reply on news.ycombinator.com