Hacker News Reader

Comment by genericpseudo | Hacker News Reader

The central tension alluded to here is "database people" vs "big data people". Which is a real dichotomy – as a "big data person" I've got a lot of leverage out of going "c'mon, we don't need Hadoop, we just need a relational database here, let's use Postgres", which is an option which gets culturally dismissed rather than on technical grounds. Me, I'm happy that I get to do less work and look clever while I'm doing it, but...

Marketing and cultural positioning matter. That really isn't news. But when people have identity invested in denying that, on both sides, it's difficult to overcome.

View on HN

In almost all applications using Postgres (or virtually any other decent RDBMS system) is the right technical choice.

But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. The why is this - data and access to data is now trumping applications for business value. Where as I can run almost any of my applications on Postgres (almost) and many of them can co-habit on one server relatively happily, I cannot run all of them on it. This means that I will end up moving data from one of them to the others, but which?

Alternatively I will shift the data from the RDBMS to Hadoop (or another data warehouse solution if I am working in oil-and-gas-three-years-ago or a hedge-fund-or-something now) where the insight that I am being kicked for daily can be extracted by combining the data product of 20 systems.

Or we can use Hadoop as the core of the data system and feed RDBMS slaves (or use Impala sometimes). This has the benefit that the data is managed and clean (ho ho ho) on the core system and we can move at the speed of security checking and data understanding rather than the speed of budgets for writing and running data extracts.

> But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop.

Google's F1 paper makes it clear that they were running Adwords, their most (maybe only) important product, on MySQL on Jan 1 2012. GFS and Big Table were not enough, and until Hadoop provides a robust ACID compliant data store, you should probably keep your pgsql books around.

Agree - there's no doubt that where transactions are needed you need something that can manage transactions (in the particular way you need them managed).

I actually think that transactions on Hadoop should be strictly limited to data import a-la HIVE but my arguement is that the enterprise data master is best on Hadoop rather in the slave systems.

>But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop

This has been my experience as well. Worked on a data warehousing project that eventually had to get moved to Hadoop because management underestimated the scale of the data. We could have saved a lot of pain if we just started on Hadoop.

About how this cultural divide came to be, part of me thinks that blanket rejection of "what works" is a sound strategy as an individual.

You don't want to work on a successful project. Successful projects are the ones you have been cramming 190 man.years of business rules into, and they did not explode.

You can thank Java's accessors, refactoring capabilities; you can thank SQL's ability to produce rich views without changing the data storage. You're still stuck with having to deal with at least all the essential complexity that is there.

On the other side, preach that the future is Go-Mongo-on-the-cloud and you've got yourself a greenfield project. Rarely have I heard about a Go project that is there only because it needs Go's unfair advantage (namely hugely high concurrency that you have to cram into one single box)

Same thing goes for microservices. Microservices are: I don't want to deal with other people's code.

> The central tension alluded to here is "database people" vs "big data people".

Then there's the group who think they're big data people, but actually fall within the domain of SQLite.

That seems to be the majority of self proclaimed big data people in my experience.

Which, in fairness to and out of love for SQLite, is an enormous domain.

I may have perpetrated things like this. (20GB working set for a recommender? No problem.)

Or Excel, even

Reply on news.ycombinator.com