Hacker News Reader

Comment by throwaway_pdp09 | Hacker News Reader

> People always looked down on those that used Hadoop or somesuch for <1GB of data, but while it wasn't needed from a technology perspective it gave a structure to the project.

What 'structure'? Why is it so important that it makes it worthwhile firing up a large, complex framework? I'm beyond baffled.

View on HN

The same 'structure' that makes it easy to onboard new co-workers because they've seen the same project 'structure' before in the past. In that sense, the bottleneck in an organization is getting people productive as fast as possible, even that means using a cleaver instead of a scalpel.

If all they can use is a massive cleaver (big data tools), and have no experience with scalpels (small, sharp, cheap and fast data tools), IMO your company has a serious, fundamental and systemic problem (no, let's call it failure) towards employee experience, training and knowledge. Edit: and resource management.

Seems to be a sort of inverse of the massive spreadsheets that run supply chains on accretions of spaghetti-macros.

But, a tree chipper can serve as a paper shredder, and I imagine a lot of shops in certain markets saw it as a sort of prestige asset around 5-8 years back, when a bunch of companies started hiring data scientists for no apparent rational reason.

(Not bashing data scientists or data companies. Just remembering the fad that went around Bay Area companies a while ago.)

Reply on news.ycombinator.com