> 1.75GB
> 3.46GB
These will fit in memory on modest hardware. No reason to use Hadoop.
The title could be: "Using tools suitable for a problem can be 235x faster than tools unsuitable for the problem"
> 3.46GB
These will fit in memory on modest hardware. No reason to use Hadoop.
The title could be: "Using tools suitable for a problem can be 235x faster than tools unsuitable for the problem"
People have a desire to use the 'big' tools instead of trying to solve the real problem.
People both underestimate the power of their desktop machine and the 'old' tools and overestimate the size of their task.
Specs be damned!
I need to start selling boxes
- Edward Tufte
Applies to more than just design.
SQLite does all of that and more with every release (https://sqlite.org/testing.html). There's a reason I consider their test coverage impressive...
Not only that, people seems to love claiming that they're "big data", perhaps because it makes them sound impressive and bigger than they are.
Very few of us will ever do projects that justifies using tools like Hadoop and to few us are willing to accept that our data fits in SQLite.
That was exactly his point.
You'll end up with a half assed, homegrown Hadoop implementation, wishing you had just used Hadoop from the beginning.
*The original article is from 2014 and references c1.medium which are no longer available. c4.large is the closest you can get without dipping down into the t2 class.