very simple processing, not memory bound, tiny data-set - of course its going to be faster locally when the command itself takes less time than the networking, distribution, coordination and collation overhead of using any distributed tool...
Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.
Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.
You've seen people writing pages and pages of Java code to do ETL ?
The rest of the world does not particularly care about SV fashion trends.
Edit: We're also a fairly greenfield team, with a product in beta that's less than 4 years old. So like, the leaders of the team knowingly crafted us in this particular direction.
Well, Hive and Spark create MapReduce jobs to satisfy your queries: it's still in the background, but you don't have to think about it (as much).
I think the point if the article is exactly that: Unless you have a couple of TB to process don't even bother. I remember a similar post in the same spirit that had a couple of GB in a Postgres database and ended like "Just do a simple query!" instead of a Hadoop cluster for simple map-reduce.
My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.
At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.
Threadripper 3970x = 32 cores @ $1999
AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot)
You could buy a year of AWS for the same price as a Threadripper based PC.
For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU.
BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.
And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.
The fact that "deal with it locally" was obvious to the author, but not to the people who inspired the author to deal with it that way (he mentions an EMR job), means there are those it may be worth pointing it out to.
Certainly, being reminded that you probably should solve your problem without distribution, unless you actually demonstrate that you CAN'T solve your problem without distribution, is nothing but a positive.