Hacker News Reader

Comment by CRConrad | Hacker News Reader

> Learn, learn, learn. Find the new hotness

58 in a few weeks, and seriously considering going the other way: To jump off the treadmill of new fads and concentrate on / return to basics.

"Big Data!" "Data Lake!" Data Vault!" "Apache Spark!" "Kafka!" "Cloud this!" "Cloud that!" "Cloud the other!""Snowflake!" "R!" "Python!" "Pandas!" "This new ETL tool!" "That new ETL process!" "This other new ETL tool!" "That other new ETL process!"...

Sigh.

SQL and bash ain't going anywhere, and they're all you really need for ETL.

I'm thinking of really learning the old cool in stead.

View on HN

Yes, that's a valid path, too.

When I was just starting out (circa 1990) a good friend of mine had a pal that was making his living servicing punch card machines.

They were completely unsupported by IBM at the time, this guy had been an IBM tech and saw the opening. He embraced the old tech, knowing there were people still using it and willing to pay for some form of support.

So I don't disagree with that idea, it's definitely workable.

SQL isn't quite comparable today to punchcards in 1990, is it?

Depends on who you would ask.

Anecdata: was asked by a client why the same query was running orders slower on our infrastructure (IaaS on Xeons) than on their test server (a regular desktop with i7). I check the load, IO, yada-yada and I don't see anything what could indicate the slowness. After a bit I check the db size and ... I'm pretty speechless, because it is 77MBs. 77 megabytes and the query runs for tens of seconds. I tell the client to give me the query. They are happily oblige and provide a two FullHD screens of SQL with like... 20? More? 'SELECT *' from the same tables on and on.

After speaking nicely with the client about the origins of this query and checking their dev environment, I learned:

1. this query was autogenerated by Lavarel

2. their dev environment is 100 times smaller than the prod

3. until I forced them to copy prod data to dev they didn't believe the problem was with the query

4. between two programmers and one sysadmin on the client side NOBODY was even close to reasons of slowness.

So... for some people SQL in 2022 is pretty equal to punchcards.

My first dev job about 15 years ago I took over this in-house developed intranet. Some queries took quite long (30 to 60 seconds), and according to "the last guy" there wasn't any way to speed it up. At the time I didn't know anything about SQL (literally never used it before), but I figured there really had to be some way to do this faster. I just read some basic documentation, rewrote a few queries, and now I got the exact same results in 1 to 2 seconds. A few months later after I learned a bit more about SQL and optimized it further (just by adding an index IIRC) and now it was fast enough to appear instantaneous.

I'm still far from an SQL expert, and certainly wasn't 15 years ago, but you can get a lot of win by just spending as little as one or two days learning about SQL. It really surprises me how some people don't.

Then again, for a very long time I thought awk was basically useless to learn, until I did last year after which I kicked myself for not learning it sooner as I had spent a ridiculous amount of time cooking up inferior solutions for ~20 years, and spending just an hour or two learning awk would have been a great ROI *shrug*.

I guess the moral of the story is that you can never be quite sure if something is useful or useless until you actually learn it.

With the rise of ORMS, I'm coming across more and more devs in my work that have no idea how to write performant SQL, and don't ever check to see what gets generated and run "behind the scenes"

Pandas is a pretty fantastic tool for ETL.

Only if you've never used dplyr ;)

Awesome, thank you so much for the suggestion. That is one of the reasons why I am active on here.

I was being a little snarky. Dplyr is phenomenal but it's written in R which many people perceive to be a weird language.

Reply on news.ycombinator.com