Comment by dizzystar | Hacker News Reader

Comment by dizzystar | Hacker News Reader

Parent Full thread

Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity.

There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.

This is like... your opinion. Some people find pushing around HTML / JS / CSS absolutely soul-crushing. Considering the lion's share of websites are ugly, unusable, and slow, does this mean that front-end engineering is a breeding ground of mediocrity, so server-side devs and CFOs should all be sharing in the pain?

Some people actually enjoy working with data, and don't find ETL and pipelining horrible to do at all. It is a different set of challenges, but calling people people mediocre because of ETL is a non-sequitur.

I love ETL. I've worked with real Big Data (5PB+) and working on the data pipeline was my favorite part. The feeling you get when you rewrite a job to run 1000x faster so the company can make way more money.

That implies the work is in a revenue-center or that you're at the rare company that isn't myopically focused on sales.

No, it doesn't. If you make an M/R job work 1000 times more efficiently you save money for the company. It doesn't matter if you are a profit center or a cost center.

I don't think the author would disagree with you on that. His main point is that engineers don't want to be in a role where all he/she is doing is productionizing someone else's ideas. There should be areas of end-to-end ownership for both the engineering team and the data science team.

It doesn't sound like he's suggesting here that some types of technology are just generally horrible to work with or that it sucks to build ETL jobs in general. Maybe he could have done a better job defining what he meant by ETL engineer here, but he does qualify it in that quote with "ETL to produce data that you yourself never get to use or consume."

In general, roles that offer little areas of ownership do draw mediocre engineers.

Could not agree more. With all this talk of data and ideas, you would think he could include some data to back up his ideas. He can't, because this is all just opinion.

this.

Reply on news.ycombinator.com