Hacker News Reader

Comment by FZ1 | Hacker News Reader

I feel the term "Data Engineer" gets used for a lot of catch-all "we have problems that need an owner" situations.

There's not much consistency across job postings and interviews for this kind of thing.

I just interviewed for one "Data Engineer" position which consisted of nearly 100% stored procedures. No one knew what else to call it, and they didn't want to advertise for a DBA, because there were no real DBA responsibilities. So "Data Engineer" was chosen.

Another "Data Engineer" position was almost entirely Spark. There was no SQL involved - they expected all applicants to be Spark experts, with a deep knowledge of Scala.

It's hard to know what to expect out of "Data Engineer" positions until you walk into a place and start asking questions in the interview.

View on HN

For us it's simply the runtime you target. A frontend engineer writes code for the browser. A mobile engineer writes code for iOS or Android. A backend engineer writes code for servers. A data engineer writes code for the analytical/warehouse environment.

It was typically backend engineers doing the data pipelines to expose records from their own services, until the company got bigger and the analysts got more demanding, and that became a dedicated role.

Engineers are professionals in the business of solving problems :)

Scientists with thumbs was one definition I heard

I feel the name "data engineer" is perfectly fine for both positions. It's the job of the job post to clarify the requirements.

I don't see an issue with looking for a "data engineer with deep experience building and maintaining data pipelines based on spark".

What would you call it? "Spark operator"?

> It's the job of the job post to clarify the requirements.

I couldn't agree more. That's the exact thing I'm saying is a problem here.

When stuff is wrong, and we don't have enough people, and don't have a well-defined job, but need someone to just handle it .. "Data Engineer".

You get there, and everybody has a different idea as to what you should be doing, and what skills you should have.

Is this any different from “software engineer”?

SE usually at least knows what language they're expecting you to mainly work in. Not always the case for DE.

Ah gotcha, the problem isn't that it's a vague job title, the problem is that the job description for a particular position may not set expectations clearly enough

No - I'm trying to say that people often don't even know what the DE really needs to do. They know they have back-end problems, and need someone to clean them up.

But often the hiring manager believes it's probably mostly SQL or mostly python, and advertises that. They really don't know - because someone left, and they are trying to replace them.

So you get there, and talk to the hiring manager, and they're like "are you good in SQL?".

Then you talk to someone closer to the problem, and they're like "most of it's in Java - how's your Java?".

Then someone even closer to the position says, "well, most of the work we need done is really in Scala".

It's your job to just take it over, and make it happen. "Fix the data" - using our existing patchwork of tools ...

This isn't always the case - but given the nature of the problem, this happens a hell of a lot more often than with Soft Eng's.

Nope. Just someone that is willing to use tools in addition to build tools to deal with data.

Most who consider themselves software engineers only want to code and build new, not use existing. Mostly a mindset thing.

You're seeing the same thing happen with the "Data Engineer" title as what has happened with "Data Scientist."

In the hiring market, many positions that were previously titled "Data Analyst"/etc were renamed "Data Scientist." Similarly "Database Engineer"/"Data Warehouse Engineer"/"DBA"/etc were renamed "Data Engineer."

Why? Various reasons:

One might be the need to give people a sexier title because they don't want to be "just a data analyst."

Another case might be, hiring managers and recruiting teams changing their title to capture more views for people searching using these terms.

And why are people searching for jobs using these terms? Articles like the ones in the original post, and "Data Scientist: The sexiest job of the 21st century" ;)

I had to stop calling one of my previous roles “data engineering” because people thought I was just doing SQL script ETL... which really was maybe 30% of the job, but apparently that’s too menial to give you programmer cred.

hmm ... I didn't think about that. I wonder if the term is building a negative connotation?

Most programming is just plumbing with data, so I'd say the term data engineer would fit most programmers, most of the time.

> I just interviewed for one "Data Engineer" position which consisted of nearly 100% stored procedures.

How about data or database programmer? I've also heard "Data Analyst", "SQL Developer", etc.

But I agree that "data engineer", "software engineer", "network engineer", etc are too general to really be meaningful.

Doesn't the term software engineer side from the same ambiguity?

Reply on news.ycombinator.com