Hacker News Reader

Comment by bonoboTP | Hacker News Reader

Well, why do they want to? Is it their fault? People follow incentives.

Infrastructure work is low status, janitor-adjacent or like the guy who fixes broken toilets. Academic, abstract, scientific, work is less immediate, more rhetoric-focused, more open ended, similar to lawyers and politicians and is about selling ideas. Shoveling data is like shoveling dirt, while academic abstract modeling is like an architect drawing sketches of a palace.

Like it or not, infrastructure work is thankless. Complaining is useless, either accept it or switch jobs.

View on HN

This is some of the reason, but I've noticed the same behavior in data scientists who are "vertically integrated" and who are only measured on the quality of the final model.

In this case, their incentives are aligned with good data work, but they still suck at it and neglect it.

I believe there's a few psychological reasons:

- People aren't trained to do it in college, so it's undervalued.

- It requires domain knowledge, which is hard.

- The work is quite gruelling and annoying. It's very detail oriented.

- You don't feel like you're making progress while you're doing data work. Modelling work feels closer to the final output.

- Self-delusion during modelling is easy since you can overfit on the holdout data which gives you those dopamine hits.

- Modelling satisfies curiosity. During data work, the data scientist can't wait to hurry up to the modelling stage and see what the results look like.

- Many people are just bad with data work, they don't have the attention to detail required and produce data with glaring errors.

Again, this boggles my mind. Data cleaning is 90% of the job. If you don't like this, then you should not be working in ML/DS.

I actually enjoy data cleaning, sure it's annoying sometimes but the insight you gain actually allows you to build a much better model.

And, to counterpoint one of your points, I would note that part of my (psychology) degree was collecting data from our friends for various surveys and experiments. We'd then add everyone's data together and run analyses on this.

It's really crazy to me that stats/ML/everyone else didn't do any of this, as it certainly lead to me being a much better data scientist.

Incentives. Do you have skin in the game? Are you building your cv and career and brand or are you building the product? Do you personally benefit more from 10% improvement due to data cleaning or 1% improvement from some fancy tweak of the model which you can hold talks on and write papers about, with math formulas etc?

No, but I like making/saving money, doing analysis, writing code and talking about how to do all of those things better.

Especially the making money part, as that's ultimately why my employer/clients pay me.

You have the right practical mindset, and it might have something to do with your degree choice. People that pick psychology aren't going to have their head in their clouds and aren't doing DS for prestige reasons - DS is more a means to an end instead of an end itself. You're also trained only in fairly simple, mostly linear models, which helps.

There's an even more extreme type to the one I outlined. This type of DS dislikes data engineering and dislikes applied modelling.

They prefer to sit in an armchair and think about modelling, writing up a LaTeX document with abstract and overcomplicated math formula.

I met one DS like this who was from a theoretical physics background and his years of operating in this mode couldn't be shaken off.

Just as an FYI, DS didn't exist when I did my undergrad, and Facebook were the only people with DS during my PhD.

I got into DS because I really enjoy analysing data and running experiments, and as a backup to academia.

Not to mention it pays far less. I've interviewed for data jockey positions on AI/ML teams that were paying about $85k. The people building the models were making double or triple that.

This is the real issue. It's considered less-skilled and therefore by supply/ demand more can do it and it pays less.

I think this is probably only because it's harder to assess skill in it, though. It's easy to test for knowing the vagaries of the ADAM solver or statistics or whatever. What is the equivalent for being a good data cleaner, though?

Reply on news.ycombinator.com