Hacker News Reader

>With 10+ years in DS, I've always felt that best DS were always basically software engineers that knew math and were more interested in prototyping cool machine learning product than maintaining production infrastructure. Unfortunately this always accounted for a small fraction of DS I interacted with.

I've been a DS for 10+ years, and I feel the exact opposite. The worst "Data Scientists" I've worked with are all ex Software Engineers who seem to assume that business problems are really computation problems. So they find convenient ways to ignore the human aspects (e.g. trying to figure out why the data is a mess) and gravitate to using more complex algorithms and breaking down the problem to an achievable programming pipeline that runs in production, but the results are of low value. But it looks awesome on a resume.

Are you right or am I right about SWEs turned DS? I have no idea. But one quality that IMHO is important is the interest in actually looking at data and asking questions, which is much rarer than most people realize.

> Are you right or am I right about SWEs turned DS?

It doesn't sound much like your worst and their best is the same kind of person. I don't see necessarily conflicting views.

> Many of the people in this latter group had no idea how to write Python outside of a notebook, generally just structured problems to fit into XGBoost, and when not doing that tried to squeeze resume-boosting-complexity into any problem the could find.

To be fair, for 90% of business problems that require ML, I’d rather take the guy who throws XGBOOST at everything instead of the one trying to be fancy with neural networks. You get an explainable output and good results without deep subject matter expertise that the person likely won’t have. It also runs at a fraction of the cost.

Just use lasso first. It's so much quicker and often gives most of the value.

To be fair though, most value from data science is driven by the analysis, not the models.

With any ML/AI problem, based on long weary hours doing the grunt work, the vast majority of time spent will be getting the data into some useful format. It doesn't matter too much how fancy you can build your models if there's nothing to train it on, or worse still, unreliable or incorrect training data.

So for newly minted Math PhDs, sure go out and learn how to do some ML coding in notebooks, but if you can't get a decent dataset together to train it on it'll be all for nought. Anyone with AI/ML coding only, and no SQL, is a no hire in my book.

> They also tended to have a hilariously poor understanding of creating business value.

Is this different than your average SWE?

Yes, the average SWE is only moderately poor, not hilariously poor.

> Nearly everyone I know in the first group has switched back to just being an engineer of some sort, typically ML or AI engineer.

@OP - mind rerunning this analysis for "AI Engineer" titles? https://www.latent.space/p/ai-engineer anecdotally i saw 8 of these in the last Who's Hiring and wanted to tease out the emerging difference between ML and AI Engineer

A good DS is one who can understand the problem and tackle it using data, not someone who knows engineering well.

That is also my experience.