Hacker News Reader

Comment by listenallyall | Hacker News Reader

> novel data analysis methods

Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own from scratch -- it's not going to happen, they just don't have that skill, the instructions on the box don't account for that possibility. As soon as the task diverges from the simple step 1, step 2, step 3 that they were originally taught, you realize they have very little ability to adapt. YMMV of course.

View on HN

> Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc.

This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe work, don't hire people will recipe skills. Do not have job interviews that select for recipe people. But, that is exactly what most companies do.

Precisely.

Yep. The key is really software skills. If you’re unable to even filter the data yourself, you’re also probably unlikely to be able do implement novel analysis techniques, especially if the analysis algorithm has many complicated steps or is computationally expensive.

In all fairness, it’s basically impossible for a new grad to have those skills. 4 years of a bachelors in any field isn’t enough to cover such a wide area. Even for people with graduate degrees it’s a stretch.

The hope is that the 4 year degree gave you the ability to quickly pick up those skills on your own.

If your four year degree didn't give you the ability to learn and expand your knowledge on your own, its a colossal waste of your time and money.

Sure, but depending on what you’re doing, “quick” might be years. You can get a PhD in understanding the theory, a PhD in designing fast numerical algorithms, or spend many years becoming a strong software engineer. I think the willingness to learn a diverse set of things is much more important than learning narrow areas fast. The short length of a bachelors usually isn’t enough to get this diversity.

>Asking them to clean and scrub the source data set...it's not going to happen, they just don't have that skill

I think you've been working with conmen/conwomen. I've never seen a data science project that doesn't involve data cleaning or wrangling of some sort.

Have you read through the comment thread? Did you read the article? Most everyone is in agreement that projects require a lot of cleaning & wrangling and a lot more -- the point is that data scientists are generally not doing that stuff, they expect academic-quality, pre-processed, pristine data, so it's data engineers who are stuck preparing the data, and who are in high demand.

Yes I read both the article and the comments.

I meant a data science project in terms of a project completed by data scientists. In my experience, all data scientists are accustomed to doing extensive cleaning etc.

Reply on news.ycombinator.com