Hacker News Reader

Comment by jandrewrogers | Hacker News Reader

The salaries are already moving north of $200k even outside of Silicon Valley and New York City and getting more expensive by the month. How high do they have to be before we have a "shortage"? The problem is not lack of money, it is that demand has greatly outstripped a finite supply.

Very high wages do not automagically create new people with the requisite skills and this is the real bottleneck. It takes significant aptitude and years of training/experience to become useful as a "data scientist". It is not as easy as I think people are imagining. We train people with excellent raw skills where I work, usually strong applied mathematics backgrounds with natural programming skills. It is much easier than trying to find someone outside with these skills, though we do attempt outside recruitment. It still takes years to develop the people we train into a good, basic data scientist.

View on HN

Finally, some basic labor market economics. Just like how employers have restrictions on the number of people they are able to hire, employees have restrictions on the number of hours they are able to work. Or for the sake of this example, whether they are able to be a data scientist or not. The aspiring data scientists' restrictions all have to do with their ability. And as you point out, the cost of human capital investment in this area is very, very high.

One doesn't need to be an econometric theorist to be a data scientist, but I feel that far too few hackers truly appreciate the elegance of some standard, say, 1st year economics grad school econometric models. Things like panel models, IVs, 2SLS, GMM and even just taking seriously the basic assumptions of OLS regression -- there's a reason most econometrics classes (grad or undergrad) always start with the ~5 assumptions of OLS regression.

TL;DR Economists would make great data scientists (and better economists) if only they understood and appreciated computer science more.

In context: $200k is roughly what attorneys are paid in their 5th-6th years at large firms that service large corporations. Lawyers are in surplus right now.

In finance, talented individuals are routinely paid well multiples of $200k for their work (even post-crash).

So while $200k is high for salaries generally, it certainly is not high enough to imply a shortage in a highly specialized field.

Look, this job title is at most 2 years old. How can someone have years of experience in this? OTOH, there are plenty of people with strong applied math and good programming skills.

The set of skills existed before it had a trendy job title so you can have the experience even if it was called something else. This is true of most of the people currently working as data scientists. In a similar vein, I was designing big data systems years before "big data" became a term or trendy. For any particular odd skill mix you can come up with, there are people with that skill mix who are already doing a similar job. But usually people do not intentionally build that skill mix until it becomes an official job title and career path in the eyes of the public so it is a very small pool of people.

In the case of modern data scientists, having strong applied mathematics and programming skills is about halfway to where you need to be and a good starting point. The demand has temporarily grown much faster than the convertible talent pool can develop the additional set of skills required.

I am of opinion that if demand is high enough, companies will start hiring "halfway there" people. But this will happen only if the market grows big enough. Right now it is still a niche market where companies are cherry-picking right candidates, it seems. At least this is the impression I get from reading this thread.

The question of the size of the market is crucial. Small labor markets are very inefficient. This means that the number of qualified people is small enough, but the number of companies they can choose from is also small. It is hard to find a job when the number of companies hiring is probably less than 100.

Reply on news.ycombinator.com