Comment by eric_b | Hacker News Reader

Comment by eric_b | Hacker News Reader

Maybe these people want to leave because they realize they aren't adding value?

I've worked in many Fortune 500's with "data science" or "big data" teams. These are well staffed, very expensive teams that have large budgets for pricey hardware (sometimes on-prem, sometimes in the cloud)

I have never seen one of these teams produce insights or actionable intelligence valued anywhere near their cost. I mean not even close. Usually it is a tremendous money fire. (Also, before the pitchforks come out, I'm sure there are places where the data science team is a profitable department, but it's not the norm, not by any stretch)

Part of the problem is the business doesn't know what questions to ask. Part of the problem is the technology itself. Spark streaming, Hadoop, and all the other tools really aren't very good (very good being defined by helping businesses answer burning questions in a reliable and timely manner)

The most valuable data insights I've seen come from purpose built analytics tools using simple storage backends (RDBMS, elasticsearch etc) where the person running the team is a domain expert, not a "data scientist".

Dollars to donuts, this stuff always seems to be in that enterprise buzzword laden minefield of trendy make-work. There seems to be a tremendous amount of money and energy behind it for some reason, though.

I've seen some customers of ours being quoted six and seven figure prices by their internal data science teams to essentially slurp and transform a handful of SQL tables. It's the kind of thing that shouldn't cost six or seven hours to write the code for.

> I've seen some customers of ours being quoted six and seven figure prices by their internal data science teams to essentially slurp and transform a handful of SQL tables.

They don't wanna do it so they give a ridiculous estimate

I'm a firm believer that data science should be pulled out of IT and put into the business. However, storage is so cheap that you should never NOT collect data if you can. It's better to dump it into a data warehouse and wait for somebody who can use it to come along than it is to never collect it.

Then, and only then, should you look at your service levels and determine if there is a need for some sort of Hadoop or Spark infrastructure.

> However, storage is so cheap that you should never NOT collect data if you can.

Unless it involves customers and clients, in which case it's actually a sneaky liability... At least in terms of PR, if not legality.

I like data too but that approach has several hazards. The most obvious is liability: if you store it, you have to be responsible for protecting against misuse.

The second, however, is more subtle but often more damaging: people often assume that the data they have is the right data or complete so the drunkard's lampost problem is easy to fall into and people might not realize it as quickly because, hey, those numbers were based on so much data it took a day to run the query! Web analytics is notorious for that — people would make statements about performance, browser support, etc. from a bunch of log files and miss that this was skewed by bot traffic, confuse their server's response time as being a reasonable proxy for the user's perceived load-time, etc.

>I'm a firm believer that data science should be pulled out of IT and put into the business.

I say this from personal experience. The major risk here is that Data Scientists will constantly get pulled in as resources to support business fires and high visibility projects that should be handled by other people.

Otherwise what happens is this: "I know you are working on that Data Science project that is scoped for 4 weeks, but can you do this Business Analyst thing the insert exec name here asked for by this Friday?"

Data Scientists need to be shielded from that day to day analyst stuff, otherwise they'll never be able to do their jobs. One of the ways to do this is to put them in IT in a business facing consulting role. IT is typically (not always) is more focused on a proper solution, not day to day triage. So they can filter out projects that aren't appropriate.

This may end up repeating the mistakes of a decade or two ago: after a business leader gets two requests rejected, they are very unlikely to go back with a third request - they will simply outsource every future request. Soon the IT departments were being downsized because they couldn't get business leaders to use them instead of going outside.

Data should be moved out of IT and put into Data. Data headed up by a Chief Data Officer should be a first class citizen of every company. Not f’in IT, not Buisness. Data.

The businesses usually think they have a lot of data because they look it in terms of number of years of data but in practice their datasets are only a couple hundred megabyte large. Yet they setup Spark, Hadoop, and all of that stuff to "extract insights" from it. Usually all they need is someone who knows python to write a script to parse their data and put it in Postgres

I think this is a key point - too often, data "scientists" are brought in, given lots of tools, paid lots of money, and then told to just count things. And build dashboards that count things.

The data scientists that I have seen that are successful and bring the most value and seem to have the most satisfaction are those that spend time actually doing analysis and provide deep insights into the problems they are looking in to. The answers are the easy part - asking the right question is the difficult part.

Reply on news.ycombinator.com