marshallp

So some social skills and some database skills. That's pretty much the description of every IT job. What's so hard?

Predictive analytics has the possibility to radically transform many industries. This trend of hyping up it's difficulty is a dishonest tactic to inflate salaries. Pretty soon we'll get the fraudalent activities of lawyers and colleges with their accredition requirements if this trend isn't kept in check.

Not only that, "data scientist" is just a marketing gimmick for consultants. It's a standard set of skills that anyone trained in science or engineering has. It's ok to use it in front of pointy headed bosses but in front of nerds it's slightly dishonest.

The author clearly has a bias (I'm a data scientist so respect me and pay me a lot). He's then gone on to describe some standard programming and maths skills that a huge number of people have (taught to engineers/scientists/programmers). I'm going to get downvotes and be labeled troll but I just have to plainly disagree. Data science isn't some magic new career field, it's simply the application of standard scientific tools to tables of numbers. As netflix and kaggle competitions have clearly demonstrated, literally anyone from anywhere has a shot at be the best on any particular spreadsheet of numbers (that what it boils down to, (possibly large) spreadsheet of numbers).

Yeah, sure, whatever you say.

I wouldn't consider those really substantive compared to drawback of the military-industrial complex, radical increase in nasa/science funding, full free national health-care (including dentistry), overhaul of the education system (along the lines advocated by khan academy), ending the drug war, ending corporate welfare (and maybe even corporate personhood), and probably other major ones I can't think of right now.

Hackers should hack them into it.

Of course, none of this actually matters anymore. The corporo-verlords have bought both parties. The only way is parliamentary system with proportional representation. This isn't a political statement, just that hackers should investigate how to hack the system into that form. One clever hack is being done by John Koza (inventor of genetic programming) to do away with the electoral college http://www.nationalpopularvote.com/pages/about.php . Some more hacking is needed.

There are probably solutions to all the above. "Tapered" prize giving (even 100th place gets a small prize), more frequent changeovers (rather than tenure and multiyear student commitments, have monthly or weekly reviews about wh stays and who leaves, just like in the private sector). Anyway, I acknowledge your points, I just think the present system isn't necessarily the perfect system.

You're actually from kaggle so I'm going to look like a troll arguing with you so I won't try (though I do privately think the right question is pretty obvious always - will this combination of parameters make profit - or some other obvious single metric. Just give all data to the data scientist and have them build the model).

I understand what you're saying, but I'm skeptical that acdemic research is all that it's cracked up to be. Studies are always popping up about how most scientific findings are fraud. And then there's things like the X prizes and darpa challenges which really show who's right and who's wrong and what's actually possible. I'd much rather see open competitions used than grants. Kaggle can be that "central" competition for the machine learning community.

Well, that's rather rude. Maybe you haven't realized this is HN and not 4chan.

Are you sure grad students get paid more than that (that's who we're discussing here). And kaggle competitors aren't in it for the money. Almost every entrant will lose (it's winner take all) so all have them have day jobs.

My first comment to your first comment answered those objections. I consider all problems as optimizing a single metric. Anyway, let's let this slide.

Why not just give all data (or as much as you can afford to) to the data scientist and let them figure it all out.

Well, capitalism, which is about fostering competition, has led to faster progress than the alternatives (assigning duties to specific people - communism or dictatorship).

I'm not following your line of reasoning. Everything is data at the end of the day. All you're doing is creating a predictive model. If the business conditions change, you wold simply send it to the data scientists to reformulate a new model. That's no different to sending it to kaggle again.

The researchers actually do set up competitions to see who is best. However, there are few entrants. In kaggle there are 1000s of competitors, so it's a better judge of whether they are legitimately improving the state of the art (rather than simply schmoozing their local funding agencies).

You're saying kaggle is fundamentally different to what data scientists do? I don't understand that. The right question is usually how do I increase profits (or score this essay/image etc). So simply set up your problems that way.

All the evidence from kaggle indicates that deep domain knowledge is not required. Jeremy Howard has some youtube videos discussing this. Pretty much all the skills you outlined (except for production grade code - which is a software engineer problem, not a data scientist problem) are covered by the contest.

Why are they not entering contests? What is the point of their research if it isn't to engineer better algorithms for the task of machine learning? How do they prove that their research led to better algorithms?

Which brings up the question of what all these academic machine learningists (especially the theorists) are up to? Why aren't they winning kaggle? Why aren't Andrew Ng's own grad students collecting the prizes? Some self-reflection needs to go on.

You've completely missed the point. There are not Us vs Them. There are simply people. If you would like to label people as uneducated or muslims or whatever else, then you have engaged in that fallacy that pre-space age humankind engaged. You should really watch that link I gave as well as the Pale Blue Dot and let it really sink in.

You would like to limit immigration. What would companies like google like look after that? Would you advocate sending the head of youtube back to Iran? What about much of Detroit or New York?

You're conflating some poverty-stricken residents of impoverished parts of the world riled up by political rhetoric with a credible enemy (like a cold war style superpower).

You're arguments make as much sense as labeling all white people everywhere as Nazis because Hitler was a white guy.

Go to Liberia or the Congo or India. You'll see that people aren't civilized, they will eat each other if out of ritual. That was the case everywhere over a 100 years ago. You miss that it's not some kind of inherent badness, they are simply living in a different time, like 1800s europe for example. If children commit crimes they are not convicted in the same manner as adults, that's the correct analogy to take.

As civilized people and members of the global species that is humanity, it's a responsibility to take care of the weak. The problems you've outlined are (global) law enforcement problems. And they are disappearing, as time passes everyone on earth will become modern. Watch some Steven Pinker and see how today is the most peaceful time in human history.

Watch Carl Sagan's blue dot and realize it's not Us vs Them. That's the thinking of pre-space age man.

http://www.youtube.com/watch?v=WibmcsEGLKo (Charlie Chaplin)

First of all, "learning" is a made up word for parameter search, it's kind of a trick to fool funders to think you're doing cool stuff. Second, his entire blog post is about theory being useful only in a crude way (which basically means not useful) and he's outlined useful rules of thumb (that probaly come from experiment). Is time best spent on gathering data and running experiments to show practical usefulness of different algorithms or on pencil pushing? That isn't made clear.

But then what is the use of the maths. We already have the wealth the comes from the technology. The maths doesn't add much (instead siphons off resources).

You've given examples of old theorems, and also said they weren't actually useful at first. Later practical work lead to someone seeing use in them (but only the a vague way - limits - which could have been and probably found through experiment as well).

The example you gave of your proof doesn't say all that much, you won an argument (you said those algorithms were practically useful anyway).

I'm sincerely sorry if I come across as hostile. I'd just like to see mathematicians own up and change the corruption from the inside. A lot can come from it, you can change the world (you've taken up a lot it's resources, including some of the smartest people).

Yes, he's a pencil pushing fraud for the most part (he might even admit it in private). I don't know how old he is but assuming he's been professional for 10 years, his big contribution is the few hundred/thousand lines of VW (he's probably done other work, but let's assume this is a significant fraction). Where did the rest of his time go? If he just decided to bang VW out a decade ago he'd be at it's current state within a month of starting. VW is only useful because it's fast and it works (and that's not due to any theory). It's theoretical considerations are useful only for essentially drawing that 1 month out to years/decades.

Not at all, I never tried getting education in it in the first place. I had hunch it's useless and the older I get the more I think that was correct, although I do look at it for practical utility and never find any (I'm not arguing from a position of not knowing anything). People, like on this thread, bring up stuff and it's comes across as religious dogma, they don't explain how in their daily work a specific example of where it worked (and they should include how there were no other alternatives).

There's a huge, huge range of frauds being committed in society. I think people should point them out when they see it.

edit: I also agree mostly with the hard-headed engineers you brought up. Physics is somewhat useful, but it's also vastly overtaught and overfunded relative to it's practical utility. I'm an AI guy and think we'll get the singularity before any of the fundamental research going on pays off (and that's only a small proportion of the world's wealth being spent on physics education for people who will never find practical use for it). And I think there's also a lot of mathemagic symbol throwing in there as well to extract tax-payer money (a lot of it caused by the mathemagicians's influence on physicists).

People are tired of me criticizing maths on here so I'll stop with it.

My opinion on sports, arts etc. is the same. Publicly funded stuff is basically for the elite anyway, it's their way of cleverly siphoning tax-payer money (for there boring outdated interests which they partake in to show the illusion of sophistication). The masses pay to see their interests (and usually heavily taxed for it to boot).

I'm not angry at all. How can I add to the anti-math movement, I thought this might be a good venue, there's lots of thoughtful and productive people here. A lot of them might not be aware of the societal devastation caused by the current maths system (making most people not believe in their intellectual abilities and consequent destruction in wealth creation). Math anxiety news sometimes comes up and people like the Wolfram brothers make a case but on the whole people, including geeks, haven't realized how much computers have displaced mathematics.