dvcat

I don't understand why using mturk is against Dropbox's TOS. Surely from Dropbox's pov, the privacy (or loss of privacy) is identical in both the case of mturk and adwords.

Maybe my point wasn't clear enough: 1. I am comfortable with using someone else's library without having to reinvent the wheel but I want to know exactly what I am getting into without having to browse through tons of Java code. There are zillions of variants of algorithm X but I want to know exactly which implementation/variant Mahout uses without going through the source code. Unfortunately the docs (at least 4 months back) were pretty bad.

2. Their unit test coverage was not good enough which incidentally is how I found that there were bugs. The problem in trying to contribute back to the community by trying to rectify these bugs? When I read the source code, I get the feeling that each algorithm is owned to a great extent by one developer who brings in their own idiosyncrasies which means that you need to really study the code to make sure you don't accidentally add more bugs. The other disadvantage of this approach is that questions regarding potential bugs and puzzling issues can go unanswered or answered in an unsatisfactory manner (mainly because of the one developer writing most of the code issue).

Having said all this, I want to be charitable and chart these to growing pains. But if I were building something critical and big dataish, I would either use Python (dumbo) or Scala which are much more concise languages where it is easier to express math without introducing bugs.

Dunno about weka but my last experience ( 5 months back) with Mahout was not good. There still are quite a few bugs and the fact that entire code base is in Java makes it extremely unpleasant for someone who wants to hack and modify the code to jump right in and start tweaking stuff. However, in its defense, it is open source is probably the only hadoopified ml library out there and has given me a ton of good ideas on how to write custom code.

This makes a lot of sense. I was working through a couple of the assignments last night when I realized that a lot of the questions seemed to test reading comprehension rather than proper understanding of the material. I had a while back used Prof. Ng's video lectures and notes (available on his site) as a supplement to my own machine learning class and those are designed to both stretch your mind and test your understanding. I was kind of hoping that that level of quality was being shared with the rest of the world but that doesn't seem to be the case :(

Can anyone clarify if song data is dense? If it is dense, I am not even sure if Mapreduce is the right paradigm to use mainly because you will eventually get to a situation where transfer time overwhelms compute time.

2) "Publishing" (Blogging about this) will help. I am not too sure what you mean by "know the techniques well enough". In my case, I cracked a few books on LA/Stat and worked through them and encountered situations where I had to use those skills repeatedly as part of courses I had taken in School. Doing tiny projects where you are able to acquire mathematical intuition for certain concepts will be useful.

4) So the fact of the matter is that you should be versatile enough to absorb their tech stack as soon as possible. That is possible only if you have worked in depth in at least one tech stack. This is especially true for Map Reduce based technologies.

Its hard to be specific: All those things are doing in analytics departments. There are not many machine learning people in my place of work so I mostly find problems that require algorithmic solutions and try to see if I can solve them. Things probably are more structured in larger companies..

I do machine learning and would consider myself a data scientist. I was an engineer who decided to do an advanced degree in statistics and computer science just because I liked this stuff. I currently work in the analytics division of a small company:

1. Its not necessary to have the necessary degrees: I did but a lot of people in my team come from the social sciences and other fields. You might find it hard to cross HR but that is something that can be rectified by cleaning up the "weird parts" of your resume and highlighting the "right parts". You seem to have a good handle on what is what on that front.

2. Your statistics, linear algebra and probability skillset need to be upto par. People from a more statistical background will grill you on those things. Its extremely easy to see whether a person can think statistically by giving them a toy data problem and asking them to hack at it. The way to train for it is to play around with small datasets and I see you have been doing that a bit.

3. People who come from a more C.S. side of things will try to explore your knowledge about "machine learning algorithms" which typically are easy to learn if you know your math background. The field has a lot of jargon which might appear to make it fancy. Again, the math behind these algorithms is not hard but there are things that you learn about how these algorithms work in practice that really make a difference. So again doing small projects and putting them up on github will help you learn more and make your resume look good.

4. Technology: There are loads of languages that are used in practice. Make sure you know one scripting language (R, SciPy/NumPy or even Matlab) and are comfortable using that as your scratch pad. The people who are statistically oriented in my team use R. Other skills that are extremely valuable but won't kill you to know are to learn the Map Reduce Stack (Java (uggh), Pig).

I am currently doing machine learning on a dataset. This involves typically playing around with the data in NumPy and sometimes Matlab. Once I am comfortable with a particular choice of algorithms, I try to write it up in Pig. I use Java (Hadoop) for the worst case scenario.

Hope this helps...

I am mainly writing code for Hadoop. I find that I learn better when I actually physically write the code which is why I am trying to find a more vim like editor but the problem is that Java is incredibly verbose which makes it rather tired to keep rewriting Map and Reduce code.