Comment by physcab | Hacker News Reader

Comment by physcab | Hacker News Reader

This is a good overview, although I have some critiques.

1) The author doesn't really emphasize how complicated these algorithms are. Most people look at ML and think it can solve their problems (ie predict the future). In practical applications though (other than "toy" perfect gaussian data), you have to understand what the hell the algorithm is actually doing to your data. A lot of the questions that I encounter when doing ML research include: a) am I sure I implemented the algorithm correctly? b) is the algorithm doing what its supposed to be doing? c) how can I cross check its answers? d) if I use another algorithm, how do I gauge performance?

2) Like it or not, ML requires a very solid background in mathematics. You can get away with using algorithms "out of the box", but then its just that...a box. Knowing what goes on inside gives you the confidence that the solutions are correct and can be used reliably. It's important to know that ML is not like a car that you can just pick up and drive without knowing how an engine works.

3) I think the author should have listed easier algorithms that are used more often in industry. K-nearest neighbors is good, but I would also add k-means clustering and perhaps Principal Component Analysis (PCA) over SVM or KSVM. And if you're going to cover the more advanced stuff...you need to talk about Bayesian learning.

Nearest neighbor and k-means will return noise unless your data only has two or "at most" three dimensions, due to the curse of dimensionality. (http://en.wikipedia.org/wiki/Curse_of_dimensionality) This is the #1 trap that newbies fall into, since these algorithms seem deceptively intuitive but there's this huge unintuitive pitfall.

I agree wrt points #1 and #2. Especially #1.

My favorite supervised learning algorithm is decision trees. They have built in feature selection, and they are simple to understand and express in code.

Boosted decision stumps are my second favorite.

Both are "far better than" support vector machines for the real world problems I've ran into. Except the algorithm names don't sound nearly as cool as SVM.

The only reason why I advocate the easier algorithms is because most people understand how to take the mean of something and can intuitively understand what kmeans or knn will do to a dataset. And I'd even argue that most people wouldn't even consider something that was more than 3 dimensions.

Contrast that with SVM where you need to understand Lagrange Multipliers and differential calculus to see how those margins are setup.

Anyways, I think the author explained it far better than I could.

Any particular hypothesis why decision trees work better for you than SVMs?

Sorry for the late reply.

Here are the two I can think of off the top of my head: the first is that decision trees (or even naive bayes) produce results that are intuitively understandable and human readable. In contrast the result of training an SVM is a huge vector of 100's of floats and there's no real way to explain to someone (e.g. someone buying your tech, a manager, etc) what's actually going on.

I didn't mention this before but SVM's do get you something that decision trees, naive bayes, and others don't: they will look at all kinds of combinations of attributes. This becomes critical for applications like machine vision where e.g. looking at any one single pixel as an attribute won't do anything for you and so you need something that will understand lots of different possible combinations of attributes.

Which is also the second weakness of SVMs: most data sets that aren't images don't express strong patterns across a wide variety of combinations of features. To take naive bayes as an example, it is a hugely popular algorithm and the 'naive' part (which tends to work well on my data sets) is the assumption that each attribute is statistically independent!

K-means should work fine even with more dimensions as long as the statistical model that's implicit in k-means is a reasonable model of your data-generating process. Specifically: In the "relaxed" form (aka "continuous" form) of k-means that allows cluster centroids to be any coordinate and not just the coordinates of the observations, k-means clustering is equivalent to PCA[1]. PCA works under the assumption that your data generating process can be modeled by a linear factor model with isotropic Gaussian noise[2]. If your data fits these assumptions, or if you can preprocess the dataset accordingly, k-means should be able to give you more than 2 to 3 useful clusters. Machine "learning" usually works best when the dataset is first preprocessed and massaged to expose the hidden structure in a way that can be captured by the algorithms you're using.

Kernel SVMs are really mathematically cute with their infinite-dimensional-basis-functions and all that jazz, but I agree that in practice there's a lot more to be gained from understanding the problem at hand and its underlying stochastic process then selecting/designing algorithms accordingly.

1. http://portal.acm.org/citation.cfm?id=1015408

2. http://www.robots.ox.ac.uk/~cvrg/hilary2006/ppca.pdf

I disagree about kNN ineffectiveness. It produces satisfactorily results even for gene expression classification, where you have about 100 examples described with 10000 attributes. It does not succumb to the curse of dimensionality that quickly as it evaluates each attribute separately and only combines the results afterwards. The curse of dimensionality is more evident if the learning algorithm tries to deal with attribute interactions of higher orders.

Learning algorithms are just tools. No tool is perfect for every task. Also, you should use the tools you know well, because you can leverage them best.

SVMs are quite hard to be used correctly due to sensitivity to parameters, which do not seem intuitive. You need to know your data well to set them right.

Reply on news.ycombinator.com