Hacker News Reader

Comment by tormeh | Hacker News Reader

What's the point in limiting yourself to small datasets? It forces you to be "clever" about preprocessing, because the levels of freedom in your learning algorithm must be limited to match the size of the dataset. Being clever like this is precisely what we're trying to avoid with machine learning algorithms. It's better to just shove the raw data into a very general algorithm like neural networks and let the data do the configuration. And do that you need _lots_ of data.

There is a conflict between an algorithm's performance and its learning speed. Restricting ourselves to low amounts of data means we get algorithms with lower maximum performance than we otherwise would. These are then claimed to be better than algorithms which need more data to generalize well.

View on HN

The goal is to allow for "fair" comparisons.

But I agree with you in general. I suspect even LFW's creators might; I don't know if they expected the benchmark to still be tested on this many years after creation.

I've always felt that all benchmarks in vision should come with expiration dates, because eventually everyone starts implicitly overfitting to them.

Also, one huge reason why we might "limit ourselves to small datasets" is because that's the only way we can compare many algorithms.

Outside training data has a huge influence on algorithm accuracy. For example, since Facebook has access to bajillions of face images that they could use to train their classifier, and since they can't share those images with the rest of the community, it's unclear whether their excellent performance is because they have gobs of data or whether it's really a better algorithm. I bet that a simple algorithm like a naive SVM might do leagues better if we could train it on Facebook's (hidden!) dataset and test on LFW. It just isn't reproducible---a measuring stick is only meaningful if everyone uses the same measuring stick.

Isn't small datasets the reason for using SVMs? You are, after all, operating in linear space preprocessed with a kernel trick. Far less expressive than a neural network.

Well, isn't it a bit awkward when dataset size is what forces you to select a certain algorithm that you otherwise wouldn't use? "Gee, I would love to train a neural network, but it's just so much data and I'm on a deadline; maybe I should just use an SVM and hope for the best ... ..."

That's one of the big surprises behind deep learning these days: it's now feasible to do things like "train a big neural network on bunches of images" in a sensible time. It's an optimization thing as much as it is a machine learning thing, in my opinion.

I didn't mean that training time is the constraint. It's that when training a more general algorithms (ANN) your hypothesis space has more dimensions than when training a more specialized one (SVM). You therefore need more data to train an ANN than a SVM. The reason for choosing SVM is not that your dataset is too big for ANN, it's that it's too small for ANN.

That's a great point, and it's exactly why the LFW results page makes a clear, distinct separation between algorithms that use outside training data and algorithms that do not: http://vis-www.cs.umass.edu/lfw/results.html#notesoutside You can probably guess which ones generally do better.

(Note that this is separate from the "unrestricted" vs "restricted" issue)

Reply on news.ycombinator.com