FWIW, I think the title change removes a lot of information about what the post is about.
313 karma · joined March 19, 2009
1. That the particular input features that you've chosen are somehow the only possible choice. But who's to say that you shouldn't add new features which are the square of each original feature? Or maybe some cross product terms, like the product of the ith feature times the jth feature. Or maybe some good features to add would be the distance from each point you've seen so far. Etc. Continuing down this path, you basically get to the question discussed in the OP about choosing a kernel for SVMs. This is just one example where hyperparameters come into play, and you need some method for choosing them.
2. That a linear predictor is impervious to overfitting. Consider the extreme case (which comes up often) where you have millions or billions of features and far fewer examples (e.g., if features are n-gram occurrences in text, or gene expression data). Then it's likely that there are many settings of weights that fit the data perfectly, but there's no way to tell if you're just picking up on statistical noise, or if you've learned something that will make good predictions on new data that you encounter. In both theory and practice, you need some form of regularization, and along with this comes more hyperparameters, which need to be chosen.
Finally, by your reasoning, it seems like you would always choose a 1-nearest neighbors classifier [1] (because it will always end up with 0 error under the setting your propose). But there's no reason why this is in general a good idea.
[1] http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm
The basic idea of automatically tuning hyperparameters (the things this post discusses tuning with genetic algorithms) is cool, though, and is becoming a popular subject in machine learning research. A couple recent research papers on the topic are pretty readable:
Algorithms for Hyper-Parameter Optimization:
http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf
Practical Bayesian Optimization of Machine Learning Algorithms:
This is the early announcement this year, where we're soliciting suggestions about other forms of data to include for this year.
Working with her was one of the highlights of my undergrad education, and her class was great, too.
http://www.google.com/support/forum/p/Google+Apps/thread?tid...
http://www.google.com/support/forum/p/Google+Apps/thread?tid...
http://www.google.com/support/forum/p/Google+Apps/thread?tid...
http://www.google.com/support/forum/p/Google+Apps/thread?tid...
I'm personally having the same problem today, when the auto renewal date is a month away. I've filed two tickets without response. It seems the only solution people have found is to remove their credit card information from Google Checkout.
With the "filter[0]" in the url, all scores came back as 0 for me.
If you have other suggestions, please let us know, and we'll add it for next year (if possible). The only thing we're trying to avoid is somebody coming in with a lot of data at the last minute, beyond the point when anybody else can realistically get it incorporated into their model.
"Incorporating Side Information into Probabilistic Matrix Factorization Using Gaussian Processes." Ryan Prescott Adams, George E. Dahl, and Iain Murray. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, 2010.
Classification: For each of the twenty classes, predicting presence/absence of an example of that class in the test image.
Detection: Predicting the bounding box and label of each object from the twenty target classes in the test image.
Neither uses the full ImageNet data set. Instead, it's images from 20 classes of object, like shown here: http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2010/exam...
The generator you are talking about is far from producing a uniform distribution (which would have maximum entropy) jointly over the n-dimensional hypercube that would represent a sequence of n draws from your generator, so it has less entropy and is thus "less random" than the distribution you'd get from n independent draws from a perfect rand().
http://en.wikipedia.org/wiki/Entropy_(information_theory)
http://en.wikipedia.org/wiki/Maximum_entropy_probability_dis...
http://en.wikipedia.org/wiki/Weighted_graph#Weighted_graphs_...
This is not to say that counterfactual thinking is not encountered at all outside of mathematics. For instance, an obvious source of counterfactual thinking occurs in literary fiction, particularly in speculative fiction such as science fiction, fantasy, or alternate history. Here, one can certainly take one or more counterfactual hypotheses (e.g. “what if magic really existed?”) and follow them to see what conclusions would result. The analogy between this and mathematical counterfactual reasoning is not perfect, of course: in fiction, consequences are usually not logically dictated by their premises, but are instead driven by more contingent considerations, such as the need to advance the plot, to entertain the reader, or to make some moral or political point, and these types of narrative elements are almost completely absent in mathematical writing). Nevertheless, the analogy can be somewhat helpful when one is first coming to terms with mathematical reasoning. For instance, the mathematical concept of a proof by contradiction can be viewed as roughly analogous in some ways to such literary concepts as satire, dark humour, or absurdist fiction, in which one takes a premise specifically with the intent to derive absurd consequences from it. And if the proof of (say) a lemma is analogous to a short story, then the statement of that lemma can be viewed as analogous to the moral of that story.
You can use factor graphs for either directed or undirected models. See this paper:
Extending factor graphs so as to unify directed and undirected graphical models B. J. Frey Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence, 257-264, Morgan Kaufmann, San Francisco, CA, August 2003.