State of Hyperparameter Selection
startup.ml
startup.ml
Most damning though is probably that neural nets do not adjust to new data quickly - we would have to train a new one every time we acquire a new data point and training neural nets is incredibly expensive (it might be cheaper to train this new neural net than the original one, since hyperparameters are almost always of lower dimension than the data being fit, but it will certainly be more expensive than modeling a Gaussian Process).