Hacker News Reader

Comment by throw_away_777 | Hacker News Reader

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.

View on HN

The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc.

XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning.

XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... and https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced)

I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests.

It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).

I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implementation is an important factor that gets overlooked in academia. And on non-NLP and non-image datasets, usually the single best Kaggle model is an xgboost model, which was probably developed in 1/10th the time it took to make a good neural net model. Xgboost has come a long way since it was first introduced, with early stopping being an example of a significant improvement.

How can you say that ease of implementation is overlooked in academia, when academia created the exact tools your are speaking of?

Academia does not have to run the model in production. It has few computational constraints, and most datasets do not have a feedback loop, requiring combating drift, debugging, and retraining. Papers are often accepted when they equal or beat state-of-the-art. Not many academics have to deal with the business side of running models in prod.

All of this leads to ease of implementation being overlooked. Especially on NLP, you see a lot of overengineering with deep neural nets (where the feature engineering is hidden inside the architecture). These models are hard to implement/reuse.

But yeah: academia/theoretical machine learning creates the very tools for applied machine learning.

There are lots of reasons, principally among them that Kaggle comps problems represent only one tiny fraction of ML problems.

On general purpose ML, with out much time rigging a optimal solution, RF/Xgboost will preform better. But in many problems, ie vision, DL is vastly superior.

The other important point is that there many new opportunities for researchers regarding DL, where statistical ML approach on supervised problems is a much more well established field.

Also note that current "leader board" standard on MNIST is 99.8, not 99% he compares against.

https://en.wikipedia.org/wiki/MNIST_database

I think the hype around CNN is the NLP aspect of it as it relates to AI. If you can hammer down NLP and translate voice or text to computer-legible commands, you've really improved the user experience.

On the other side of the CNN coin is the image recognition that's getting a lot of hype from the self driving auto crowd. I think any data scientist worth their salt understands how the different algorithms stack up against each other. You wouldn't use xgboost for a computer vision problem just like you wouldn't use CNN for a tabular data problem.

Any good links you recommend learning xgboost? I've never quite figured out how they work.

http://xgboost.readthedocs.io/en/latest/model.html

http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf

https://www.youtube.com/watch?v=wPqtzj5VZus Trevor Hastie - Gradient Boosting Machine Learning

https://www.youtube.com/watch?v=sRktKszFmSk Ensembles (3): Gradient Boosting, Ihler

Because in the real world we need robust classifiers not optimised ones.

Problem : robust && optimized are very vague terms in ML.

Reply on news.ycombinator.com