Hacker News Reader

This resonates with my experience. Use LLMs for feature engineering, then use traditional ML for your inference models.

Perhaps the reason that this approach works well is that, while the LLM gives you good general-purpose language processing, the decision tree learns about the specific dataset. And that combination is more powerful than either component.

It’s the same reason LLMs don’t perform well on tabular data. (They can do fine but usually not was well as other models)

Performing feature engineering with LLMs and then storing the embeddings in a vector database also allows you to reuse the embeddings for multiple tasks (eg clustering, nearest neighbor).

Generally no one uses plain decision trees since random forest or gradient boosted trees perform better and are more robust.

Reply on news.ycombinator.com