Hacker News Reader

Is there a solution you are using to solve this that is more accurate and cost effective? I'm working through it now so would be curious

View on HN

Is scraping and putting this in a structured format too inaccurate or expensive?

That's the whole problem. If you have tons (100s of thousands or more) of labels, then you have "structured" data, but how do you correctly classify that scraped item into the correct label?

Putting all the labels into the LLM is super expensive per call when you have millions of items to classify.

You can't reduce the number of labels becasue they are correctly organizes/structured. This class of problem exists in many different domains.

100s of thousand? In that case I would label about a 100 by hand and train a supervised learning model.

This problem has also been solved for 3 decades now.

Ahhh, but the LLM processing is the kind corporate leadership and investors will actually agree to buy!

Orginally I started writing that as sarcasm, and now I'm not quite so sure.

Reply on news.ycombinator.com