Hacker News Reader

Comment by Etheryte | Hacker News Reader

Data mining doesn't mean the model is instantly updated, that would be prohibitively expensive at scale. It's way easier to batch your data together with a bunch of other data and use it later on. That doesn't even mean it will know where to find the information eventually since models are not one to one with their inputs, because again, size and cost.

View on HN

>Data mining doesn't mean the model is instantly updated

I'm not expecting instant. Even next week it won't be there. It's like how AI never learned to count how many times the letter r appears in strawberry. Like sure, now if you ask brave, it will tell you three, but that is only because that question went viral. It didn't "learn" anything, it was just hard coded for that particular answer. Ask it how many times the letter l appears in smallville and it will get it wrong again.

every week is still way to expensive to do at scale, at best they'll update training data with each model iteration.

I didn't think for a second you could be right, so I tried with Claude. L in smallville was correct, then it suggests it'd have gotten l in parallel wrong by answering 3 instead of 2 (buts gets it right in a new chat). Then it suggests it'd get n in millennium wrong by giving the right answer, and gets it wrong in a new chat. https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19

Thanks, that was enlightening.

It wouldn’t be instant, next week or even next month. Pre-training doesn’t happen that frequently and varies between each model provider. As for the strawberry test, this is a tokenization issue that is fundamental to LLM’s, however, most models can now solve this type of question using thinking/code/tools to count the letters.

https://imgur.com/a/NqIJEx6

LLM aren't retrained and released on a weekly time-scale. The data mining may only be reflected in the training of the next generation of the model.

Both OpenAI and Claude average roughly one flagship release a year, and these are some of the best funded companies in the space. The bigger your model, the more expensive it is to train, so you want to do it as rarely as reasonably possible. Every other company will either work with smaller models and/or train even more rarely, aside from fine-tunes and customizations they put on top.

What is "your data" here? Why would pretraining on "your data" be useful to AI providers?

The AI isn't going to learn factual answers to questions by training on the text of questions! How would that work?

LLMs aren't only about knowing answers to questions, it's also understanding the question to begin with that's tricky. What questions you ask combined with what followup questions you asked is a very valuable dataset, because if there are common threads across your users, you can train your LLM to understand the first ambiguous question better. This applies to pretty much every other interaction you have with LLMs too, the quicker you can deduce what the majority of the users actually want, the less tokens you need to generate to make them happy, the wider you can make your profit margins.

Reply on news.ycombinator.com