Hacker News Reader

Comment by rahimnathwani | Hacker News Reader

  how Llama and OpenAI could be cleaning and structuring their training data

If you're interested in this, there are several sections in the Llama paper you will likely enjoy:

https://ai.meta.com/research/publications/the-llama-3-herd-o...

View on HN

But isn't it the beauty of llm's that they need comparably little preparation (unstructured text as input) and pick the features on their own so to say?

edit: grammar

Yes, if you want an LLM that doesn't listen to instructions and just endlessly babbles about anything and everything.

What turned GPT into chatGPT was a lot of structured training with human feedback.

Exactly. Section 4.3.7 briefly explains how they trained the model to better follow instructions ('steerability').

Reply on news.ycombinator.com