Hacker News Reader

Comment by GolDDranks | Hacker News Reader

GPT-3: 174B parameters. Phi-2: 2.7B parameters.

Indeed, parameter-wise that is small, 65 times smaller. However...

GPT-3: trained with 300B tokens. Phi-2: trained with 1400B tokens.

The volume of training data, on the other hand, is around 5 times larger.

For curiosity, just the other day I calculated that a human baby learns a language with around 30M "token-equivalents" of learning data. This sounds, to me, a reasonable argument for innatism, that is, human "architecture" is biologically geared for language acquisition and contains some strong "guides" or constraints that reduce the hypothesis space of "possible human languages". I wonder if language models are able to find similar architectures that would allow learning with less data.

View on HN

The baby, unlike the LLM, isn't learning just from a stream of text. Anything the LLM knows about the world that a lot of that text describes has been inferred from the structure of the text, but a baby learning (say) English learns what a car is by having someone point at a car and say "car", etc. I think this means (1) that there's a lot more than those 30M tokens of actual input and (2) the input is better structured for teaching the baby about the world as well as the language.

Of course, you are totally correct about the baby not being limited to text input. It would be interesting to see in the future whether building "world models" through multimodal input would help scaffolding language ability with less data.

I wonder if it isn't 100x more effective for the baby that they don't learn from language per se, they learn from direct feedback.

They try to duplicate/imitate others, but generally they get immediate feedback (to the point where it's becomes annoying) about exactly what they're doing/alternatives/...

I've always found that this works, but adults often don't like it. Learning from direct feedback, having someone immediately check your work, so to speak, helps enormously. Even if it's in text (I learned C++ on IRC and mailing lists, for example). The books help, but you never have any direction.

How did you come up with "less data"? Video and audio are much larger then text even when compressed. You can put whole Dune book original series in space of a single MP3 song, and the entire text of English Wikipedia in a single high quality movie.

I'm talking about linguistic information here, because I'm interested in linguistics and language learning.

Humans abstract phonemes from sound, and morphemes/lexical items from phonemes. Once we have a rough measure of how many words of input the toddlers get, we can roughly equate the amounts of _linguistic_ data that goes into acquiring linguistic capabilities.

While GPT gets tokens as input, from the lexical lexical level onward, there can be an argument that we can roughly estimate the volume of the _linguistic_ data that goes in to the system regardless of earlier decoding stages.

I don't claim that there are a lot of other kinds of data feeding into a human brain, and the modality is totally different. (Multimodal-interactive vs. "predict the next token") GPT, therefore, is at disadvantage in that it needs to learn stuff – other than linguistic stuff – _only_ from linguistic input, whereas a human baby can use world model gained through other modalities as a scaffolding.

But for acquiring linguistic capabilities, auditory and visual data seems hardly relevant as input data, other than helping building that the scaffolding that can help "top-down" understanding.

Tiny models can already learn language (as in grammar and syntax) from very little very quickly.

It's the semantics that separates LLMs from CharRNN.

That's interesting! You mention CharRNN. Any links? I find it especially interesting to test whether models that know syntax, know superficially human language -like syntax, or actually syntax that conforms to the deep (syntactic) structure of human languages.

https://github.com/karpathy/char-rnn

The LLM is also trained to predict next token given a sequence of "real" tokens. This means that any case where the LLM predicts tokens that don't look like the "real" token string is Out of Domain.

Children learn in an iterative manner generating language, self-evaluating their own language, given validations from parents, and interactions with others. We've also cultivated a curriculum learning approach to make sure that students are always learning language.

We really don't know how small of a corpus is required for an LLM to learn. Perhaps there is an optimal curriculum which could be generated that demonstrates progressively more complex language and knowledge tasks to keep gradients large throughout training. Or a larger LLM could generate new training examples based on the cross-entropy loss of a given batch.

Won't we get the same if we train LLMs on video that has people speaking but things seen that might be related? Training data is easy to come by with cameras (although privacy concerns have to be addressed, or might prevent this from happening at all).

Surely someone has thought of this already.

Also, a Baby is extending the learning set with iterated synthetic dreaming material :)

No one ever brings up the role of unsupervised learning for early children. Children don't have labels and only limited capabilities to "self-supervise" until they are toddlers. The overwhelming majority of information that they take as input is "clustered" or learned in an unsupervised fashion. Baby learns that apple and orange are different objects before it knows anything else, like names or properties of those objects.

Babies take in far more data (but not text!) in an information theoretic sense than ChatGPT does. This is why I think these ideas that LLMs take far too much data for them to get good is sort of not true

The pre-training phase of GPT for next work prediction is essentially that - dump all text you can find and let the model figure out the relationship between tokens/words.

I agree though - what’s missing in neural architectures is advanced clustering capabilities purely from observing without any labels.

Then the labeling phase is cheap and doesn’t require as much data.

It's an apples and oranges comparison due to different training sets (as well as model sizes), but it seems generally their tests are indicating Phi-2 performing (in their specialized domains) at level of a more general model 5-10x it's size, not 100x.

As far as training data, it seems it's more the quality/focus of the data than the quantity. They refer to "textbook quality data", unlike the LLMs like GPT-3 that are trained on web scrapes etc.

I really wish people would stop comparing LLMs to human learning. They're really not equivalent, nowhere nearly as similar - I understand that LLMs are pretty impressive, but it's really time to stop anthropomorphizing them.

Why though? It's interesting. I'm not claiming it's equivalent or even remotely similar. (So I agree with your point about _anthropomorphising_) But as a former linguist, in first time history, we witness a system other than humans that exhibits language-like behaviour. Studying these system by no means tells us anything about how _humans_ learn language. But it gives us ideas and viewpoints of what's possible in the "systems-learning-languages" space.

> Phi-2: trained with 1400B tokens.

Where did you get the information. Not doubting you, just I couldn't find it.

It says so on the website: "Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding."

I wish they shared their 1.4T dataset.

I love that the RedPajama dataset is openly shared on HuggingFace.

Phi 1 was trained on 7B tokens. This is quite a jump then.

> a human baby learns a language with around 30M "token-equivalents" of learning data

I wonder if there’s a standard data set that’s reasonably like that? A common crawl of baby books and parents speech commonly heard by babies...

There is a wearable system for researching infants' linguistic environment called "The Language ENvironment Analysis system (LENA)". This review collects and analyses available data[1]. I just took an average AWC (adult word count) per day, and roughly calculated how much linguistic input infants get during their first 4 years. Of course, it's just a kind of a Fermi estimate, but the scale should be about right: 10000 times less input than GPT-3.

It would be an interesting challenge to create a "baby-like" dataset. I guess a system like this could help collecting it.

[1]: https://pubmed.ncbi.nlm.nih.gov/28824021/

TinyStories is pretty close (but synthetic). It’s been used to train a few tiny models.

Reply on news.ycombinator.com