Hacker News Reader

> a human baby learns a language with around 30M "token-equivalents" of learning data

I wonder if there’s a standard data set that’s reasonably like that? A common crawl of baby books and parents speech commonly heard by babies...

View on HN

There is a wearable system for researching infants' linguistic environment called "The Language ENvironment Analysis system (LENA)". This review collects and analyses available data[1]. I just took an average AWC (adult word count) per day, and roughly calculated how much linguistic input infants get during their first 4 years. Of course, it's just a kind of a Fermi estimate, but the scale should be about right: 10000 times less input than GPT-3.

It would be an interesting challenge to create a "baby-like" dataset. I guess a system like this could help collecting it.

[1]: https://pubmed.ncbi.nlm.nih.gov/28824021/

TinyStories is pretty close (but synthetic). It’s been used to train a few tiny models.

Reply on news.ycombinator.com