Hacker News Reader

Comment by breatheoften | Hacker News Reader

Who is commenting on this and what will they say??!

1-2 trillion tokens of training for gptx class models

It would take 20 years for a human to read that many tokens at 8 hrs of reading per day (although i'm pretty unclear on whether that's unique token sequences or randomly selected token pairs -- feels like a big difference between the two!!)

We are all here now -- who thinks we are anywhere of note together?

Personally I think that models that do not experience incremental change remain an entirely separate class of intelligence from those which evolve continuously -- but I am open to being persuaded otherwise (certainly I never thought making gpt2 bigger would be as impressive as gpt4) ...

View on HN

Video from a pair of eyes is many terabytes per year though. If you compress it with best codecs. Raw video would be closer to 1PB per year.

I think people overestimate the data efficiency of the biological neural networks w.r.t. transformer models.

People with bad eyesight are not (to the best of my knowledge) less intelligent than those with perfect vision. We could probably supply video at a much lower resolution without significantly affecting the outcome.

True, but it's not like sound and/or touch are low bandwidth channels either.

The power effiency is pretty brutal though, in favour of bio-gelpacks.

How can you be sure about power efficiency, if you don't know how data efficiency compares? It is low power. But so is a phone running llama.cpp

You must be pretty clever to write that, and you are human right? Your data effiency and power effiency seems top notch to me.

Maybe I am, but how many are out there who aren't? And how many smartphones can run alpaca?

>1-2 trillion tokens of training for gptx class models

>It would take 20 years for a human to read that many tokens at 8 hrs of reading per day

Normal reading is ~200 words per minute.

That's 12,000 words per hour.

That's 288,000 words per day (reading for 24 hours straight).

That's 105,120,000 words per year.

That's 2,102,400,000 words per 20 years.

A trillion is 1,000,000,000,000.

Plus almost no one is reading academic papers at 200 words per minute. So for the level of material it could mean that number is halved per 20 years.

And, as humans, we don't only get "tokens" from what we read, but from our entire sensorium.

Reply on news.ycombinator.com