Hacker News Reader

Parent Full thread

binarymax

GPT-4 was rumored to be trained on 13 trillion tokens. https://www.kdnuggets.com/2023/07/gpt4-details-leaked.html

Not sure where you’re getting the 45Gb number.

Also, Google doesn’t use GPT-4 for summaries. They use a custom version of their Gemini model family.

View on HN

The number of tokens trained on is separate from the model size.

Gemma 3 270M was trained on 6 trillion tokens but can be loaded into a few hundred million bytes of memory.

But yeah GPT-4 is certainly way bigger than 45GB.

It wouldn't matter if they trained on a quadrillion tokens, or another ten orders of magnitude. There's only so much information you call stuff into a given set of numbers.

But once again I am reminded, never make arguments based on information theory. Nobody understands it.

Reply on news.ycombinator.com