Comment by andreyk | Hacker News Reader

Comment by andreyk | Hacker News Reader

Parent Full thread

weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?"

It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now.

Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be the paper to pay attention to. Google already has a 1.2T model with 64 experts, while Microsoft Bing’s modes are different mixes accordingly"

There is also this linked tweet https://twitter.com/LiamFedus/status/1536791574612303872 - "They are all related to Switch-Transformers and MoE. Of the 3 people on Twitter, 2 joined OpenAI. Could be related, could be unrelated"

Which links to this tweet: "Today we're releasing all Switch Transformer models in T5X/JAX, including the 1.6T param Switch-C and the 395B param Switch-XXL models. Pleased to have these open-sourced!"

Anyway... remember not to just read the headlines, they can be misleading.

More Efficient In-Context Learning with Generalist Language Model (GLaM) on the Google blog (2021): https://archive.is/cuyW0

Btw, MoE is one of the 5 other ways Google thinks LLMs can be more efficient: https://archive.is/2XMvh

(OP here) - yeah i know, but i also know how AI twitter works so I put both the headline and the caveats. i always hope to elevate the level of discourse by raising the relevant facts to those at my level/a little bit behind me in terms of understanding. think theres always a fine balance between getting deep/technical/precise and getting attention and you have to thread the needle in a way that feels authentic to you to do this "job"

ultimately my goal is to Learn in Public and demonstrate to experts that spending their time teaching/sharing with me is a good use of time because i will augment/amplify/simplify their message.

(pls give the full podcast a listen/read/watch, George went deep on tinygrad/tinybox/tinycorp and there's lots there he IS the authority on, and people are overly fixated on the GPT4 rumor https://www.latent.space/p/geohot#details )

Makes it worse, since you have this understanding and still went with this explanation that it's 1.7 trillion patients

1.7 trillion patients? We're going to need more hospital beds ...

Oh, is that how Bing's image descriptions work? If you upload an image and ask for a description, it'll give a really good one. But if you ask it where it got it from, it says it got it from another algorithm. So is it using another expert AI to describe, and then feed it into GPT4 for conversion into language?

Pretty sure this is how the human brain works. If we add models that can transform information into updates to existing models, then add a model that can process streams of information and determine which models to update with the information (and can also apply that process to itself), we start to approach a primitive brain, no?

> Pretty sure this is how the human brain works

How we thought it works in the 1960's yes.

How do we think it works currently?

For a moment I thought you were Andrej Karapathy!

Title should be: "GPT4 is 220B"

Factual, and surprising to many.

Reply on news.ycombinator.com