Hacker News Reader

Comment by causal | Hacker News Reader

Thanks for responding to that point - it's the one most on my mind.

So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context.

So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me.

Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.

View on HN

Yes but we can distinguish between embedding provided to customers, and internal embeddings. One is optimized for usage in certain types of applications, but the internal embeddings need to be optimized to support long contexts and are not constrained by the customer facing embeddings.

For example, with Open AI I believe it’s known that the internal dimension for Gpt3 was 12,288.

Are the same embeddings not used internally? I thought they were. Maybe I'm wrong about that.

Mistral uses a 1024 dimension embedding for 8K context. I think the point about trying to capture that rich of a context into a smaller number of dimensions still stands?

Yes definitely, you have a valid concern.

For long contexts this is a key consideration along with what self attention optimizations the model chooses to implement.

They don’t make this public, but we can infer they can’t be using full self attention pairs at 1,000,000 tokens because it scales quadratically and would take Terabytes of RAM.

There are different approaches like sparse attention, and the only way to really know how well their choices work is to test it.

So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me.

Will compute allow that number to go up? Or is that an optimal number?

Definitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.

Has it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.

I’m not aware that internal embedding dimension size has been made public for Gpt4 et al.

In general for models we know its trended upward, but for sure it’d be interesting to know what they’re using now.

Reply on news.ycombinator.com