The Secret Sauce behind 100K context window in LLMs: all tricks in one place

I tend to do this with GPT-4 even on the context window in default ChatGPT (or more often I bookend it with instructions). I find it pays off at even 1000 tokens.

I've found Claude to be better than GPT4 at creative writing and explanations, while GPT4 seems to be better at logic-puzzlish stuff.

Maybe because it's basically impossible to get access to it right now…

Anthropic is a relatively new startup that probably has 3) & 4) from their history at OpenAI. But I don't see how they could have 1) & 2).

This would be similar with humans if everything was given verbally.

The huggingface transformers library exposes this as past_key_values, here's an example on GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#tran...

Scaling laws have resulted in GPT-4 and they keep going.

The design space for LLMs is bigger than just transformer architectures. It includes long prompts, retrieval, chains, data quality, compression, etc, which potentially stack on top of HW acceleration.

These are the worst models that we will see over the next decade. It’s going to be wild.

> Conditional computation avoids applying all model parameters to all tokens from the input sequence. [CoLT5] applies heavy computations only to the most important tokens and processes the rest of the tokens with a lighter version of layers. It will speed up both training and inference.

[CoLT5]: https://arxiv.org/abs/2303.094752

Programming, primarily. Code takes many more tokens per kilobyte of text than written English. So even quite short blocks of code eat up a lot of tokens.

The current AIs can do trivial, generic things using popular libraries. None can really help make changes in a large proprietary codebase where the prerequisite knowledge is the structure, design, and APIs of the private code.

With 100K token windows, a model could be given entire database schemas, or reams of interface definitions, Rest API schemas, or whatever, and then make edits based on that context.

It wouldn't even matter if it was slower than human, as long as it was cheaper.

Look at it this way: An 8-GPU NVIDIA DGX server is what, $400K to purchase at retail pricing? That would be "good enough" to run really beefy LLMs. If you use that server for about 3 years, then even factoring in all ancillary costs, that's about $13/hour to run. Or about 30 cents per minute.

So even if it takes some huge 100K token super-smart model a full minute to run through a prompt like "given the following reams of context, find the bugs in the given code below", then that's almost certainly worth it to most dev shops. Bugs can cost thousands of dollars to find and fix.

Merely finding half of the bugs for mere cents per function could yield staggering savings.

This seems like part of the conversation where advances in software are expected to deliver bigger gains than hardware for a while.

>At inference time, thanks to ALiBi, MPT-7B-StoryWriter-65k+ can extrapolate even beyond 65k tokens. We demonstrate generations as long as 84k tokens on a single node of 8 A100-80GB

https://huggingface.co/mosaicml/mpt-7b-storywriter

Edit: here is a snip of the interview - https://share.snipd.com/snip/93d96e73-9841-4361-bd82-ae086e1...

We need benchmarks for tasks that requiere large context sizes (like recalling facts and understanding them in long stories). I'm sure OpenAI have internal benchmarks for these tasks.

Similarly, your computer is not physically connected to mine but I'm still able to read your comment.

Then even with all that debugging the model, and making architecture choices is a whole other thing he barely covers. Would love a good course on that.

If you learn the nuts and bolts you can point to any part of the transformer model and describe what numbers and operations are happening in the forward and backward pass. That sort of is the reality. Reading other people’s fuzzy explanations is probably like understanding quantum mechanics by reading quanta (you get the layman example but don’t really understand… btw I know little about QM!)

I needed to hop on other materials to exist. I van understand some of this but not all and running it and trying things out… probably not yet.

Well never for 100k context size as I don’t want to sell my house to pay for compute :-).

…but you will be disappointed if you expect/hope to be able to recreate or modify things yourself.

You need a massive (multi TB) dataset of high quality data, and an array of 80GB graphics cards.

The barrier to entry isn’t knowledge; it’s money.

(So learning, yes. Try doing the fastai course. Change settings? No. Not unless you have a couple of million $$ in cloud credits to burn)

Don't listen to people telling you to learn the ML theory, if you don't know it, learn the functionality of the program. The one I recommended is one you can run on a normal computer.

Sometimes I wish there was a way to tell our browsers "I really don't care about SSL on this page, honestly, and I'm qualified to tell when it matters."

The main way that long contexts are useful is via the key query "attention" mechanism, since the key may be matched at any position in the context. This allows something at the end of the context to refer to something at the beginning (or anyplace else).

The ALiBi stuff just makes sense to me. I don't understand why the Positional Sinusoidal Encoding was used initially. I assume there were good reasons for it but I haven't seen an explanation, (pointers to one appreciated).

The OP doesn’t explain anything. It just vaguely talks about a few things that might break when scaling context. But that means nothing.

Take for example sinusoidal embeddings they talk about. Of course it breaks for large contexts but in no one uses it. GPT uses learned positional embeddings so the entire section is irrelevant.

Copy this for pretty much everything else.

Being an expert in a field has never been this exhausting