100K Context Windows
anthropic.com
anthropic.com
This sort of needle-in-a-haystack retrieval is definitely impressive, and it makes a lot more sense to achieve this in-context rather than trying to use a vector database if you can afford it.
I'm curious, though, whether there are diminishing returns in terms of how much analysis the model can do over those 100k tokens in a single forward pass. A human reading modified-Gatsby might eventually spot the altered line, but they'd also be able to answer questions about the overarching plot and themes of the novel, including ones that cannot be deduced from just a small number of salient snippets.
I'd be curious to see whether huge-context models are also able to do this, or if they start to have trouble when the bottleneck becomes reasoning capacity rather than input length. I feel like it's hard to predict one way or the other without trying it, just because LLMs have already demonstrated a lot of surprising powers.
I might be wrong, but the point isn't comparing a modified The Great Gatsby to the original one. Of course that's not impressive and it's an easy thing to do.
The point of the exercise is supposed to be[1] that the model has the entire novel as context / prompt and so can identify (within that context), whether a paragraph is out of place. That is impressive and I wouldn't know how to find that programmatically (would you have a list of "modern" words to check? But maybe the out of place thing is hidden in the meaning and there's no out of place, modern word).
[1] I say supposed to be because the great Gatsby is in the training sample and so maybe there is a sense in which the model "contains" the original text and in some way is doing "just" a comparison. A better test would be try with a novel or document that the model hasn't seen...or at least something not as famous as the great Gatsby.
Most of these AI start failing pretty hard when you ask it to do the same task on something completely novel to it (like a company document). Sometimes they'll get it right. Other times, they'll spit out gibberish that's clearly some generic answer.
I didn't need to have seen this book before to know this wasn't in the original 1980s text.
Similarly, if I were reading the Great Gatsby for the first time, and it identified a character as a software engineer, I would notice.
Interesting part is if it can deduce something that's out of place within a huge document.
This is as meaningful as saying most of the hominids can't count. You can't usefully generalize AI models with the rate of change that exists right now. Any statements/comparisons about AI has to contain specific models and versions, otherwise it's increasingly irrelevant noise.
Vector dbs work by fetching segments that are similar in topics to the question, so like "Where did <Character> go after <thing>" will retrieve segments with locations & the character & maybe talking about <thing> as a recent event.
Your question has no similarity with the segments required in any way; & it's not the segments that are wrong it's the way they relate to the rest of the story
Which makes me wonder if the opposite, but more laborious approach might work - request it identify all characters and plot themes, then request summaries of each. You'd have to review the summaries for holes. Lotsa work, but still maybe quicker than re-reading everything yourself?
But I've started experimenting with the second part, of sorts, not to find plot holes but to have it create character sheets for my series of novels for my own reference.
Basically have it maintain a sheet and feed it chunks of one or more chapters and asking it to output an a new sheet augmented with the new details.
With a 100K context window I might just test doing it over while novels or much larger chunks of one.
I feel this is mostly a prompting issue. Specifically GPT-4 shows surprising ability to abstract to some degree and work with high-level concepts, but it seems that, quite often, you need to guide it towards the right "mode" of thinking.
It's like dealing with a 4 year old kid. They may be perfectly able to do something you ask them, but will keep doing something else, until you give them specific hints, several times, in different ways.
In particular, there are 0 mentions of the phrase "machine learning" in The Great Gatsby, so adding one sentence that introduces the phrase should be easy for self-attention to pick out.
It'll be tough to find good benchmarks on long context windows. A human cannot label using 100k tokens of context.
A better comparison would be if it can pick out any differences that can't be picked out by more traditional and simple algorithms.
My immediate thought as well was '... Yeah, well vimdiff can do that in milliseconds rather than 22 seconds' - but that's obviously missing the point entirely. Of course, we need to tell people to use the right tool for the job, and that will be more and more important to remind people of now.
However, it's pretty clear that the reason they used this task is to give something simple to understand what was done in a very simple example. Of course it can do more semantic understanding related tasks, because that's what the model does.
So, without looking at the details we all know that it can summarize full books, give thematic differences between two books, write what a book may be like if a character switch from one book to another is done, etc.
If it doesn't do these things (not just badly, but can't at all) I would be surprised. If it does them, but badly, I wouldn't be surprised, but it also wouldn't be mind bending to see it do better than any human at the task as well.
USER: It's a stochastic parrot.
GPT: I know you are, so what am I?
(I think I’ve got to read up on how transformers actually work.)
I see 6 ways to improve foundation LLMs other than cost. If your product is best at one of the below, and has parity at the other 5 items, then customers will switch. I'm currently using GPT-4-8k. I regularly run into the context limit. If Claude-100K is close enough on "intelligence" then I will switch.
Six Dimensions to Compare Foundation LLMs:
1. Smarter models
2. Larger context windows
3. More input and output modes
4. Lower time to first response token and to full response
5. Easier prompting
6. Integrations
GPT4-32K costs ~$2 if you end up using the full 32K tokens, so if you're doing any chaining or back-and-forth it can get expensive very quickly.
https://azure.microsoft.com/en-us/pricing/details/cognitive-...
I'd add open source to the list, which neither "open"AI or this is.
Businesses will certainly care about cost, but just as important will be:
- Customization and fine-tuning capabilities (also 'white labeling' where appropriate)
- Integrations (with 3rd party and in-house services & data stores)
- SLA & performance concerns
- Safety features
Open Source AI will have a place, but may be more towards personal-use and academic work. And it will certainly drive competition with the major players (OpenAI, Google, etc) and push them to innovate more which is starting to play out now.
Though those cloud platforms all have their own proprietary components most users are savvy enough to constrain and compartmentalize their use of them lest they find themselves having all their profits taken by a platform that knows it can set its prices arbitrarily. The cloud vs in-house adoption is what it is in large part because the cloud offerings are a commodity and a big part of them being a commodity is that much of the underlying software is free software.
OpenAI already upset a lot of (admittedly non-paying academic) users when they shut off access to the old Ada code model with only a few week's notice.
One would think the same in the 90s but yet, for some reason, Open Source prevailed and took over the world. I don't believe it was about cost, at least not only. In my career I had to evaluate many technical solutions and products and OSS was often objectively superior at several levels without taking account the cost.
The first really successful alternative to "Open"AI will:
* gather many talented developers
* will quickly become a de facto standard solution
* people will rapidly start developing a wide range of integrations for it
* everybody will be using it, including large orgs, because, well, it's open source
Problem again, is centralization of LLMs by either the governments (and they always act in your best interest, amirite?) and corporation, which Non-FOSS LLMs prevent.
Democratization of the models is the only way to actually prevent bad actors from doing bad things.
"But they'll then have access to it too" you say. Yes, they will, but given how many more people who will also have access to open LLMs we'd have tools to prevent actually malicious acts.
We provide a low code data transformation product (prophecy.io), and we’ll never close sales at any volume, if we have a to get an MSA that approves this. Might get easier if we become large :)
OSS AI will open up more diverse and useful services than the first-party offerings from relatively risk averse major vendors, which customers *will" care about.
This is why cloud services are so popular. They’re easy and they don’t cost the decision makers personally.
is it that important to open source models that can only run on hardware worth tens of thousand of dollars?
who does that benefit besides their competitors and nefarious actors?
I've been trying to run one of the largest models for a while, unless 30,000$ falls in my hand I'll probably never be able to run the current SOTA
We've already seen big advancements in tools to run them on lesser hardware. It wouldn't surprise me if we see some big advancements in the hardware to run them over the next few years, currently they are mostly being run of graphics processors that aren't optimised for the task.
Yes, because as we've seen with other open source AI models, it's often possible for people to fork code and modify it in such a way that it runs on consumer grade hardware.
I can for example, afford the hardware worth tens of thousands of dollars. I don't want to, but I can if I needed to. Does that automagically make me their competitor or a bad actor?
But for commercial usecases, open source is very relevant for privacy reasons as many enterprises have strict policy not to share data with third party. Also it could be a lot cheaper for bulk inference or to have a small model for particular task.
However, in many applications there is a limit on how intelligent you need the LLM to be. I have found I am able to fall back to the cheaper and faster GPT-3.5 to do the grunt work of forming text blobs into structured json within a chain involving GPT-4 for higher-level functions.
* Text summary/compression
* Creative writing (fiction/lyrics/stylization)
* Text comparison
* Question-answering
* Logical reasoning/sequencing ("given these tools and this scenario, how would you perform this task")
IMO, for stuff like text comparison and question-answering, some combo of speed/cost/context-size could make up for a lot, even if they do "worse" versions of stuff just that's too slow or expensive or context-limited in a different model.
Intelligence is the most important dimension by far, perhaps an order of magnitude or more above the second item on the list.
When someone says they want it available they mean running on their own device.
This is hackernews, nearly everyone on this site should have their own self hosted LLM running on a computer/server or device they have at their house.
Relying on 'the cloud' for everything makes us worse developers in just about every imaginable way, creates a ton of completely unnecessary and complicated source code, and creates far too many calls to the internet which are unnecessary. Using local hard drives for example is thousands of times faster than using cloud data storage, and we should take advantage of that in the software we write. So instead of making billions of calls to download a terabyte database query-by-query (seen this 'industry-standard' far too many times), maybe make one call and build it locally. This is effectively the same problem in LLMs/ML in general, and the same incredible stupidity is being followed. Download the model once, run your queries locally. That's the solution we should be using.
FYI the waitlist form submits a regular POST request so it'll reload the main page instead of closing the modal dialog. I opened network monitor with preserved logs to double check that I made it on the list :facepalm:
If I could train a useful model, on my own data, in a reasonable time
I would want to have a CI-training pipeline to always have my models up to date
If your main concern is question answering or summarization or code completion there are plenty of ways to do that now. If you really require the advanced emergent properties of LLMs, you'll have to work with a company that can afford to train a transformer on the Entire Internet.
Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...
What would the quality of the model be, compared to what Karpathy uses on his video “let’s build gpt from scratch”?
In that video, he builds a decoder-only transformer model, that learns Shakespeare from 1MB of data and trains in 15min
I'm not sure what the analogous term is for a similar process on LLMs, but that will be huge when there is a service for it.
If you want for example to train the model to learn to use a very large API, or access the knowledge in a whole book, it might need fine-tuning.
E.g., were you trying to distinguish an object vs nothing, a bicycle vs a fish, a bird vs a squirrel, or two different species of songbird at a feeder?
How much would the training requirements increase or decrease moving up or down that scale?
Then use GPT-2
At least, for people who need large context windows, they would not be the first choice anymore.
The ChatGPT equivalent is 3x speed and was somewhere between ChatGPT and GPT4 on my TriviaQA benchmark replication I did
Couple tweets with data and examples. Note they’re from 8 weeks ago, I know Claude got a version bump, GPT3.5/4 accessible via API seem the same.
[1] brief and graphical summary of speed and TriviaQA https://twitter.com/jpohhhh/status/1638362982131351552?s=46&...
[2] ad hoc side by sides https://twitter.com/jpohhhh/status/1637316127314305024?s=46&...
GPT3.5 just got an update a few days ago that resulted in a pretty good improvement on its creativity. I saved some sample outputs from the previous March model, and for the same prompt the difference is quite dramatic. Prose is much less formulaic overall.
It can't even solve simplified versions of problems it had zero issues with just a week ago.
Random Q, I don’t use the ChatGPT front end much past month or two, used it a week back and it seemed blazingly faster than my integration: do you have a sense of if it got faster too?
Impressions:
Bad enough compared to GPT-4 that I default to GPT-4. I think if I had api access I’d use it instead, right now it requires more coaxing, and using Poe.
I did find “long-term” chats went better, was really impressed with how it held up when I was asking it a nasty problem that was hard to even communicate verbally. Wrong at first, but as I conversed it was a real conversation.
GPT4 seems to circle a lower optima. My academic guess it’s what Anthropic calls it “sycophancy” in its papers, tldr GPT really really wants to do more like what’s in the context, so the longer the conversation with initial errors goes, it’s actually harder to talk it out of the errors.
Let's say I have a book and I want to ask multiple questions about it. Every query will pay the price of the book's text. It would be awesome if I could "index" the book once, i.e. pay for the context once, and then ask multiple questions.
Or anything where the question answer isn't 'close' to the words used in the question?
How well does this work vs giving it the whole thing as a prompt?
I assume worse but I'm not sure how this approach compares to giving it the full thing in the prompt or splitting it into N sections and running on each and then summarizing.
Background: https://summarity.com/hyde
Demo: https://youtu.be/elNrRU12xRc?t=1550 (or try it on findsight.ai and compare results of the "answer" vs the "state" filter)
For even deeper retrieval consider late interaction models such as ColBERT
Does the embedding structure somehow expose the themes? And if so, is it more the embeddings that are answering the question by how it groups things?
The problem is that humans have continuous information retrieval and storage where the current crop of embedding systems are static and mostly one shot.
Just a few manufacturers hold the effective cartel monopoly on LLM acceleration and you best bet they will charge out the ass for it.
Ironically, that's an example I like to list as "pure sci-fi fantasy, divorced from economic reality."
The total cost of the iron ore that goes into making a new a car is about $200-$300 dollars, depending on various factors (size of the car, ore spot price, etc...).
Even if -- magically -- asteroid mining made not just "iron ore", but specifically the steel alloy used for car bodies literally free, new cars costing $30,000 would now cost... $29,700.
You can save more by skipping the optional coffee cup warmer, or whatever.
In reality: 90% of iron and steel is recycled, and asteroid mining is not magic.
It's stuff like platinum and germanium that makes asteroid mining potentially interesting.
Barring a Chinese invasion of Taiwan, these APIs will halve in price over the next year.
On the other hand, for all peoples worrying about China, they are pretty restrained given the enormous turmoil the invasion would cause. If they wanted to break the USA, now would probably be the time to do it?
It makes the economics slightly trickier.
Perhaps it's because under the hood there's additional safety analysis/candidate generate that is resource intensive?
If you do the math of how much memory bandwidth is required by a forward pass vs. how much compute, you'll see that inference is entirely limited by memory bandwidth and will use compute resources very inefficiently. In contrast, input processing is able to fully use the available compute.
Of course, there are ways to mitigate this problem, like processing multiple token streams in parallel, but the fundamental problem remains.
[1] I'm not sure if these huge context lengths are achieved the same way (i.e. a single input vector of length N) but given the cost is constant for input I would assume the resource usage is too.
Otherwise, it might make sense to have a separate routine which compresses the context as efficiently as possible. Auto encoder?
MosaicML did say something about MPT-7B-StoryWriter-65k+: https://www.mosaicml.com/blog/mpt-7b. They are using ALiBi (Attention with Linear Biases): https://arxiv.org/abs/2108.12409.
I think OpenAI and Anthropic are using ALiBi or their own proprietary advances. Both seem possible.
A second example is the analysis of long documents. Today, hacks like chunking and HyDE enable us to ask questions about a long document or a corpus of documents. But is far superior if the model can ingest the whole document and apply attention to everything, rather than just one chunk at a time. Chunking effectively means that the model is limited to drawing conclusions from one chunk at a time and cannot synthesize useful responses relating to the entire document.
Given that the conventional cost of training attention layers grows quadratically with the number of tokens I think Anthropic is doing some kind of approximation here. Not clear at all that you would get the same results as vanilla attention.
Also if you are training on a database of code.
This is incredibly fast progress on large contexts and I would like to see if they are actually attending equally as well to all of the information or there is some sparse approximation leading to intelligence/reasoning degradation.
Claude by Anthropic has more favourable responses then ChatGPT
"Given that Beth is Sue's sister and Arnold is Sue's father and Beth Junior is Beth's Daughter and Jacob is Arnold's Great Grandfather, who is Jacob to Beth Junior?"
It's still below GPT4, but it is closer to 4 than 3.5
Good luck waiting for it.
If you want me to test something for you, lmk and I’ll send it through the api.
I’m in an adjacent industry and this is what I’m looking forward to.
- Price per token doesn't change compared to regular models
- Existing api users have access now by setting the `model` param to "claude-v1-100k" or "claude-instant-v1-100k"
- New customers can join waitlist at anthropic.com/product
[1]: https://cdn2.assets-servd.host/anthropic-website/production/... [2]: https://console.anthropic.com/docs/api/reference#parameters
Although with flash attention, who knows if marginal cost scales that consistently.
https://twitter.com/AnthropicAI/status/1656743460769259521?s...
The new model are a different model identified that's not listed in the pricing doc, although it sounds like the intent may be to replace the base from looking at the API docs: https://console.anthropic.com/docs/api/reference#-v1-complet...
You can go through Poe.com
In fact, Anthropic explicitly discusses putting words into the assistants‘ mouth to be able to shape it’s responses and make it better align with the desired output.
This is cool but does it also work the other way around? Generate a book's worth of content based on a single prompt?
This huge context window is awesome though, I'm trying to use LLMs to do small town social interaction simulations, with output in a structured format. Finding ways to compress existing state and pass it around, so the LLM knows the current state of what people in the town did for a given day is hard with a tiny token limit!
[1] For my use cases, early instructions tend to be describing a DSL syntax for responses, if I add too much info after the instructions, the response syntax starts getting wonky!
As she kept asking for more, I prompted "great, do another one" and eventually my original instruction fell out of the context window. It continued to generate a children's story, but with no more blanks.
There is no good way to tell it "this isn't a conversation, just repeat the answer to the initial prompt again".
The solution is to just re-paste the initial prompt in each time, but still it isn't ideal. There isn't a good way to tell chatgpt "you can throw away all the context after the initial prompt and up until now".
Of course the entire point of ChatGPT is that it maintains a conversation thread, so I get why they don't fix up this edge case.
My problem is more of, I give ChatGPT some complicated instructions, and it'll start forgetting the early on instructions long before any token limit is reached.
So for example, if early on I ask for certain tokens to be returned in parens, well my initial prompt is too long, it'll forget the parens thing and start returning tokens without the surrounding (), which then breaks my parser!
Exciting to see competition across LLMs for increasing context window size.
I can't find updated pricing anywhere. Previous prices are here: https://cdn2.assets-servd.host/anthropic-website/production/... but don't seem to be embedded directly on the Anthropic website. I tried messing with the URL (apr -> may/jun) but 404'ed.
Maybe. I think the debate is going to continue about prompt optimization vs. context window size.
A while ago, I had a rather interesting conversation with GPT-3.5 about forgetting things. Knowing what to forget, or delete from the prompt, may be just as important as what to put in it.
Putting the kitchen sink into the prompt probably isn't going to help much, past a certain point and it may be putting certain things in there based on time and context is a better strategy.
Holler if you want help with it. I have some more code I'm adding to it this week.
Now I tried to request access again on their form and it just redirected. Can't even tell if that worked.
Does anyone know if this can program as well as GPT-4? Because if so then the larger context window is a big improvement.
Also having a massive context length is not necessarily a good thing from perspective of cost. It also doesn't work great with a chatbot as you will have to feed the same 100k worth context back into the chatbot for every question which will turn out to be very expensive. At some point you will have to discard some parts of the context to be specific to the question being asked and that is where vector embeddings come into play. For one off research/Q&A 100k limit works great!
I wonder if Claude-100k could be used to ingest this entire thread and then answer questions based on it, or summarize or identify the pros/cons of certain aspects of Claude, large context windows, vector embeddings, etc.
If contexts keep growing, the landscape of LLM application engineering will as well
https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula...
[0] https://wanderinginn.neocities.org/statistics
[1] https://www.reddit.com/r/Parahumans/comments/rz8ogt/wildbows...
Also anthropic has a "google cloud partnership" basically they are hooked on cloud credits like openAI and Azure.
Product race? My understanding is they've been so concerned with safety/harm that they've been slow to implement a lot of tools - then OpenAI made an attempt at it anyway.
Google has generally been ahead from a research perspective though. And honestly it's going to be really sad if they just stop releasing papers outright - hopefully the release their previous gen stuff as they go :/
Google has a product issue, not an AI research one.
It's worse than GPT-3.5. Go see for yourself at bard.google.com, which is running on PaLM 2 everywhere but the EU as of yesterday.
> After lunch, I like to snack on a juicy and crisp apple to satisfy my sweet tooth.
> In the fall, many families enjoy going to apple orchards to pick their own apples and make homemade apple pies.
> The new MacBook Pro features a powerful M1 chip and a stunning Retina display, making it the perfect tool for creative professionals who work with Apple software.
I'm not sure that counts as 'great' though.
They invested $300m in Anthropic in late 2022: https://www.ft.com/content/583ead66-467c-4bd5-84d0-ed5df7b5b...
(Non-paywall: https://archive.is/Y5A9B)
Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a context width of 65k, but then give a bit more detail to say they are using ALiBi - a type of positional encoding that allows longer contexts at inference time than training (i.e beyond the real context width of the model).
For these types of "extended context" models to actually reason over inputs longer than the native context width of the model, I assume that there is indeed some sort of vector DB trickery - maybe paging thru the input to generate vector DB content, then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ?
Maybe someone from Anthropic or MosaicML could throw us a bone and give a bit more detail of how these are working !
Who knows about Anthropic.
A bit disappointing
What is the approach to increase the sequence length here?
Now we've gone from using ML to implement slow, unreliable databases, to using ML to implement slow, unreliable string comparison, I guess
I say this, because I'm not sure how all of this is really going to scale on GPUs. It feels like LLM's are just as magical as quantum computing.
I do not have a clue what you are talking about. What happened?