Overcoming the limits of current LLMs
seanpedersen.github.io
seanpedersen.github.io
Training only on factual data won't solve this.
Anyway, I can't help but feel saddened sometimes to see our talented people and investment resources being drawn in to developing these AI chatbots. These problems are solvable, but are we really making a better world by solving them?
When you ask yourself that question -- and you do ask yourself that, right? -- what's your answer?
LLM stuff is not in a different echelon
I am also saying LMs output should cite sources and give confidence scores (which reflects how much the output is in or out of the training distrtibution).
And I think looking to the training data for sources is a little silly - that’s the training data for intuitive language use, not true statements about the world. If you haven’t checked it out yet, two terms you’d love are “RAG” and “Manuel De Landa”
If so, building this could be quite complex depending on the domain. In the legal field even one simple word that is changed can have large consequences.
Manually curating "tidy", properly-licensed and verified datasets is immensely more difficult, expensive, and time-consuming than stealing whatever you can find on the open internet. Wolfram Alpha is one of the more successful attempts in that curation-based direction (using good-old-fashioned heuristic techniques instead of opaque ML models), and while it is very useful and contains a great deal of factual information, it does not conjure appealing fantasies of magical capabilities springing up from thin air and hands-off exponential improvement.
Arguably the bigger problem is that many of those datasets e.g. WSJ articles are proprietary and can be exclusively licensed like we've seen recently with OpenAI.
So we end up with in a situation where competition is simply not possible.
> So we end up with in a situation where competition is simply not possible.
Exactly, and Technofeudalism advances a little more into a new feud.
OpenAI is trying to create its moat by shoring up training data, probably attempting to not allow competitors to train on the same datasets they've been licencing, at least for a while. Training data is the only possible moat for LLMs, models seem to be advancing quite well between different companies but as mentioned here a tidy training dataset is the actual gold.
If you treat the web as a free-for-all and you scrape freely, you get sued by the content platforms for copyright or term of service violation.
If you license the content, you let the highest bidder get the content.
No matter what happens, capital wins.
Every tiny bit of text you write is a vote in the election of our future AI overlords.
This assumption is based on the mistaken assumption that science is about objective truth.
It is confusing the map for the territory. Scientific models are intended to be useful, not perfect.
Statistical learning, vs symbolic learning is about existential quantification vs universal quantification respectively.
All models are wrong some are useful, this applies to even the most unreasonably accurate versions like QFT and GR.
Spherical cows, no matter how useful are hotly debated outside of the didactic half truths of low level courses.
The corpus that the above seeks doesn't exist in academic circles, only in popular science where people don't see that practical, useful models are far more important that 'correct' ones.
Here are a few examples of statements that are not "factual" in the sense of being derivable from a universally coherent data set, and that nevertheless we would expect a useful intelligence to be able to generate:
"There is a region called Hobbiton where someone named Frodo Baggins lives."
"We'd like to announce that Mr. Ousted is transitioning from his role as CEO to an advisory position while he looks for a new challenge. We are grateful to Mr. Ousted for his contributions and will be sad to see him go."
"The earth is round."
"Nebraska is flat."
Yet every child seems to manage, when raised by a small village, over a period of about 18 years. I guess we just need to give these LLMs a little more love and attention.
"Coherent" is doing a lot of lifting here. All humans have highly flawed models, and we've been culturally conditioned to grade on a curve to hide the problem from ourselves.
At the end of the day, none of these theoretical techniques prevailed in the field of AI, and we ended up with, empirically successful, neural networks (and LLMs specifically). We know they model uncertainty but we have no clue how they do it conceptually, or whether they even have a coherent conception of uncertainty.
So I would pose that the problem isn't that we don't have the technology, but it's rather we don't understand what we want from it. I am yet to see a coherent theory of how humans manipulate the human language to express uncertainty that would encompass broad (if not all) range of how people use language. Without having that, you can't define what is a hallucination of an LLM. Maybe it's making a joke (some believe that point of the joke is to highlight a subtle logical error of some sort), because, you know, it read a lot of them and it concluded that's what humans do.
So AI eventually prevailed (over humans) in fields where we were able to precisely define the goal. But what is our goal vis-a-vis human language? What do we want from AI to answer to our prompts? I think we are stuck at the lack of definition of that.
Rather than just an imperfect technology as we have here.
Many people object to the term enshittification for foul-mouthing reasons but I think it covers it very well because the principle it covers is itself so very nasty. But that's not at all the case here.
The behavior of text models is similar enough that the wording stuck, and it's not all that bad.
They simply do not behave like humans of sound minds, and "hallucinations" conveys that in a way that "confabulations" or even "bullshit" does not. (Though "bullshit" isn't bad either.)
And I couldn't find a single one of my friends who hadn't experienced "phantom vibration syndrome".
Both I'd say are "Hallucinations", without any real negative connotation.
I think what calling the times they get things wrong hallucinations is largely an advertising trick. So that they can sort of fit the LLMs into how all IT is sometimes “wonky” and sell their fundamentally flawed technology more easily. I also think it works extremely well.
Edit: It retrospect, perhaps a better analogy would involve gasoline, as its explosive nature is what's being actively being exploited in normal use.
"Lastly, I want to reassure investors and members of the press that we take these concerns very seriously: The Ford Pinto-II will only contain only normal and stable gasoline, and not the rare and unusual burning kind, which is merely a temporary hurdle in this highly dynamic and explos--er--fast growing field."
It is the memory pathways leading them astray. It could be thought of a memory system that at certain point any longer can't be fully sure if whatever connections they have are from actually being trained or it or created accidentally.
I suppose so, in the sense that someone could simply be lying about pink elephants instead of seeing them. However it's hard to argue that the machine knows the "right" answer and is (intelligently?) deceiving us.
> It is the memory pathways leading them astray.
I don't think it's a "memory" issue as much as a "they don't operate the way we like to think they do" issue.
Suppose a human is asked to describe different paintings on the wall of an art gallery. Sometimes their statements appear valid and you nod along, and sometimes the statements are so wrong that it alarms you, because "this person is hallucinating."
Now consider how the entire situation is flipped by finding out one additional fact... They're actually totally blind.
Is it a lie? Is it a hallucination? Does it matter? Either way you must dramatically re-evaluate what their "good" outputs really mean and whether they can be used.
Like you ask me for a birthdate of some obscure political figure from history? I'm going to try to feel out what period in history the name might feel like to me and just make my best guess based on that, then say some random year and a birthdate. It just has the lowest odds of being beaten. Was I hallucinating? No, I was just trying to not get beaten.
This is transparently wrong. It gets so many things right in a response that the few things it gets wrong are tremendously frustrating. I think people underestimate how much correct "knowledge about the world" is expressed in a typical chat gpt response and focus only on the parts that are incorrect.
If it were wrong about _everything_ at rates no better than chance, we wouldn't even be having this conversation because nobody would be using them.
LLMs don't do it because they are out of their right mind. They do it because every single answer they say is invented caring only about form, and not correctness.
But yeah, that ship has already sailed.
Humans hallucinate. Programs have bugs.
It's inherent to how LLMs work and is expected although undesired behaviour.
But there's nothing at all different about what the model is doing between these cases -- the models are hallucinating all the time and have no ability to assess when they are hallucinating "right" or "wrong" or useful/non-useful output in any meaningful way.
: to fill in gaps in memory by fabrication
> In psychology, confabulation is a memory error consisting of the production of fabricated, distorted, or misinterpreted memories about oneself or the world.
It’s more about coming up with a plausible explanation in the absence of a readily-available one.
But "hallucination" was already (before LLMs) being used in a figurative sense, i.e. for abstract ideas that are made up out of nothing. The same is also true of other words that were originally visual, like "illusion" and "mirage".
Heck, I think detecting sarcasm would be an easier goal, and still tricky.
Well that's actually good news. With a large enough labelled dataset of actually-sound and fallacious text with similar grammatical features you should be able to train a discriminator to distinguish between them using some other metric. Good luck with getting that data set though.
Not when the better metrics are likely alien/incompatible to the discriminator's core algorithm!
Then it's rather inconvenient news, because it means you have to develop something separate and novel.
As the other poster already mentioned, if we can't even get them to reliably count how many objects are being referred to, how do you expect them to also handle logical syllogisms?
Remember NP is equivalent to second order logic with existential quantified. E.g. for any X there exists a Y
And that only gets you to truthy Trues, co-NP is another problem.
ATP is hard, and while we get lucky with some constrained problems like type inference, which is pathological in its runtime, but decidable, Pressburger arithmetic is the highest form we know is decidable.
It is a large reason CS uses science and falsification vs proofs.
Gödel and the difference between Symantec and syntactic completeness is another rat hole.
the prompt interfaces + smartphone apps were (from the beginning), and are ongoing training for the next iteration, they provide massive RLHF for further improvements in already quite RLHFed advanced models.
Whatever tokens they're extracting from all the interactions, the most valuable are those from metadata, like "correct answer in one shot", or "correct answer in three shots".
The inputs and potentially the outputs can be gibberish, but the metadata can be mostly accurate given some implicit/explicit (the tumbs up, the "thanks" answers from users, maybe), human feedback.
The RLHF refinement extracted from getting the models face the entire human population for to be continuously, 24x7x365, prompted in all languages, about all the topics interesting for the human society, must be incredible. If you just can extract a single percentage of definitely "correct answers" from the total prompts answered, it should be massive compared to just a few thousands of QA dedicated RLHF people working on the models in the initial iterations of training.
That was GPT2,3,4, initial iterations of the training. Having the models been evolved to more powerful (mathematical) entities, you can use them to train the next models. Like is almost certainly happening.
My bet is that one of two
- The scaling thing is working spectacularly, they've seen linear improvement in blue/green deployments across the world + realtime RLHF, and maybe it is going a bit slow, but the improvements justify just a bit more waiting to get trained a more powerful,refined model, incredible more better answers from even the previous datasets used (now more deeply inquired by the new models and the new massive RLHF data), if in a year they have a 20x GPT4, Claude, Gemini, whatever, they could be "jumping" to the next 40x GPT4, Claude, Gemini, a lot faster, if they have the most popular, prompted model in the market (in the world).
- The scaling stuff already sunk, they have seen the numbers and it doesn't add by now, or they've seen disminished returns coming. This is being firmly denied by anyone on the record or off the record.
import time
import random
SESSION_DURATION = 50 * 60
start_time = time.time()
while True:
current_time = time.time()
elapsed_time = current_time - start_time
if elapsed_time >= SESSION_DURATION:
print("Our time is up. That will be $150. See you next week!")
break
_ = input("")
print(random.choice(["Mmm hmm", "Tell me more", "How does that make you feel?"]))
time.sleep(1)
You can get good results by combining different models in chat, or even the same model with different parameters. Model usually gives up on hallucinations when challenged. Sometime it pushes back and provides explanation with sources.
I have a script that puts models into dialog, moderates discussion and takes notes. I run this stuff overnight, so getting multiple choices speeds up iteration.
Imagine a dictionary where the words are only mostly in alphabetical order. If you look up a word and don't find it, you can't be certain it's not in there. It's as useful as asking someone else, or several other people, but it's value as a reference is zero, and there's no shortage of other people on the planet.
On the contrary, it's arguably the breakthrough that allowed us to model concepts and meaning in computers. A sufficiently high-dimensional embedding space can model arbitrary relationships between embedded entities, which allows each of them to be defined in terms of its associations to all the others. This is more-less how we define concepts too, if you dig down into it.
> Imagine a dictionary where the words are only mostly in alphabetical order. If you look up a word and don't find it, you can't be certain it's not in there.
It's already the case with dictionaries. Dictionaries have mistakes, words out of order; they get outdated, and most importantly, they're descriptive. If a word isn't in it, or isn't defined in particular way, you cannot be certain it doesn't exist or doesn't mean anything other than the dictionary says it does.
> It's as useful as asking someone else, or several other people
Which is very useful, because it saves you the hassle of dealing with other people. Especially when it's as useful as asking an expert, which saves you the effort of finding one. Now scale that up to being able to ask about whole topics of interest, instead of single words.
> it's value as a reference is zero
Obviously. So is the value of asking even an expert for an immediate, snap answer, and going with that.
> and there's no shortage of other people on the planet
Again, dealing with people is stupidly expensive in time, energy and effort, starting with having to find the right people. LLM is just a function call away.
LLMs do math by trying to match an answer to a prompt. Mathematica does better than that.
> LLMs do math by trying to match an answer to a prompt. Mathematica does better than that.
Category error. Pencil and paper or theorem prover are better at doing complex math than snap judgment of an expert, but an expert using those tools according to their judgement is the best. LLMs compete with snap judgement, not heavily algorithmic tasks.
Still, it's a somewhat pointless discussion, because the premise behind your argument is that LLMs aren't a big breakthrough, which is in disagreement with facts obvious to anyone who hasn't been living under a rock for the past year.
Is this article even written by a person? Hard to know; they have a real blog with real article, but stuff like this reads strangely. Maybe it's just not a native english speaker?
> Hallucinations are certainly the toughest nut to crack and their negative impact is basically only slightly lessened by good confidence estimates and reliable citations (sources).
> The impact of contradictions in the training data.
(was this a prompt header you forget to remove?)
> LLM are incapable of "self-inspection" on their training data to find logical inconsistencies in it but in the input context window they should be able to find logical inconsistencies.
Annnnyway...
Hallucinations cannot be fixed by a good corpus in a non-deterministic (ie. temp > 0) LLM system where you've introduced a random factor.
Period. QED. If you think it can, do more reading.
The idea that a good corpus can significantly improve the error rate is an open question, but the research I've seen tends to fall on the side of "to some degree, but curating a 'perfect' dataset like that, of a sufficiently large size, is basically impossible'".
So, it's a pipe dream.
Yes, if you could have a perfect corpus, absolutely, you would get a better model.
...but how do you plan to get that perfect corpus of training data?
If it was that easy, the people spending millions and millions of dollars making LLMs would have, I guess, probably come up with a solution for it. They're not stupid. If you could easily do it, it would already have been done.
my $0.02:
This is a dead end of research, because it's impossible.
Using LLMs which are finetuned to evaluate the output of other LLMs and using multi-sample / voting to reduce the incidence of halluciations that make it past the API barrier is both actively used and far, far more effective.
(ie. it doesn't matter if your LLM hallucinates 1 time in 10; if you can reliably detect that 1 instance, sample again, and return a non hallucination).
Other solutions... I'm skeptical; most of the ones I've seen haven't worked when you actually try to use them.
It's trivial to create a corpus of True Maths Facts and verify that they're correct. But an LLM (as they're currently structured) will never generalise to new mathematical problems with 100% success rate because they do not fundamentally work like that.
When sampling data all over the internet, your data set only represents people who did write, did respond to questions - with no representation of what they didn't. Add into that confidently wrong people - people who respond to questions on, say, StackOverflow, even if they're wrong, and suddenly you have a data set that prefers replying bullshit, because there's no data for the people who didnt know the answer and wrote nothing.
Inherently there's no representation in the datasets of "I don't know" null values.
LLMs are forced to reply, in contrast, so they "bullshit" a response that sounds right even though not answering or saying you don't know would be more appropriate - because no-one does that on the internet.
I always assumed this was a big factor, but am I completely off the mark?
May be it is a pipe dream to drastically improve on hallucinations by curating a self-consistent data set but I am still interested in how much it actually impacts the quality of the final model.
I described one possible way to create such a self-consistent data set in this very blog post.
Maybe something between Cyc and Google's math and geometry LLM's could help.
You want planning, you can do monte carlo tree search and use LLM to evaluate which node to explore next. You want verifiable reasoning, you can ask it to generate code(an approach used by recent AI olympiad winner and many previous papers).
What is even "planning", finding desirable/optimal solutions to some constrained satisfaction problems? Is the llm based minecraft bot voyager not doing some kind of planning?
LLMs have their limitations. Then augment them with external data sources, code interpreters, give it ways to interact with real world/simulation environment.
I do think LLMs could be used to assist in building a world model that could be a foundation for an AGI/agent system. But it won't be the major part.
Given how inevitable it is, it seems to me that it might be.
There are still lots of issues and limitations but it's a very different experience than with gpt 3 early on. A lot of the smaller OSS models are a bit of a mixed bag in terms of hallucinations and utility. But they can be useful if you apply some skills. Half the success is actually learning to prompt these things and learning to spot when it starts to hallucinate.
One thing I find useful is to run ideas by it in kind of a socratic mode where I try to get it to flesh out brain farts I have for algorithms or other kinds of things. This can be coding related topics but also non technical kinds of things. It will get some things wrong and when you spot it, you can often get a better answer simply by pointing it out and maybe nudging it in a different direction. A useful trick with code is to also let it generate tests for its own code. When the tests fail to run, you can ask it to fix it. Or you can ask it for some alternative implementation of the same thing. Often you get something that is 95% close to what you asked for and then you can just do the remaining few percent yourself.
Doing TDD with an LLM is a power move. Good tests are easy enough to understand and once they pass, it's hard to argue with the results. And you can just ask it to identify edge cases and add more tests for those. LLMs take a lot of the tediousness out of writing tests. I'm a big picture kind of guy and my weakness is skipping unit tests to fast forward to having working code. Spelling out all the stupid little assertions is mindnumbingly stupid work that I don't have to bother with anymore. I just let AI generate good test cases. LLMs make TDD a lot less tedious. It's like having a really diligent junior pair programmer doing all the easy bits.
And if you apply SOLID principles to your own code (which is a good thing in any case), a lot of code is self contained enough that you can easily fit it in a small file that is small enough to fit into the context window of chat gpt (which is quite large these days). So, a thing I often do is just gather relevant code, copy past it and then tell it to make some reasonable assumptions about missing things and make some modifications to the code. Add a function that does X; how would I need to modify this code to address Y; etc. I also get it to iterate on its own code. And a neat trick is to ask it to compare its solution to other solutions out there and then get it to apply some of the same principles and optimizations.
One thing with RAG is that we're still under utilizing LLMs for this. It's a lot easier to get an LLM to ask good questions than it is to get them to provide the right answers. With RAG, you can use good old information retrieval to answer the questions. IMHO limiting RAG to just vector search is a big mistake. It actually doesn't work that well for structured data and you could just ask it to query some API based on a specification of use some sql, xpath, or whatever query language. And why just ask 1 question? Maybe engage in a dialog where it zooms in on the solution via querying and iteratively coming up with better questions until the context has all the data needed to come up with the answer.
If you think about it, this is how most knowledge workers address problems themselves. They are not oracles of wisdom that know everything but merely aggregators and filters of external knowledge. A good knowledge worker / researcher / engineer is one that knows how to ask the right questions in order to come up with an iterative process that converges on a solution.
Once you stop using LLMs as one shot oracles that give you an answer given a question, they become a lot more useful.
As for AGI, a human AI enhanced by AGI is a powerful combination. I kind of like the vision behind neuralink where the core idea is basically improving the bandwidth between our brains and external tools and intelligence. Using a chat bot is a low bandwidth kind of thing. I actually find it tedious.
1. Do I want LLMs to be trained with licensed data, that's arguably well curated. Or, do I want LLM to scrape the web because it is more democratic in opinions?
2. If hallucination is not about training data but how LLM uses that data to extrapolate info that's not directly present in training data - can we teach it this skill to make better choices?
3. It's easy to define good data for facts. How to define good data for subjective topics?
4. For subjective topics, is it better to have separate LLMs trained with each theme of opinions or one big LLM with a mix of all opinions?
5. Is using LLM to improve its own training data truly helpful as the author claims? If yes - is this recursion method better or it's better to use multiple LLMs together?
Dang! If I interview for a position that requires knowledge of AI - every question they ask will be answered with more questions. smh!