LLMs hallucinate 100% of the time - hallucinating is what they Do. Often they’re hallucinating things that also occur in reality, but it sounds like the legal products are only doing so 66-83% of the time.
It is perhaps worth noting that mathematicians at all three of the above stages of mathematical development can still make formal mistakes in their mathematical writing. However, the nature of these mistakes tends to be rather different, depending on what stage one is at:
1. Mathematicians at the pre-rigorous stage of development often make formal errors because they are unable to understand how the rigorous mathematical formalism actually works, and are instead applying formal rules or heuristics blindly. It can often be quite difficult for such mathematicians to appreciate and correct these errors even when those errors are explicitly pointed out to them.
2. Mathematicians at the rigorous stage of development can still make formal errors because they have not yet perfected their formal understanding, or are unable to perform enough “sanity checks” against intuition or other rules of thumb to catch, say, a sign error, or a failure to correctly verify a crucial hypothesis in a tool. However, such errors can usually be detected (and often repaired) once they are pointed out to them.
3. Mathematicians at the post-rigorous stage of development are not infallible, and are still capable of making formal errors in their writing. But this is often because they no longer need the formalism in order to perform high-level mathematical reasoning, and are actually proceeding largely through intuition, which is then translated (possibly incorrectly) into formal mathematical language.
The distinction between the three types of errors can lead to the phenomenon (which can often be quite puzzling to readers at earlier stages of mathematical development) of a mathematical argument by a post-rigorous mathematician which locally contains a number of typos and other formal errors, but is globally quite sound, with the local errors propagating for a while before being cancelled out by other local errors. (In contrast, when unchecked by a solid intuition, once an error is introduced in an argument by a pre-rigorous or rigorous mathematician, it is possible for the error to propagate out of control until one is left with complete nonsense at the end of the argument.)
https://terrytao.wordpress.com/career-advice/theres-more-to-...
I wouldn't expect a human to be able to just spit out all relevant case law, completely correctly with every last aspect of the citation accurate down to every last digit, based on solely their internal language model either.
Why is anyone even defending this? LLMs are objectively unsuited for this. LLMs are objectively failing at this. LLM architecture is obviously not a good choice for this task to anyone with even a basic understanding of them. But... who cares? All this means is that LLMs are not suited for this task. I wouldn't be banging on this except for the engineers running around here with stars in their eyes thinking that despite all the evidence they've finally found the tool that does everything. No. We haven't. They don't. This isn't an attack on the viability of AI as a whole or LLMs specifically, any more than pointing out that a hammer is not a good tool for cutting grass is an attack on hammers.
Providing truthful information is one of the latter and one of the flaws is our memory. Writing it down or recording it correct this and the next problem was retrieving the snippet we need. That's one of the things lawyer does. They don't invent laws out of thin air, they're just better at finding the helpful bits. Even today, I spend a lot of time looking at libraries and languages documentation to verify a hunch I have. But I still have to go through the learning phase to know what to search for.
In Dreyfus Model of skill acquisition, we have these four qualities:
Recollection (non-situational or situational)
Recognition (decomposed or holistic)
Decision (analytical or intuitive)
Awareness (monitoring or absorbed)
We need better tools, not probability machine.
Re: LLMs and law, I agree with all your complaints 100% if we constrain the discussion to direct/simplistic/“chatbot”-esque systems. But that’s simply not what the frontier is. LLMs are a ground breaking technique for building intuitive components within a much larger computational system that looks like existing complex software. We’re not excited about (only) crazy groundbreaking products, we’re excited about enhancing existing products with intuitive features.
To briefly touch on your very strong beliefs about LLM models being a bad architecture for legal tasks: I couldn’t disagree more. LLMs specialize in linguistic structures, somewhat tautologically. What’s not linguistic about individual atomic tasks like “review this document for relevant passages” or “synthesize these citations and facts into X format”? Lawyers are smart and do lots of deliberation, sure, but that doesn’t mean they’re above the use of intuition.
As far as we’re in an argument of some kind, my closing argument is that people as a whole can be pretty smart, and there’s a HUGE wave of money going into the AI race all of a sudden. Like, dwarfing the “Silicon Valley era” altogether. What are the chances that you’re seeing the super obvious problem that they’re all missing? Remember that this isn’t just stock price speculation, this is committed investments of huge sums of capital into this specific industry.
It makes sense TO conflate them so that people can better understand what's going on under the hood. We need to "de-personalize" these things as much as possible so we don't make stupid mistakes surrounding them.
Think of it like this: How USEFUL is it to say "I googled 'the closest ice cream shop to my house' but it 'hallucinated' the one that was the second-closest'"
It's just a dumb term.
a google search doesn't do that though. It will actually select ice cream shops from it's database, determine their location, your location, and calculate the distance. And then select the closest.
It may mis-calculate, but it will not "hallucinate".
This is incredibly different than what LLMs do, which is why hallucinate is appropriate for an LLM but not appropriate for a conventional search or map engine
The computer is crunching numbers and probabilities of things and giving its best guess based on that, in both cases. If it's appropriate for one, it should be for the other.
(which, I'd say, it isn't, because "hallucination" implies "mistake by a human-like thing")
Google maps isn't choosing a nearest ice cream store based on probability, it's using an algorithm designed for this purpose.
I don't really think there's any winning this propaganda war until it shatters under its own illusions. It's shouting into the wind.
if LLM was just another "framework" that a php developer could pick up we could build systems to deal with these "hallucinations".
Well, it’s either that or you were being a human elitist and portraying a difference of degree as a difference in kind. Idk if we’ve settled on terms yet but I’d probably level ones like “automatist”, “bio supremacist”, or maybe “radical humanist”…
Insofar as the mechanism by which LLMs operate mirrors the mechanisms by which the brain operates - which they do, to some degree - yes, the fact that LLMs hallucinate reflect the gap between perceptions, thoughts, and "reality" in human behavior, and there is indeed a deeper philosophical lesson to be drawn here. If LLMs manage to disabuse the field of software engineering of its apparently common conceptions about the brain works and reflects reality, I'll consider it a win.
> Well, it’s either that or you were being a human elitist and portraying a difference of degree as a difference in kind. Idk if we’ve settled on terms yet but I’d probably level ones like “automatist”, “bio supremacist”, or maybe “radical humanist”…
I'm not sure how I could be considered a bio-supremacist for pointing out that a hammer is not a drill. LLMs are phenomenally complicated tools, but as currently deployed, they're tools, not beings. We may at some point create a system that one could credibly claim is an actual artificial intelligence, and that system may contain things that look or operate like LLMs, but right now what we're looking at are large statistical model that we're operating by providing it inputs and reading the outputs.
A first-line criteria here, for me, would be that the system operates continuously (ie: not just when someone puts a token in the box). The LLMs we're discussing here do not do that. Someone generates a series of tokens, provides those as inputs to the equation, and then runs the equation, and then decides whether to continue to provide inputs into the equation until they're satisfied by the result. That's a tool, not a "being".