Comment by photochemsyn | Hacker News Reader

Comment by photochemsyn | Hacker News Reader

The thing about LLMs that is revolutionary is just how fast you can find a solution to a problem or expand on answers to a problem even to the point of generating a realistic computational-mathematical model of the problem.

For example, the author brings up Kepler, so let's ask:

> "Please explain in concise terms how Kepler used Tycho Brahe's observational data to come up with Kepler's three laws, on ellipitical orbits sweeping equal areas and the square:cube ratio and so on."

Now I want to see if I can build a computational model of Kepler's Laws:

> "Is there a popular orbital mechanics library for the Python language capable of expressing Kepler's Three Laws in code?"

Okay, now I want a simple model to build in code:

> "How would I go about using poliastro to build a dynamic model of the solar system in silico, starting with just the Sun and the the planets Mercury, Venus, Earth, Mars, Jupiter and Saturn?"

Now trying to use Google Search or anything similar to do that, okay maybe you'd eventually find some forum board or stackoverflow physics discussion of orbital dynamics, but this is an incredibly quick entry point to a complex and obscure subject. Of course, you'd want to use Google Search to check the answers to some degree, maybe see what the real astrophysicists are using to run their models, but there's no doubt that this whole thing is a pretty fundamental game-changer, at least for people who understand its limitations.

P.S. the real question will be if we can build AI systems capabale of generating Kepler's Laws from Tycho Brahe's data, instead of just a predictive model. A similar issue is if these AIs can construct novel mathematical proofs.

It's all fun and games until the AI completely hallucinates the answer, while making it sound completely pausible and correct.

Well, sure, but in the context of the previous example I can now go look at poliastro (whatever that is, never heard of it before) documentation and do some more queries and so on.

Certainly anyone just running LLM code output without doing a bunch of tests and checks is a lunatic.

So....like a person?

The difference to a person is that most (though not all) people actually have an understanding if they know something, if they guess something, if they are making something up, or if they are outright lying. Which is about the first thing you train during a scientific education.

And in an honest interaction, they will tell you.

ChatGPT etc. does not. So basically it acts like a pathological liar (who happen to be right as long as it has been trained on something that comes close enough).

Yes, agreed.

Animals (including humans) predict experiences, while language models only predict text. Text is not very closely linked to reality, while experiences are. So it is not surprising that we (humans) have a better sense of what we know than language models.

> The difference to a person is that most (though not all) people actually have an understanding if they know something, if they guess something, if they are making something up, or if they are outright lying. Which is about the first thing you train during a scientific education.

Scientist? STEM workers? Maybe. People who value truth / consistent world model for its own sake? Sure.

Normies? Not so much. It's not that they can't - I think they never learned to pay enough attention. What I mean is, most people tend to say things in confidence, and maybe even believe them, regardless of how they acquired the information. They don't seem to process the distinction between "I read it in a book", "A colleague told me that their colleague heard on the radio that...", etc. They don't even track provenance of the information, which is a critical skill you need to not end up confidently making things up.

It's a learnable skill, and I think it's even quite easy to pick it up from osmosis - but someone has to make the person feel that it's important.

> So basically it acts like a pathological liar

I think it's more of a bullshitter than a liar, in the sense that it doesn't care about truth value of what it says.

Lying involve knowing the truth, or at least knowing that the thing you're saying ain't it. Making a mistake involves you thinking you're saying the truth, but actually being wrong about it. Bullshitting is just saying whatever helps you achieve your goal; the truth value of what you say doesn't even enter the picture.

The issue of hallucinations is overblown. I use GPT4 all the time and don't see any hallucinations at all. It's a big problem with Google BARD and GPT3 and earlier models. But GPT4 fixed the issue of hallucinations completely.

I wonder if it's "fixed" or if it's just less obvious. It seems that an LLM would "hallucinate" a bogus answer if it didn't actually have a good answer somewhere in its training. Is GPT4 so much more trained that it rarely encounters something it doesn't have a reasonable answer for? In which case, it would still "hallucinate" if cornered on some more obscure matter?

I mean, like, what would it mean to actually solve the problem? I would not expect any computer system to know the answer to literally everything, so the fix is not to train it more, but rather, for it to realize and acknowledge if it doesn't have enough data to give a good answer, and tell you that rather than make something up.

Does GPT4 do that? (I have yet to use GPT4 myself.)

In my experience, GPT-4 is equally willing to make things up if it doesn't know something but it has so much more knowledge than GPT-3.5 that this happens less often in practice.

On that note, I’ve found that just including in the prompt a request for GPT4 to consider its confidence level in an answer and inform me of that confidence level, to reconsider its answer if its confidence is low, and that accuracy is critically important for the topic of the conversation, also can result in better steering it.

I mean, kind of works with humans too. In a high pressure work or school environment, people can confabulate the answers someone wants to hear to avoid discomfort “oh yes, we did the training exercise, the trucks and tanks are in great shape.” Sometimes people need to know they can admit they are wrong. I wonder if some aspect of current LLM tuning/training could be modified so LLMs are more “comfortable”, for lack of a non-anthropomorphized term coming to mind, with admitting they are unsure.

When I tried this, it made up random, non-repeatable confidence values that had little bearing to the veracity of its statements.

If you read the GPT-4 paper, they claim that the base model was well calibrated but the calibration was completely broken by RLHF

GPT-4 hallucinates significantly less than those other models but it is going way too far to say that it has fixed the issue completely.

In my experience, it probably hallucinates about 3x less than GPT 3.5. I use GPT-4 a lot but really only for code generation and answering questions about documentation.

I'm not including cases where it gives answers that are out of date as hallucations, I consider that an entirely separate failure mode.

Oh, GPT-4 does hallucinate. It's more subtle than with GPT-3.5, but it's there. Most of the time I had it happen, it could either correct itself, or counter-hallucinated (with further corrections oscillating) - the latter is quite easy to spot.

It's not a big deal in practice, though, as long as you remember to take a probabilistic approach. GPT-4 is not an oracle, it's a 4 year old savant, that tries its best, but has attention span of a hamster, and likes to extrapolate instead of saying "I don't know". In domains you have at least minimal experience in (e.g. you can program, but not in the language/framework you're asking about), it's relatively easy to verify things by common sense and/or by paying attention to the conversation - if the follow-up message seems to contradict the previous one, it's likely at least one of them involves a hallucination. Etc.

The overall feel I get for GPT-4, at least in terms of code, is that its hallucinations tend to mostly be of the "I don't know for sure, but seems logical that..." kind.

A real example from earlier today: I asked GPT-4 to refactor some code C++ that contained function calls like InitializeSomething(), AddWidget(), etc. It decided to put all those calls into RAII objects, and hallucinated the existence of corresponding function DeinitializeSomething(), RemoveWidget(), etc. I kind of understand why it did that - it feels only logical that such functions exist too.

In most cases it does acknowledge that it doesn't know.

You shouldn't call it overblown given GPT4 was released not even a month ago, but if it's fixed now then great.

Hey, check out this paper, it’s probably got a bunch of descendants by now:

https://arxiv.org/abs/1810.10525

Also DreamCoder is relevant.

I reckon the answer to your real question is “yes”

Reply on news.ycombinator.com