Hacker News Reader

Comment by swatcoder | Hacker News Reader

By “when it was stumped” I assume you mean “when the text completion ranked invoking a human expert as a more likely next phrase than an immediate answer”

It doesn’t invoke your human expert when it’s “stumped”. It invokes it when the question associates with training data where invoking an expert is more likely (i.e complicated question, esoteric questions, etc) — it has no knowledge and no insight into your question or its answer; it just can imitate that certain looking questions are followed by expert referrals.

We really need to stop using the wrong language for these things.

View on HN

We need to realize that the sum may be greater than its parts. I'm not aware of any concrete evidence that says text token prediction can't be just a good of a path to intelligence as whatever "code" our wetware is running.

For something that has "no knowledge and no insight" it sure is damn good at taking unique social, technical, and other problems I have and offering an array of solutions, with pros/cons and probable outcomes of each.

> For something that has "no knowledge and no insight" it sure is damn good at taking unique social, technical, and other problems I have and offering an array of solutions, with pros/cons and probable outcomes of each.

What that might be telling you is that your problems aren’t so unique after all.

Getting a sound answer from ChatGPT is very literally like looking up some weird concern that never occurred to you before in a search engine and finding a hundred upvotes and a three page long discussion from 2012. Those upvotes and comments that happen to come up in Google are only a tiny share of the online and offline encounters with that same concern, and tell you that your completely novel-to-you problem is actually not so novel after all.

For that class of problem, ChatGPT can usefully synthesize content that can almost always look right and often be accurate. Just like Google, it has a huge database of source material and just happens to synthesize an original text similar to many samples in it rather than pointing out one specific sample.

But as you dig into more specificity, idiosyncrasy, and esoteric “problems”, the reference material gets too thin and it starts to generate noise because it lacks alternative ways of reasoning or knowing.

There might be another leap forward in the next few years, but this generation of tools only do what they do.

I tried asking ChatGPT easy but completely off-the-wall questions and it consistently got the answers surprisingly right. For example, I asked it "Which is more powerful, a DEC VAX or a GTX 1070?" and it both noted that this comparison was questionable, as the two are not used in the same context, and correctly answered that the GeForce has much more processing power than a VAX. There are no Google results that directly answer that question, and yet ChatGPT knew that was what it should say.

So while it is obviously just guessing what to say, the idea that it is just regurgitating existing material seems wrong to me. It does some pretty impressive synthesis to come up with its answers.

I wonder how much of that is because the way such questions are asked is so formulaic that it's possible to "guess" the response from that information.

If you were to ask a truly nonsense question, like "Which is more powerful, a GTX 1070 or an F-16?", what would the response be?

"It's difficult to compare a graphics card like the GTX 1070 and an aircraft like the F-16 as they are designed for very different purposes and operate on completely different scales.

The GTX 1070 is a powerful graphics card that is designed to render high-quality graphics for gaming, video editing, and other visual computing applications. It has impressive processing power and can handle complex graphics and visual effects with ease.

On the other hand, the F-16 is a highly advanced fighter aircraft designed for air-to-air and air-to-ground combat missions. It is a complex machine that requires a highly skilled pilot to operate and is equipped with state-of-the-art avionics, weapons systems, and engines.

In terms of raw power, the F-16 is undoubtedly more powerful than the GTX 1070, with its advanced capabilities in speed, agility, and firepower. However, it's important to note that these are two very different types of machines, and a direct comparison of their power is not really possible."

LLMs keep demonstrating that they have an internal "world model" representation of concepts and the relationships between them.

Meanwhile ostensible "expert" humans continue to regurgitate obvious falsehoods like "it's just a fancy autocomplete" or "you're being fooled by a system that just predicts the most probable next word".

That world model is an emergent phenomenon arisen out of training, they are not explicitly trained a world model.

I’d phrase it the other way around, learning a world model is necessary in order to make such good token predictions.

I don't see why "internal model" and "predicts the most probable next word" are mutually exclusive.

Who said they're mutually exclusive?

The internal "world model" are things like "GPU's are faster than CPU's", "Military hardware beats nonmilitary things" etc. Then when you ask the comparison question it categorises the items, and for the plane vs card you get military vs nonmilitary response. Or computer vs gpu you get the cpu vs gpu response or similar. I showed the example of how to reason about this in a post below.

Anyway, while you can call that a "world model", that world model is very brittle since its just a bunch of heuristics, if you reverse engineer the heuristic it uses it is really easy to find cases where it fails completely. Like if you ask it to compare a modern integrated GPU to an old dedicated GPU, it will say that the old dedicated GPU is faster even though it is ancient and slow, since when people talk about dedicated vs integrated GPU they almost always say that the dedicated GPU is faster.

It doesn't have a model it is a model and it's a model of language. It's never interacted with the world, only with text. There are patterns and relationships among words that can describe the real world, that's why we can use it to communicate complex ideas. The problem is that the map is not the territory.

You've never interacted with the world, only with the sense impressions that were triggered in your nervous system. All world models are inferred.

Bad argument. My sense impressions consist of far more than just text input. We don't know if consciousness consists of one model, multiple models or is something different from a statistical model but makes use of one or more of them. I also don't consist of only a nervous system, I also have an endocrine system that has very pronounced effects on my behaviour and my interpretation of input.

There's a giant gap here that people keep glossing over when they treat LLMs as almost-human.

Sense impressions are mediated by fairly simple neuronal firing mechanisms, somewhat different, but similar in concept to the wires we use to encode language tokens.

I'm not really trying to make a point here about consciousness, although I was interested by the article - it seems to imply that gpt-3 performs better at some tasks when it has a train of thought / stream of consciousness.

People keep saying that there's a giant gap, but I still don't really know what they think it is, and there seem to be a lot of points of similarity. It's very clear that these large language models are, despite their problems, much more capable than was expected by nearly everyone not very many years ago.

To me, it comes down to whether or not you think the sum total of a human being is language. If your life consists almost entirely of communication based activities then it's easy to see why one would think that is the case but if you spend any appreciable amount of time doing work that deals with physical objects and their interactions with the environment you find that language is only one tool in the box and isn't capable of replacing the others. That's not even bothering with the metaphysics of consciousness or subjective experience.

> Bad argument. My sense impressions consist of far more than just text input.

Multimodal models are being trained right now using text images, video, and audio. Eventually you can add data from pressure, heat, and acceleration sensors and motors (sense of touch). We can further add additional "senses" - data from RADAR/LIDAR, magnetometer, multispectrum vision, radiation sensors if desired.

AI will come to know our world very well.

Why wonder?

"It's not appropriate to compare a graphics card (GTX 1070) to a military fighter jet (F-16) as they are completely different things designed for entirely different purposes.

A GTX 1070 is a high-end graphics card designed for gaming and other graphics-intensive tasks. It has a lot of processing power and can handle complex visual computations quickly and efficiently.

On the other hand, an F-16 is a military fighter jet designed for air-to-air and air-to-ground combat. It is equipped with advanced weapons systems, avionics, and other technologies that allow it to perform a wide range of military operations, including surveillance, reconnaissance, and combat missions.

In short, while a GTX 1070 is a powerful graphics card, it is not designed for the same purposes as an F-16, which is a highly advanced and specialized military aircraft."

Try asking if it is faster than a low end modern card, in a comparison it looks at comparisons people have made before and decide based on those, and most comparisons with 1070 the 1070 probably came up ahead since it is a relatively high end card. It probably hasn't seen many DEC VAX comparisons, so it would decide mostly based on the other part, and if that is low end likely it would say the DEC VAX is faster since it is so used to say that the other part is slower.

Or it has a separate category in its state space for old computers and will say the same thing for all of those in these comparisons, but you can trip it up using the weakness I talked about above in most cases. Its just that the comparisons people come up with can often be answered correctly by that method since people like to compare "powerful thing to weak thing", and in those it just looks at how often each wins and decides based on that, instead of comparing them directly.

I tried doing the same thing with the Radeon R7 250 (an old GPU that was weak when it was introduced) and got essentially the same answer, so it doesn't seem like it's just regurgitating praise.

What impressed me just as much about these answers, though, was that it correctly identified that the comparison was silly. I don't believe it has seen that exact comparison before, so it seems like it has to have a granular enough categorization of tokens to know that people would say a comparison between a GPU and a computer is silly, and that those are a GPU and computer, and that one is old and the other is new.

GPU's typically compute faster than CPU's, so maybe that is the conversation it identifies it as? Meaning, when people ask about power difference between CPU and GPU they say they aren't directly comparable, but the GPU has much more processing power.

That would mean it would identify an old GPU as faster than a modern CPU, even though it isn't, since it would slot that into the same conversation. The hard part with finding out the source for the logic is that it can map items to similar items in many ways, and there are trillions of conversations out there that it could map it to, so it can do quite a lot of heuristics that will work fairly well for naive questions even though it can't apply any direct logic.

If it could link to similar conversations that it bases its reasoning on then that would probably greatly help understand where its logic would fail. So before it can do that we probably wont be able to eliminate these failure modes, because humans aren't random enough to generate the edge cases unless they understand what logic the model used. For example, comparing an old computer to a modern GPU might seem random to you, but to ChatGPT it just sees something like "This looks like a CPU vs GPU comparison, aha so I map it to: {CPU} isn't really comparable to {GPU}, but {GPU} has much more processing power". If you could see something like that then it would be obvious where it would fail, but without that it looks like magic.

> but to ChatGPT it just sees something like "This looks like a CPU vs GPU comparison, aha so I map it to: {CPU} isn't really comparable to {GPU}, but {GPU} has much more processing power"

Jup, just tested with this: "what is more powerful Athlon PRO 3125GE or Geforce 6800". It said that they aren't comparable, but that the GeForce is probably faster:

"In summary, while the GeForce 6800 is likely to be more powerful than the integrated graphics in the Athlon PRO 3125GE, the two cannot be directly compared as they have different functions and are designed for different purposes."

Which is wrong, the modern integrated GPU is much faster, and they are directly comparable. So that is how it answered your question, and since I maanged to figure out what conversation I could find this failure mode where it gives the completely wrong answer.

Once you understand that ChatGPT just slots what you ask for into some conversation based on fuzzy matching then it is really easy to understand what sort of answers it can give and can't give.

> What impressed me just as much about these answers, though, was that it correctly identified that the comparison was silly.

That's been one of ChatGPT's most consistently impressive features, but it's clearly a function of a lot of training to encourage strings that reject comparisons when they're in different categories. Sometimes it does so based on simple heuristics like "FamousCharacter is fictional", sometimes it demonstrates a surprisingly nuanced grasp of historical periods or events so it will insist that x and y lived in different centuries or tell you that during a conflict p attempted to invade q, not the other way round. Even when it makes logical errors (no sooner has it told you that if Finland did not exist, there would be no Winter War, it suggests that the absence of Finland would not have affected anything about the outcome...) it's pretty good overall.

But the trouble with lots of training to reject silly user inputs, is when it gets it wrong it can end up insisting it's been a good Bing dealing with a bad user...

You, sir, have a misconception about the tool.

Damn, my magic 8 ball can't give me probable outcomes! How good are chatGPT's by the way? what are the odds I fart in the office tomorrow?

> what are the odds I fart in the office tomorrow?

Outlook not so good.

So far so good, but it's early!

"what can be asserted without evidence can also be dismissed without evidence."

https://en.m.wikipedia.org/wiki/Hitchens's_razor

Reply on news.ycombinator.com