Comment by tobiasSoftware | Hacker News Reader

Comment by tobiasSoftware | Hacker News Reader

Parent Full thread

I recently got into stories generated by GPT-3. What I notice is that it seems to be missing an understanding of state that causes constant inconsistencies.

For example, a popular Youtube video has a battle between Link and Kirby in it: "He finally releases the Hylian Shield and lets Link be engulfed in a massive fireball. When Link is reduced to a pile of ashes, Kirby is victorious. Kirby wins the fight to the death. Link stands there, dazed by the attack."

Most of that actually sounds pretty darn good and even sounds written by a human. It's to the point where there is a sensible structure to the story because the AI is getting the relationships between words. Massive fireball -> pile of ashes -> victory -> wins the fight to the death. That all looks good. The problem is that "Link is reduced to a pile of ashes" should put Link into a "dead" state, and when in the "dead" state Link can't stand and be dazed.

The problem of course is that the computer can't understand all of this. It can understand that there is a probabilistic link between "pile of ashes" and "fight to the death" so after writing the first it is much more likely to write the second. But it still doesn't understand what "death" actually means. I've thought for a while that neural nets alone aren't going to solve machine generated speech and that the real solution will be some sort of hybrid between a neural net and some sort of finite state automata. The finite state automata could then put a character into a "dead" state and know that when in a "dead" state they can't "stand" or "be dazed."

Source: (Video by DougDoug where he manually sets up battles between characters with a few paragraphs and then lets the AI generate the text of the battle. Sometimes it makes sense and other times someone's face will turn into a button or their eyes will shoot lasers) https://www.youtube.com/watch?v=PwY-jVSM-f0&t=2835s

I've played a lot of AI Dungeon and this is one of my main issues. You need to constantly correct the AI or retry the latest action (that there are big easily accessible buttons for such actions is itself telling).

One of the best sessions I had was a fairly epic story about a god of shadows whose unruly shadow monsters were attacking all humans. The god himself wanted them stopped and sought my (a powerful mage) help. We needed to make our way to his lair where a powerful ritual could destroy him and banish all his minions.

After an epic tale the plan succeeds, the god is destroyed, his shadows disperse. As the dust settles the god congratulates me but reminds me that I must make haste for the god of shadows has sent his monsters to attack all humans and he must be stopped...

This is more or less the concept of time, correct? I think an AI can understand state in simple cases (i.e. it can likely answer correctly "I didn't water my plant in 2 months, is it dead?"), but the way the current models are designed its just request / response, they perhaps from the very root way of how they are implemented don't (or can't) have a sort of _narrative_ sense of state. This is also present when you have a conversation with them. They won't bring up topics from the start or earlier part of the conversation, because they don't really "know" they happened. They simply receive and reply, that doesn't actually change the state of the model itself. To me this is one of the main keys that still need to be unlocked in AI capabilities. You need a neural net to modify itself in real time and track those changes to be able to have this sense.

To provide an example, it's almost like you need a _time series_ of GPT-3s, not just a single GPT-3 neural network, and the model itself would need to be able to self-inspect those time series and say to itself "ah yes, this was my old foolish understanding, now I have this new, better, understanding". I have no idea how this would look in technical terms, these are just the musings of a somewhat more-than-casual AI observer.

It's not really consistent outside of temporal issues.

As mentioned there are buttons for redoing or changing the AI response. This is because it often spouts nonsense or loses the plot.

In my example above the god who should have been dead talked about himself as a third person.

The AI also often mixes up people's gender or roles. One issue that used to be very common but has improved lately is that it used to mix up who said what in dialogues.

> The problem is that "Link is reduced to a pile of ashes" should put Link into a "dead" state, and when in the "dead" state Link can't stand and be dazed.

Yes, this kind of problem is real. But recent papers show you can ask the model to do reasoning / chain of thought / rationales before coming up to the answer. They can do complex tasks in a series of small steps instead of trying to do it in one step and failing. I believe it's not a fundamental limitation, just a matter of "blurting out something stupid" vs "taking your time to think before you speak".

> "taking your time to think before you speak"

One of the foremost core requirements for actual general intelligence.

Do you have any specific papers in mind?

I saw that mentioned in OPT175 release. How is the model forced to give reasoning?

It’s not forced to, they just give a couple examples with reasoning and then the model figures out that that’s what it’s supposed to do

> the computer can't understand all of this

It could...

> the real solution will be some sort of hybrid

An engine that built ontologies even just through ANNs could maybe suffice. It's still a game of entities and relations ("state" is still a relation, and relations can be implemented in ANNs).

Meaning that the network has to define "battler", "instance", "Link", "Kirby", "engulf", "fireball", "victory", "death", and progressively "know" what those things are and what they imply. It has to build a world including the laws and the entities.

Check out PaLM and chain-of-thought prompting for a marked improvement on reasoning.

https://ai.googleblog.com/2022/04/pathways-language-model-pa...

GPT-3 anecdotally can't pick up on chain-of-thought quite as well. https://www.lesswrong.com/posts/EHbJ69JDs4suovpLw/testing-pa...

How is the model forced to do chain-of-thought?

> The problem is that "Link is reduced to a pile of ashes" should put Link into a "dead" state, and when in the "dead" state Link can't stand and be dazed.

For a cartoonish video game, that's not too far fetched...

I've seen more ridiculous things in animes and such.

That said, I do get your point, and I agree.

Have you ever read stories written by young children? Kids learning to write have similar issues albeit with a much smaller vocabulary.

I was thinking the same thing, but you can explain to a child what the problem is. They can learn, in just a few minutes, how not to make that mistake again and improve their model of the world. It’s not as clear to me how you’d do that with GPT3, could you construct a text that includes this information and have it ingest it?

Yes, you can put it into the prompt. The prompt can contain the task name, a task description, examples, and example rationales. Instruct GPT-3 can get the meaning of the task very fast, usually with just the task name.

Regarding this:

> ... the real solution will be some sort of hybrid between a neural net and some sort of finite state automata.

From https://datascience.stackexchange.com/a/25819:

> There is research showing that a DNN can simulate any FSM. Since there are more frameworks for DNN and DNN can perform more tasks than FSM, it appears to be more useful just to forgo FSM altogether.

> More specifically, "Neural network for synthesizing deterministic finite automata" shows how a relatively simple neural network (NN) can quickly and automatically learn the correct deterministic finite automaton (DFA).

> some sort of finite state automata.

Why not go one step further and have build in game engines that will work like imaginations of future states based from interactions within and between engines.

Reply on news.ycombinator.com