Hacker News Reader

Comment by tomohelix | Hacker News Reader

The LLMs are getting better that is for sure. But I think there is an inherent weakness in the way they are made that will prevent it from ever reaching a real AGI: it can't improve by itself.

Correct me if I am wrong, but these models and their weights are static after training. So while they can appear different depending on what you prompt them with, their core system never changes. In other words, they can't adapt. They can try their best, but that is it, they can never improve, evolve, learn from their mistake, internalize new knowledge, etc.

One of the best trait of life is adaptability. That was how we human spread everywhere on this planet and how even in the most extreme of environments, you can still find some bacteria living there. And these LLMs are inherently incapable of that.

View on HN

This is a textbook example of the “AI effect”, a special instance of moving the goalposts so common in the artificial intelligence domain that it has its own Wikipedia page.

A few years ago the best chatbots basically just worked like a phone tree, trying to filter you into pre-written responses. Then GPT3 came along and was able to synthesize entire paragraphs of plausible text. ChatGPT was released only ~4 months ago and I was complaining that due to fundamental shortcomings it wouldn’t be able to play chess.

Now it seems that GPT4 can. And that was a change over only 4 months. If the last hill to fight over is whether GPT4 can self improve, well, it’s only a matter of time until GPT4 is placed into some kind of self-improving CICD pipeline.

Trust me I am one of the people absolutely baffled and pleasantly surprised by all the new things these LLMs can do. There are plenty of potential in this tech and I am not so scared to not acknowledge that.

But what I am asking is a core feature of these models. The transformer is what make LLMs possible but it is by design static after it was trained. You can't simply install another pipeline or wrap it in a program and fix this problem. From what I understand, RNN isthe self improving type of AI and it was supplanted by the transformer type and one of the key differences between them is the ability to do learning from its own output. Transformer model sacrificed that for the ability to process a large amount of data in parallel. And that is by design. You can't change it as far as I know.

I actually don't know that much so that is why I am asking if what I said was wrong. Also, maybe someone will make a new model merging both transformer and RNN. But what I am saying is right now, the type of LLMs we have probably can't learn no matter how sophisticated its training is or how many parameters it is built with.

Neither RNNs nor transformers will change their parameters during a forward pass. In production GTP just does a forward pass through the network, but never calculates the parameter gradients, and this would be true if GTP was built on RNNs as well, and without those gradients the AI will not improve.

That seems an implementation detail that would be entirely solvable with some additional work rather than a fundamental limitation though.

Unfortunately we have this thing called the curse of dimensionality and specifically we do not understand how to prevent catastrophic forgetting. This is why incrementally learning NNs are still so rare in practice.

You could periodically retrain the model using the original dataset plus the conversations from the period. Sort of a commit of the short term memory into long one.

I think you could say this effectively is what is happening within the narrow confines of any single conversation.

Although it throws away that conversation once it's done.

Ha. So, it needs to 'sleep', kind...

The comment to which you are responding is pointing out a real limitation of current machine learning systems though - most of them have a hard separation between training-time and runtime - which obviously doesn’t exist for human and animal intelligences.

Now, I’m sure sooner or later someone will come up with a system that lacks that hard separation-and it will unlock new capabilities that current systems lack - which will further close the capability gap between AIs and natural intelligences - which still hasn’t gone away, although it clearly has shrunk significantly in the last few months.

There is no hard separation. Turning on gradient updates simply destroys the weights that were already there. We don't know how to make the methods we know about work well.

There might not be a hard separation in theory, but there certainly is one in practice

They can learn from mistakes, within the context window. I often will correct GPT4 on a mistake it's made and then ask it to write me a new prompt from scratch that covers the edge case.

GPT4 has a 32k token context window. But I see this number the like the 32mb of RAM of computers of old.

With Flash Attention, the attention calculation is now basically O(n). So we could maybe someday see context windows in the billions of tokens, where "learning" is just looking back at a mistake in its past and avoiding it.

Good point. I guess it is a matter of scale. Maybe when we get to something like billions of token windows or even more, it becomes "memories" for the AIs. And they can probably make better "judgements" using these memories to train next generation of AIs to internalize these improved logics without the need to have the context fed to them.

Kinda similar to how we human do it too. It is certainly an exciting time that we are seeing so many parallels between silicon and meatbags. I didn't think I would live to see this.

The main difference between ML in its current state and biological systems is the huge compute/energy asymmetry between training and inference. Biological systems have it too, but it's much less prominent, as they have neuroplasticity which allows them to learn on the fly.

I guess the context window can be considered a (poor) substitute for the missing neuroplasticity, but it's really limited in what it can do.

When people say this, I feel like it's useful to frame it in terms of what they think "real AGI" enables, since we don't even know how "real GI" works

Before using ChatGPT I couldn't form a clear picture of that, but now it's clear that a lot of the more realistic things we imagined real AGI would enable, are going to be possible with LLMs.

I mean, GPT4 can't improve itself... but it can improve your ability to utilize it. You can take a product idea that uses GPT4, feed it to GPT4 and GPT4 will help you write a frontend for it, come up with marketing campaigns, model personas and generate user stories.

It can train other LLMs too. After they lowered the message cap, I had the bright idea to use it to generate several pages of fine tuning data for davinci to get better JSON responses. So far it's looking like a success.

It's not like 3.5 couldn't do all of this, but 4 already seems like an uncanny leap over 3.5 in that regard. And afaik we haven't established the ceiling on how much better LLMs will get as we simply spend more time training them, and compute simply gets faster.

What's preventing an LLM from coding a better LLM?

Data contamination I think. Right now training heavily depends on human to guide it to what is right and what is wrong. If an AI try to train its successor, it will basically just make a copy of itself and nothing better can be made or taught.

We are still better than an LLM, collectively. The best programmer is still miles ahead of GPT, the best writer can write something that make GPT look like a child learning the alphabet, etc. So we can still train it using the data from all of humanity.

But if we aren't here, then the best an AI can train is a copy of itself. There are already discussions about where to get more authentic data to train the next generation of LLMs when so much of the internet might have been saturated with AI generated content.

I'm imagining a Luddite using a LLM to generate terabytes of slightly wrong text to poison the water.

One could setup a blog using GPT-2 to generate the text for that purpose.

You mean like $(FAMOUS BILLIONAIRE) who claimed he would develop an AI aligned with the values of $(POLITICAL PARTY)?

Transformer based LLMs were not developed by someone sitting down in deep though.

They're a product of trial and error.

LLMs are not particularly good at reasoning. They can't make experiments. They can't look up the information in their weights (because no text on the internet has that information).

For these reasons they are not likely to be good at designing new LLMs.

Do you think it's possible an LLM might be able to evaluate if another LLM is better than itself?

it can definitely evaluate whether an LLM is better aligned than itself (for some narrow definition of "aligned") - this is the motivating principle behind Anthropic's Constitutional AI idea.

"Better" is pretty vague, but "more capable" would be difficult, because I don't think ChatGPT has a good idea of its own capabilities.

It's possible that a chatbot can be one part of the evaluation process for new chatbots. I'd bet they already are.

But they can't be the only part. New models must be evaluated on many different kinds of tasks. And if current models fails at a task, how can they evaluate new models on that task accurately?

Isn't it even a little more fraught than that? I.e, the sophistication and worth of these models are directly relative to the crude and subjective human labor of feedback upon them. That is, the RLHF part?

Hook an LLM up to an embeddings database and you’ve got something special. Tell it it can query the database and then give it back the highest ranked entries.

They do learn from their mistakes while training though. That affects their weights, so yes it’s frozen now, but it wasn’t when the model was being built.

They learned until convergence, which basically is the point where more training doesn't seem to improve output anymore. At which point they call it completed. And that is the best it can ever be. It will never get better or be made to be better unless the entire system is scrapped and retrained.

At any point though, an LLM that you can interact with is static. And that is my point. They can't improve once they are "born". And I do not see any feasible way for them to fix this as it is inherently a feature of the LLMs.

I think this is way too simplistic.

First, LLMs seem to be still on a trajectory of more parameters is better. So the convergence is a man-made decision to stop for the sake of memory usage and money spend on training.

I don't think there is anything inherent in LLMs which prohibit adjusting weights over time or branch out to another model which is more regularly updated and contains novel stuff. I would well imagine that they integrate a "news model" which is trained every 24h to incorporate text scraped from news-sites. This can be called from the main LLM.

If we look at the human brain we realize that there are dedicated systems for specialized processing (think spatial reasoning vs judging colors). I would imagine that OpenAI would eventually to have a similar more complex system structure.

A more parameter LLM is basically a new model that you have to train from scratch. You can't pick up say GPT3 and turn it into Chinchilla-grade LLMs with 175B params with just the difference in training between the two. The former has "converged" and more effort and money put into it would bring negligible benefits. The progress is basically reaching an asymptote.

And of course there is nothing saying future progress won't make a new model that can learn on the go. But right now, the transformer structure that made LLMs such a huge thing is inherently incapable of learning from its own output. Or at least, it isn't really feasible with how demanding the cost to it would be.

ChatGPT has a “knowledge-cutoff”. What if you continually fed new information (like a subscription to a news wire service) into the training process so the knowledge cutoff continually moved forward?

LLMs do support additional training (fine-tuning). Why couldn’t that fine-tuning be a continual process, absorbing new information every day?

What if OpenAI took all their records of public interaction with ChatGPT, and used that to fine-tune ChatGPT? In fact, given upvote/downvote and “regenerate response” allow users to give feedback on responses, maybe that is part of their plan?

> ChatGPT has a “knowledge-cutoff”. What if you continually fed new information (like a subscription to a news wire service) into the training process so the knowledge cutoff continually moved forward?

> LLMs do support additional training (fine-tuning). Why couldn’t that fine-tuning be a continual process, absorbing new information every day?

Definitely sounds possible. However, I suspect that it would be extremely expensive in terms of energy requirements: ChatGPT 3 training has already been evaluated to be equivalent to ~100k days of energy use of an average household in the west, iirc, and ChatGPT 4 training was undoubtedly worse.

I don't know how much energy it costs to fine-tune a LLM. But I suspect that what you're proposing would involve a cost similar to training ChatGPT 4, except every few iterations of fine-tuning.

> What if OpenAI took all their records of public interaction with ChatGPT, and used that to fine-tune ChatGPT? In fact, given upvote/downvote and “regenerate response” allow users to give feedback on responses, maybe that is part of their plan?

Given the adversarial nature of Internet, it sounds like this would be used very quickly to poison ChatGPT. Plus, let's face it, there are good chances that we, as users of ChatGPT, are actually worse than the hand-curated sources that were picked to teach ChatGPT.

Don't forget the ability to finetune the LLM.

Reply on news.ycombinator.com