What exactly is the architectural problem that has you convinced LLMs cannot learn how to strategize and architect well after enough RL?
Edit: I like this 10 years old blog post in regards to explaining the problem much better than I could do: https://pron.github.io/posts/correctness-and-complexity
Yes. The combinatorial explosion is a great assessment of the "why".
But the problem is more nuanced than that because one has to assume that LLMs can reason accurately all the time which is demonstrably false.
The current LLMs cannot even state true sentences as true if they sound confident, without hedging.
Future LLMs may mitigate this, though it looks unlikely.
The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too. While that suggests neither humans nor LLMs can reliably solve every planning or verification problem, it says nothing about which will do better on the software problems people actually encounter.
It certainly does not present a fundamental problem with LLMs as they exist today.
I do not mean that LLM is a classic symbolic system that generates symbols according to rules, but instead it's a system that generates symbols according to recorded probabilities. The focus here is on symbols: the LLM reasoning starts with <think> token, ends with </think> token, and everything in between is symbols (tokens). Quite different from how human brain works, where symbols are not that much involved in reasoning.
> The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too.
Yep, however I would like to assert that today we have evidence of systems architected by humans and behaving correctly (like Linux kernel), while there are no evidence of systems architected by LLMs that behave correctly. Even the systems which are architected by LLMs with human assistance (such as Claude Code or Codex) routinely have horrendous bugs (you're welcome to search "Claude Code Bug" and "Codex Bug" on this very website).
> It certainly does not present a fundamental problem with LLMs as they exist today.
The article was created 10 years ago that is why it does not address LLMs. However it asserts that mathematics modeling is not that helpful for writing correct programs. As long as LLM reasoning is closer to mathematics modeling than human mental modeling, LLMs will be worse at writing correct programs than humans. Some future technology less focused on symbolic reasoning could change that, but today it's not on sufficient level.
darkwi11ow, in the sibling comment, gave a substantive response on the underlying mechanics with a link. Godel's incompleteness theorems also yield similar insight. But I want to shine some light on a different phenomenon.
What happens in the future is up for debate, but the current architectural reasoning capabilities of LLMs are pathetic.
Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
Or try any novel architectural discussion and see for yourselves.
Modern LLMs cannot even confidently state many true sentences as "maximally" true as possible without hedging and tone policing as if we were doing politics ;).
Hence an expert is required at every turn.
Unable to see the architectural problem mentioned is just icing on the cake.
How do you do this?
>> Its not happening, at least not with LLMs
> Correct.
>> Writing such a complex software as database engine requires strategy, awareness and rigor.
> Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.
> Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
> What happens in the future is up for debate,