Yes! See basic computer science such as the halting problem, Godel’s theorem, not to mention Hofstader’s book Godel, Escher, Bach. Using a neural network does not repeal the laws of computer science as much as people are inclined to think.
However the example at the top of the page isn't quite right. You could implement something like this. An LLM's memory only exists in its context window of its own prior output[0], but you could train it to tag its "chain of thoughts" and just not show them - now it looks like it's "thinking".
[0] this is not exactly true because you could also implement non-greedy search, ie back up and start again if you think it's gone down the wrong path. Now it doesn't execute in finite time either.
I don't think an LLM can be considered as capable of becoming an AGI on its own with the current architecture, but potentially combined with other techniques, models and supervision, I can see the potential for it to evolve in that direction. Frankly, for what LLMs are right now, they're _surprisingly_ effective. If you chopped the Broca's area out of my brain and hooked it up to an API, I suspect it wouldn't do half as good of a job.
To be fair a lot of people speculating about AI aren’t aware of the foundations of computer science, but some of them are vaguely aware and just forgot.
Actually LLMs do have a way to break some of those constraints, in their own way, in that those constraints don’t apply to systems that don‘t always get the right answer. You can sort a list in O(1) some of the time but not all of the time.
People didn't give up on ZFC because of Godel's theorem. There's currently 42 or 43 unproven 5-state Turing machines we need to confirm BB(5).
We are solving the halting problem. We are solving ZFC. That's the entire point of mathematics.
Turing's proof of the halting problem being undecidable is extraordinarily vague. It shows that for any halting procedure, there is a program (without any constraints) which violate its output. It does not, for example, rule out whether you can write a Halt procedure for no-input Turing Machines with N states. In fact we have already written them for N=1,2,3,4.
Considering Chaitin-Kolmogorov complexity, of course we can't use N bits to describe unbounded information. This doesn't in any way preclude us from making an N bit program to describe halting behavior of K<N bit programs for some N.
We are not. We're determining whether particular classes of turing machines halt, and we're determining whether particular theorems hold in ZFC (and doing a lot of mathematical work which is neither). This is not evidence that the human brain is super-turing, because these are computable problems.
> It does not, for example, rule out whether you can write a Halt procedure for no-input Turing Machines with N states. In fact we have already written them for N=1,2,3,4.
Turing didn't, but later work does. BB(748) is known to be independent of ZF. The real bound is likely much lower.
Scott Aaronson has a positive view of BB(n) being independent of ZFC. He says, we’ll need different foundations to solve k>n. And the different foundations will have their own n and etc.
Page 6 here https://www.scottaaronson.com/papers/bb.pdf