If you think that there is some process in the brain that would be theoretically impossible for computers to model, that would be an interesting topic of discussion.
This article seems to be arguing "no." The biggest thing missing I think is an understanding of how human memory works.
Because the responses are human-written, they can seem locally coherent, but since the model generally isn't tracking any state, the conversation never really goes anywhere. Also, if you try to talk to one of these response rankers about something it doesn't have any canned responses for, obviously it doesn't work.
Natural language generation on the other hand, where the model writes each response character-by-character or word-by-word, has the potential to do something much more interesting, but the state of the art there isn't quite at the level of "fairly coherent" AFAIK.
Hopefully we'll figure out a more useable level of abstraction that still allows for intelligent behavior.
It seems like the argument for AGI appearing reasonably soon isn't ultimately that long. It is just the ingredients seem to be assembling and there's no fundamental limit preventing it. We're getting closer to complexity of a brain, our machines are general purpose, we're getting more of the specific capacities of the brain. How we can get there seems a fundamental unknown but it's hard to see there being a fundamental barrier.
The problem is any more detailed argument has to involve one or another prescriptions concerning exactly how to create AGI. And such arguments are going to be much more controversial.
And the brain really isn't that great. Signals travel through the brain about a million times slower than electrons through silicon. Neurons are large macroscale structures. Compare to our transistor technology that is approaching the limits of how small you can build things with atoms. These are just the hardware specs, I don't see any reason to believe the software is much better.
Correction. Electrons' drift velocity in conductors is quite slow and it doesn't affect signal transmission speed.
We have AI algorithms today like the DeepMind Atari playing stuff that are general and so you could call AGI but that are way below human intelligence. Basically things advance every year in hardware and software and so will probably get there one day.