I suppose I should be more up-front about "yes, I am rather influenced by the LessWrong/AlignmentForum views on AI risk". Also, it sounds like I may be over-estimating how commonly held some of their views are?
(Though, I generally consider the chances of AGI actually ever being developed, to be much lower than they do, but for primarily religious reasons (as a reason that it wouldn't, not couldn't, happen), and am interested in their work.)
My impression of the people on AlignmentForum wasn't that they were confident that "just scale up neural nets and use the right training procedure" couldn't work, but yes that there is a qualitative difference between what we have, and something which is as general-purpose as a human (not necessarily as competent, but capable of reasoning about the world and it being embedded in it, etc.).
Personally, I find it hard to imagine that there could not be a qualitative difference in at least the training procedure. But also, (while it might not be the most efficient way to achieve it, even by many orders of magnitude, or it could be, idk) I find it hard to imagine that it would be impossible for a neural net, with the appropriate architecture and training procedure, to form one.
(At the least in a trivial sense of "you can represent whatever known computation you want with something that could at least somewhat-reasonably be called a neural net")
____
Regarding that history of Goertzel's views and such, that is interesting, thank you!
Yes, I suppose falling in with the LW/Miri/AlignmentForum camp and Goertzel, I would agree that gpt3 isn't a general intelligence. I've been perhaps so mired in that background that I, find it hard to imagine disagreeing with that, so perhaps I need to seek out some other views?
So, I guess maybe I need to better understand the views of researchers at deepmind and facebook, etc.
I'm aware that DeepMind had their paper about training agents in diverse environments, which I thought was cool.
This to me, and things like it, would seem to me to be qualitatively different (in e.g. how it is trained) than e.g. gpt3 , even though both are neural nets.