Does GPT-4 Understand What We’re Saying?
nautil.us
nautil.us
I think hang-up -- the automatic insistence on understanding and projection of anthropomorphic traits -- is that our instincts strongly tell us that mere statistical coordination shouldn't be capable of such feats. Maybe the closest analogy is how simple game space search could produce software much better than us playing chess. Something that seemed like it required human thought could be fully mechanized, and mechanized in a way that humans do not think, and yet achieve superior results. We're used to it now, but it must have unsettled people ~50 years ago in a similar way to how ChatGPT does.
I consider it a very pressing problem in philosophy, actually. There's the concept of the "philosophical zombie", which is a being which would look like, talk like, and by all external indicators, sound, think, and feel like a human being. But which has no sentience. The lights aren't on. It's a pure automaton.
Some philosophers are sort of torn on whether the notion even makes sense, and is properly framed. But among those who do think it is a properly defined idea, it's roughly split, according to a survey I saw a few years ago [1], on whether they are, in theory, metaphysically possible. Is there something about consciousness that is fundamental to explaining everything humans can do? Would the laws of nature, philosophy, the mind, whatever, allow such a thing to exist?
I think LLMs tend to hint somewhat at their metaphysical possibility, which is rather disturbing to many people. It's much more comforting to project understanding on to it, than to envision the possibility of mindless human-like intelligence.
As you say, the original sin is anthropomorphizing AI. We all want to do it, because it's a convenient way of looking at it and extremely gratifying to our inner science-fiction nerd. It has the unfortunate side-effect of implying something has a soul, or a lived experience that is the sum of something greater than text. It's not. Even AI trained on text and images and video is not capable of simulating a human life. The only way to accurately model the success, failure, longing and achievement of humankind is by growing into it one year at a time. Claims of "superhuman" writing speeds may well be true, and the text might even be written inscrutably well. It's not human, though, and humanity is a poor benchmark for LLM scientists that ought to know better.