I expect the exact opposite from what I would qualify as AI: something that doesn't look like any plane a human would design but yet it flies!
Continuing on that parallel: it took ~400 years from the flying machines like the one designed by Leonardo da Vinci to transition from a superficial imitation of birds to a design that didn't look like birds but could actually fly.
Trying to create text that looks like text written by humans? Is that really what GPT-3 is designed to do?
I'm reminded of one of my wife's responses, on being told that a person speaks N languages fluently ... "Does s/he have anything interesting to say in any of them?"
> Trying to create text that looks like text written by humans? Is that really what GPT-3 is designed to do?
My understanding is, it's designed to attempt to "understand" the inner structure of text written by humans, and to create new text using the same structure based on its "understanding". "Understanding" is only defined as the creation of a probabilistic model - it doesn't "understand" any meaning, only the structure of sentences and paragraphs, Yet, such a mindless machinary can demonstrate a wide range of interesting behaviors.
Using the plane analogy: If you feed the airplanes designed by humans throughout history into GPT-3, GPT-3 is able to produce an original design of convincingly-looking airplanes of all nationalities and types. For example, if you say "1960, turboprop, Britain", it gives you one, and it's also capable of mixing things, such as putting a jet engine on a biplane, or designing a hypothetical WW2 Allies plane using Axis military technology. If you are lucky, some of the planes come with semi-functional systems. Some might be able to ignite its engine before it explodes, and few are even capable of taking off before it explodes (and some are unable to take off, but if you put them in the mid-air, it may be able to travel for one minute before it explodes). And it's all done under unsupervised learning, and without any domain-specific knowledge on Newtonian mechanics, aerodynamics or aviation. All it needs is an astronomical number of blueprints collection.
To me it's an impressive achievement, even if it doesn't understand how to design airplanes.
David Chalmers [0] wrote,
> When I was a graduate student in Douglas Hofstadter’s AI lab, we used letterstring analogy puzzles (if abc goes to abd, what does iijjkk go to?) as a testbed for intelligence. My fellow student Melanie Mitchell devised a program, Copycat, that was quite good at solving these puzzles. Copycat took years to write. Now Mitchell has tested GPT-3 on the same puzzles, and has found that it does a reasonable job on them (e.g. giving the answer iijjll). It is not perfect by any means and not as good as Copycat, but its results are still remarkable in a program with no fine-tuning for this domain.
> What fascinates me about GPT-3 is that it suggests a potential mindless path to artificial general intelligence (or AGI). GPT-3’s training is mindless. It is just analyzing statistics of language. But to do this really well, some capacities of general intelligence are needed, and GPT-3 develops glimmers of them.
This is done mostly because it's very easy to get a huge amount of data and score its performance numerically on that data without any manual process of deciding what the correct answer is. It turns out given enough data and a large enough network, it becomes very good at it, even to human eyes.
To me, on the contrary, GPT-3 is quite a saddening thing. With every example I come across, I get a more and more clear idea of the limits of current mainstream approaches in AI.
When it comes to predicting or achieving human-level general behavior it seems to me that the SOTA is reaching a hard limit that lies far below actual AGI.
I'm not a researcher in this field, so this opinion should be taken with more than a grain of salt, but if I'm right, 10 years from now we're still gonna be stuck at GPT-3-ish levels of clever mimicry.
I like this analogy, especially because it alludes to the Dijkstra quote [1]. In this light it's probably unfruitful to try to create submarines that mimic swimming or that look like fish - you'll just end up with fancy looking stones.
[1] "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim"
I wonder if this holds up in other languages as well.
Compared to chess AI for instance which I think still can't play like humans.