For comparison, Nvidia's "Selene" supercomputer only has 2k a100 GPUs and is #7 fastest supercomputer in the world: https://www.hpcwire.com/2020/06/22/nvidia-nabs-7-spot-on-top... a100 GPUs are around 3x faster than v100 GPUs.
It would take some time for governments to catch up :)
2. Most of the largest super computers are owned by states.
That said, only so many actors could've secretly sourced 10K+ v100s with fast interconnect...
I don't know if you're being hyperbolic here, but if not, I consider that a massive step forward for AI.
Remember that logic had to be invented by Aristotle. It was a mechanical system meant to approximate how humans make decisions naturally.
Yes, otherwise a markov chain would be reasoning, too.
> "10 + 10 = 20. 20 + 20 = ___"
You can do the same in Prolog and similar languages. Is the language/the compiler reasoning?
> All of these go far beyond basic syntactic concerns like putting subjects and verbs in the right spots.
I'm not sure about that. The output obviously is a wonderful achievement, and will find a multitude of applications, but isn't it technically still a stochastic model describing the probability of a sequence of events, albeit at unprecedenced scale?
Reasoning needs conscious thoughts, sentience, awareness, and we don't have the slightest hints these are present in GPT. Yes, humans reason about something and then write a coherent text, but that doesn't mean that the presence of a coherent text is proof of reasoning - just as the absence of the capability to produce coherent text isn't proof of the absence of reasoning (e.g. in the horrible cases of the locked-in syndrome https://en.wikipedia.org/wiki/Locked-in_syndrome )
Yes, of course. Reason isn't magic.
Many linguists think that every human language follows fundamental patterns (e.g. [0]). In that context, the achievement of GPT is that it indirectly derived such a model by working through ungodly amounts of data. The results sound meaningful for us - but that doesn't imply that GPT intended meaning.
Every theory of reason I know has consciousness as a hard requirement. I'm not trying to be pedantic, but the topic of this thread is exactly the kind where clear definitions of words are important.
If Prolog is reasoning, then a scientific calculator is, too. But now we just need another word for the thing that differentiates us from calculators.
If I think about a process using a certain mechanism, and the AI thinks about a process using a similar mechanism, but also I have a consciousness attached on top and the AI does not, then it seems petty to assign these processes different labels based on a component whose mechanical relevance is not shown. I'm not doubting the impact of conscious, reflective reasoning on human capability, mind! But most of the thinking I do is not that.
Also as a general rule, you should be skeptical of considerations of reason that are based largely on introspection; the process is inherently biased towards consciousness as a load-bearing element, since consciousness is so heavily involved in the examination.
Yes, absolutely. The Prolog interpreter is a resolution-based theorem prover. You will be very hard pressed to find any AI researcher arguing strongly that "that's not reasoning". In fact I believe automated theorem proving is one of the very few cases were you can find so little disagreement about whether something is "reasoning" or not.
Note also that "reasoning" is not necessary to solve a problem like "10 + 10 = 20. 20 + 20 = ___" (1). Knowlege of addition is sufficient: given knowledge of addition the result of "20 + 20" can be derived without reference to "10 + 10 = 20". And, absent knowledge of addition, even if a system can answer (1), it will not be able to answer the majority of similar problems, indicating that it has only memorised the answer to (1).
A better test of what a system has learned is a question like "10 + 10 = 30. 20 + 20 = ___" (2). The answer to that should still be "40", but again that's not because of any reasoning; it's because "20 + 20" is always "40", even when preceded by a false statement. So this kind of question is really not any way to test reasoning abilities.
Edit: Actually, "Alice was friends with Bob. Alice went to visit her friend ___" (3) is not a very good test for reasoning, either. If I were to answer "Alice", would you be able to say whether that's true or false? The only way to make such a decision is in the context of a "closed world assumption" (incidentally, central to Prolog's theorem proving). However, now you're making a much more precise claim, that "GPT-3 has learned to answer questions by making a closed-world assumption". You can test this claim much more convincingly by asking questions like "Alice is friends with Bob. Is alice friends with Alice"? The answer should be "no" (or "false", "incorrect", etc).
Has this kind of more formal test been carried out, with GPT-x?
This kind of weird characterisation that people keep bringing up is totally wrong. It's more like a savant, who can generate random stories that have no bearing on reality.
It's just a bit better than GPT-2, which spat out mostly incoherent crap.
What's interesting about this is that it seems like the approach still seems to scale, and at some point, it might make something that actually generates useful output... and the ability of the model to handle general NLP tasks is a bit better.
So yeah, it's interesting, but no, it's not a massive step forward for AI, in the way having an actual 3rd grader would be.
Ouch, we've all just been burned by an AI!
And additionally he can draw, do image recognition, run circles, climb trees, pick fruits and mine rare minerals. Seems like we already have the business proposition of most AI businesses!
That's actually just a particular special case of how this model is used, where they use the model to predict "the rest" of a story. It is not the nature of the model itself, and it can be used for other things, such as compression or other NLP tasks.
In my opinion, in some regards the AI performs well above a third grade level in some areas (being able to write a complex paragraph, terminal completion, writing code) and below in some other areas (not knowing basic multiplication, doesn't understand spatial context).
If we can make an AI with an intelligence of a human, we can train that to be very good at some problem (like a human expert in some area). And then we can clone those intelligences into thousands in order to progress in that area much quicker.
A human-level intelligence is also a very important goal because it, even by definition, produces the singularity point. (AIs that can make better AIs, exponentially.)
> We don't care if it can't make a decent cup of tea.
We absolutely do, although it depends on the price at which they can do that. But once an AI can do that at all, it is only a matter of time before we can make them do it cheaply (we are very good at making things cheap once we know what things we need), at which point it becomes a driving force in our progress. Once there are self-driving cars, there will be a revolution in the labor force, for example, and driving is only a small portion of what a "human-level" intelligence is capable of.
But, on the way to AGI - which seems to me a long way off, despite some of the interesting arguments made here - it's not really true that "being very humanlike" is a big commercial advantage. As long as we have (say) less than the intelligence of a 2 year old child, or a chimp, then we'd rather have computers be doing very un-human things.
You could of course find cases where this term makes other sense (or does not at all), since English is a flexible language, but I think that in areas where we obviously discuss AI/ML, let's just use the de-facto term and make everyone's lives easier.
A common quip, but deeply inaccurate. Making productive agents who will advance your community is not cheap (neither in money nor in effort), and while some parts are fun, many parts are extremely not-fun. Productivity is clearly part of the goal of AI.
If the best AI you get is one that thinks like a 4-month-old fetus, making those is cheap and fun.
If that's your perspective, that would be considerably staggering progress.
There is an implicit goalpost for AI that laypersons have that assumes if it can't kill us all then it's not "real AI."