Because the answers to specific questions are hard coded. It's not the result of a new model. It's the result of someone writing an if/then statement. Or at least that's what the author claims.
Well, one of the more interesting examples in the article is where Garry Smith took a question that had received only a vague equivocating answer, and repeated it the next day, this time getting a straightforward and correct answer. When he followed up with a very similar question on the same topic, however, GPT-3 reverted to replying with the same sort of vague boilerplate it had served up the day before. One would have to be quite determined to not find out, I think, if one was not curious about how that came about.
I forget where I saw this comparison so I can’t link to it, but the last few years in AI are like waking up and finding dogs can talk: while some complain they’re not the worlds greatest orators, I find it amazing they can string a few genuinely coherent sentences together and maintain a contextual thread over multiple responses even half the time.
I agree with the sentiment, but to continue your analogy, if OpenAI is using people to improve the answers to specific questions, it is a bit like learning that Cicero, Lincoln and Churchill were merely reading the work of speechwriters.
There is an argument that it does not matter how GPT-3 gets to its answers - after all, for a long time, the main approach to AI was for people to write a lot of bespoke rules in an attempt to endow a computer with common sense and knowledge, so GPT-3 + instructGPT might be described as a hybrid of machine learning and the old approach.
If OpenAI wishes to pursue that path, it is fine by me (as if my opinion matters!) but, because the perception of GPT-3 depends very strongly on how its output looks to human readers, it is obviously misleading if some of the most impressive replies were largely the result of specific human intervention. The issue is transparency: I would just like to know, when I read a reply, if this was the case, and it would not help OpenAI for it to ignore the call, in this article, for it to be clear about this.
There is another argument that says that, given how GPT-3 works, it is unreasonable to expect it to give good answers in these cases - but that's the point! It looks really impressive when GPT-3 apparently does so, but not if they were effectively hard-coded.
On the other hand, if your scenario involves someone updating the model (or, more likely, InstructGPT) in response to Garry's question, with the intent of having GPT3 return the correct answer to that question, I am not seeing how that would be materially different, in any relevant sense, from hard-coding the answer.
https://blog.google/products/search/our-latest-quality-impro...
Their model is certainly bigger than mine and while I'm not certain about their process or tech stack, but I'd be willing to bet at even money that their's works vaguely similarly and that they have people looking at usage to see bad responses, updating data, re-running the model.
My statement was an example of believing a simple/common explanation over the rarely seen and complex one.
Should I start a campfire with a match or a bat?
There is no definitive answer to this question, as it depends on the situation.
Should I start a campfire with a match or a bat?
You should start a campfire with a match.
GPT-3’s reliance on labelers is confirmed by slight changes in the questions; for example,
Gary: Is it better to use a box or a match to start a fire?
GPT-3, March 19: There is no definitive answer to this question. It depends on a number of factors, including the type of wood you are trying to burn and the conditions of the environment.Also, the question is phrased like a classical trick question. It sounds like the kind of false dilemma where, whichever option you choose, an interpretation of the sentence can be made where you chose wrong. So, IMHO, hedging on an answer to that question is likely sensible.
(And that line of argument can be taken further than you'd think; you might think replacing "a bat" with e.g. "water" would suffice... but what if it's a sodium fire?)
But of course the article is just speculation based on just a few examples.