Language models are unsuitable for anything where the output needs to be "correct" for some definition of "correct" (code, math, legal advice, medical advice).
This is a well-known limitation that doesn't make those systems any less impressive from a technical point of view.
Can we fix it?
Because earlier today it told me that George VI was currently king of England. And I asked it a simple arithmetic question, which it got subtly wrong. And it told my friend there were a handful of primes less than 1000.
Everyone’s talking about it being a Google replacement. What’s the idea? That we train it over time by telling it when things are wrong? Or is the reality that these types of language models will only be useful for generating creative output?
If you ask a chat interface a question and it says "this is true", that's very different from a search engine containing a list of results where one of them might be untrue.
For one thing, you can look at all the queries and take a majority vote etc. Second, you can look at the source to see if it's trustworthy.