So on the front end you might ask it "What is the capital of Italy", but on the back you translate that into "What is the capital of Italy, how could you factually validate that, and give me some example searches you might type into Google to check your work." Then run the searches and return them: "Here are the searches from credible databases, they say X, Y, and Z, does that conflict with your assumption?". Eventually you feed it enough data and provide enough context to the conversation and the AI will naturally head in the right direction. Computers are fast, we should be able to do all this in a matter of milliseconds.
That would require the AI to have a comprehension of facts, and the entire point is these models don't.
So no, I see no evidence that the parts exist to solve this. If anything, the fact the problem isn't already solved suggests to me there is no easy answer.
It's far beyond the scope of a few companies putting a few decades and a couple upstart tech people to the problem to make any real progress on understand and producing truth, which is why ChatGPT answers every question like a middle schooler, because it's just simulating understanding.
The accuracy of this fact is itself is subject to the very issue you are noting though! :)
I’ve done a little testing of this (using HN comments!) and ChatGPT seems to be reasonably good at distinguishing fact vs. opinion. It seems in that limited testing to be slightly more likely to view as opinion things I would view as likely fact claims but where the operationalization intended is ambiguous as opinion, but the boundary between a fact claim with an unstated operationalization and an opinion claim is fuzzy.
That doesn’t get a lot closer to checking fact claims, or identifying and deciding when and how to report supporting evidence of them, but I don’t think that distinguishing fact claims is, itsef, as far beyond reach of current technology as you are describing it.
It won't help for those cases where the facts are right, but the reasoning around them is wrong, I suppose; but critical reasoning is a generally essential skill.
[0] https://www.pcc.edu/library/research/reasons-for-citing-sour...
Statistical generation of text doesn’t work in a way which makes “providing citations” a straightforward thing; It’s not reading a bunch of text, making notecards on ideas, and then using the notecards to build a written response where ideas directly come from sources that are quoted or paraphrased.
“Providing citations” can be a goal for the output, but its not much in the way of an an answer to “how”. And you can ask some of the current tools to provide citations, and when they do it at all, they can hallucinate citations just as well as anything else. (This has been seen in many of the attempts to get ChatGPT, etc., to produce legal arguments.)
Now, checking citations, to some degree of utility, is probably tractable if the cited material is available, as AI’s are good at summarizing text, so comparing summary of a passage with a citation against the summary of the cited material should be possible, but I don’t think any AI system yet has found a way to provide good enough citations (even with some occasional hallucinations) that citation checking to raise flags on the bad ones would be enough to get us to useful results. The challenge is how to both recognize when citations are appropriate and generate good ones a high percentage of the time.
It's a great party trick though.
FWIW, I prompted ChatGPT with "Describe how tail recursion works in programming, and why it is important. Include citations." and it actually included two relevant links in its sources, but did not indicates which part of the text depended on them.
Nor should you, as a consumer. I am just saying that, in the discussion of how to get where we want to be, “provide citations” is much more a description of the destination than the route we are trying to map.