> I could essentially replace it with Google for basic to slightly complex fact checking.
I know you probably meant "augment fact checking" here, but using LLMs for answering factual questions is the single worst use-case for LLMs.
I know you probably meant "augment fact checking" here, but using LLMs for answering factual questions is the single worst use-case for LLMs.
Once you get an answer, it is easy enough to verify it.
As other posters said, relying on LLMs for factual answers to challenging questions is error prone. I just want the LLM to give me the links and I'll then assess veracity like a normal web search. I think a web search interface allowed disambiguating multi-meaning keywords might be even better.
You’ll still want to fact check it, and there’s no guarantee it’s comprehensive, but I can’t think of another tool that provides anything close without hours of research.
I will say LLMs are great for taking an ambiguous query and figuring out how to word it so you can fact check with secondary sources. Also tip-of-my-tongue style queries.
In 2025 Google is trying very hard to serve the most profitable results instead, so it'll latch onto a random keyword, completely disregard the rest, and serve me whatever ad-infested garbage it thinks is close enough to look relevant for the query.
It isn't exactly hard to beat that - just bring back the 2010 Google algorithm. It's only a matter of time before LLMs will go down the same deliberate enshittification path.
This works nicely when the LLM has a large knowledgebase to draw upon (formal terms for what you're trying to find, which you might not know) or the ability to generate good search queries and summarize results quickly - with an actual search engine in the loop.
Most large LLM providers have this, even something like OpenWebUI can have search engines integrated (though I will admit that smaller models kinda struggle, couldn't get much useful stuff out of DuckDuckGo backed searches, nor Brave AI searches, might have been an obscure topic).
The fact that it provides those relevant links is what allows it to replace Google for a lot of purposes.
For example if I’m asking about whether a feature exists in some library, the AI says yes it does and links to a forum where someone is asking the same question I did, but no one answered (this has happened multiple times).
But when I want to actually search for content on the web for, say, product research or opinions on a topic, Perplexity is so much better than either Gemini or google search AI. It lists reference links for each block of assertions that are EASILY clicked on (unlike Gemini or search AI, where the references are just harder to click on for some reason, not the least of which is that they OPEN IN THE SAME TAB where Perplexity always opens on a new tab). This is often a reddit specific search as I want people's opinions on something.
Perplexity's UI for search specifically is the main thing it does just so much better than google's offering is the one thing going for it. I think there is some irony there.
Full disclosure, I don't use Anthropic or OpenAI, so this may not be the case for those products.
Im also someone who refuses to pay for it, so maybe the paid versions do better. who knows.
Running these things is expensive, and they're just not serving the same experience to non-paying users.
One could argue this is a bad idea on their part, letting people get a bad taste of an inferior product. And I wouldn't disagree, but I don't know what a sustainable alternative approach is.
Same issue with Gemini. Intuitively I'd also assume it's trivial to fix but perhaps there's more going on than we think. Perhaps validating every part of a response is a big overhead both financially and might even throw off the model and make it less accurate in other ways.
However, when I want sources for things, I often find they link to pages that don't fully (or at all) back up the claims made. Sometimes other websites do, but the sources given to me by the LLM often don't. They might be about the same topic that I'm discussing, but they don't seem to always validate the claims.
If they could crack that problem it would be a major major win for me.
I had it consume a "deep thought" style output (where it provides inline citations with claims), and then convert that to a series of assertions and a pointer to a link that supposedly supports the assertion. I also split out a global "context" (the original meaning) paragraph to provide anything that would help the next agents understand what they're verifying.
Then I fanned this out to separate (LLM) contexts and each agent verified only one assertion::source pair, with only those things + the global context and some instructions I tuned via testing. It returned a yes/no/it's complicated for each one.
Then I collated all these back in and enriched the original report with challenges from the non-yes agent responses.
That's as far as I took it. It only took a couple hours to build and it seemed to work pretty well.
There is a working paper from McKinnon Consulting in Canada that states directly that their definition of "General AI" is when the machine can match or exceed fifty percent of humans who are likely to be employed for a certain kind of job. It implies that low-education humans are the test for doing many routine jobs, and if the machine can beat 50% (or more) of them with some consistency, that is it.
By the way, doing a better job than the average human is NOT a sign of intelligence. Through history we have invented plenty of machines that are better at certain tasks than us. None of them are intelligent.
When I need to cite a court case, well the truth is I'll still use GPT or a similar LLM, but I'll scrutinize it more and at the bare minimum make sure the case exists and is about the topic presented, before trying to corroborate the legal strategy with a new context window, different LLM, google, reddit, and different lawyer. At least I'm no longer relying on my own understanding, and what 1 lawyer procedurally generates for me.
Non niche meaning: something that is taught at undergraduate level and relatively popular.
Non deep meaning you aren't going so deep as to confuse even humans. Like solving an extremely hard integral.
Edit: probably a bad idea because this sort of "challenge" works only statistically not anecdotally. Still interesting to find out.
This was with ChatGPT 5.
I mean it got a generic built in function of one of the most popular languages in the world wrong.
See
https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
If you know that a source isn’t to be believed in an area you know about, why would you trust that source in an area you don’t know about?
Another funny anecdote, ChatGPT just got the Gell-Man effect wrong.
https://chatgpt.com/share/68a0b7af-5e40-8010-b1e3-ee9ff3c8cb...
Obviously it works for you (or at least you think it does), but I can confidently say it's fucking god-awful for me.
If I ask ChatGPT a question and it gives me a wrong answer, ChatGPT is the fucking problem.
Why don't you try the original prompt using thinking model and see if I'm cherry picking?
If it works for you, cool. I think it's dogshit.
How do you expect to find a ground truth from a non-deterministic system using anecdata?
You might want to develop a sense of humor. You'll enjoy life more.
You basically ignored all of those specifics, and spuriously accused them of cherry picking when they weren't, and now you don't want to take responsibility for your own words and are using this conversation as a workshopping session for character attacks in hopes that you can make the conversation about something else.
I'm sure if you keep repeating yourself though I'll change my mind.
JustExAWS replied with an example of getting Python code wrong and suggested it was a counter example. Simianwords correctly noted that their comment originally said thinking mode for factual answers on non-niche topics and posted a link that got the python answer right with thinking enabled.
That's when you entered, suggesting that Simian was "missing" the point that GPT (not distinguishing thinking or regular mode), was "not always right". But they had already acknowledged multiple times that it was not always right. They said the accuracy was "high enough", noted that LLMs get coding wrong, and reiterating that their challenge was specifically about thinking mode.
You, again without acknowledging the criteria they had noted previously, insisted this was cherry picking, missing the point that they were actually being consistent from the beginning, inviting anyone to give an example showing otherwise. At no point between then and here have you demonstrated an awareness of this criteria despite your protestations to the contrary.
Instead of paying attention to any of the details you're insulting me and retreating into irritated resentment.
Instead you've tried everything from saying I need to "get a sense of humor", to character attacks, to insisting without specific explanation that I "don't understand", to declaring that you "don't care", to declaring that no amount of information will make you acknowledge the inaccuracy of your own comments.
So you haven't succeeded in changing the subject of the conversation, except in the sense of turning it into a tutorial about why you can't make wrong into right with character attacks or declarations about how much you don't care.
I'm sorry but this is a lazy and unresponsive string of comments that's degrading the discussion.
I agree this is a stupid comment thread, we just disagree about why.
That was never their argument. And it's not cherry picking to make an argument that there's a definable of examples where it returns broadly consistent and accurate information that they invite anyone to test.
They're making a legitimate point and you're strawmanning it and randomly pointing to your own personal anecdotes, and I don't think you're paying attention to the qualifications they're making about what it's useful for.
Comment sections are never good at being accountable for how vibes-driven they are when selecting which anecdotes to prefer.
Joe Rogan has high enough accuracy that I don't have to fact check too often. Newsmax has high enough accuracy that I don't have to fact check too often, etc.
If you accept the output as accurate, why would fact checking even cross your mind?
There is no expectation (from a reasonable observer's POV) of a podcast host to be an expert at a very broad range of topics from science to business to art.
But there is one from LLMs, even just from the fact that AI companies diligently post various benchmarks including trivia on those topics.