Hacker News Reader

kostajon Disagreement among frontier LLMs on real-world fact-checks

Yes, they are much closer verdicts. True and Mostly True are also close. Used Krippendorff's α (ordinal) to not penalize much closer disagreements. 21% of the claims have models that are on the polar opposite sides - at least one True, and at least one False.

kostajon Disagreement among frontier LLMs on real-world fact-checks

sonar-pro for the retrieval capabilities

kostajon Disagreement among frontier LLMs on real-world fact-checks

Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.

kostajon Disagreement among frontier LLMs on real-world fact-checks

It's an omission on my side. Will add in the next version.

kostajon Five frontier LLMs disagree on 67% of 1k real-world fact-check claims

Used "No explanations, no qualifiers." to force the models to answer only with one of the four labels. It's worth running a separate test with more explanation in the prompt on how to classify between the four buckets.

kostajon Disagreement among frontier LLMs on real-world fact-checks

Data collection and processing was done manually. LLMs helped with the report drafting. Everything was human reviewed before publishing.

kostajon Five frontier LLMs disagree on 67% of 1k real-world fact-check claims

Our experience with Lenz is that forcing a multi-step process, incl. adversarial debates, helps improve the verdicts.

kostajon Disagreement among frontier LLMs on real-world fact-checks

Author here. 67% (95% CI 64–70%) of 1,000 recent real user claims to a fact-checking platform had at least one of GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro+Search, and Sonar Pro dissent from the panel majority — or no majority formed at all. Panel-level Krippendorff's α (ordinal) = 0.639, i.e. nontrivial but limited agreement.

Quick context on what's in the writeup and what isn't:

- What's measured: parsed-label agreement between the 5 models. Forced 4-choice (True / Mostly True / Misleading / False), no Abstain. No LLM grader, no reference verdict — every number is direct label equality.

- What's not measured: which model is right. There's no ground truth in this paper. The 67% figure is a floor on rubric inconsistency (at least one model is label-inconsistent under the 4-bucket rubric on 67% of claims), not "model X is factually wrong on claim Y."

- Why not AVeriTeC / PolitiFact / SimpleQA: those have been public for years and almost certainly appear in current frontier training data, so measured disagreement on them confounds inference with memorization. This corpus is structurally fresh — recent user submissions, 180-day window, near-duplicates collapsed, never paired with canonical verdicts in any public training set.

- Our own platform's verdict is deliberately NOT used in this analysis. The paper measures frontier-panel disagreement only, not Lenz-vs-frontier.

- Follow-up in progress: human-labeling every claim in this corpus so we can evaluate both the panel and our own platform verdict against a human reference.

Critiques I'd most like to hear: (a) the iid CI assumption (Lenz claims cluster around topics and news events, so Wilson is probably optimistic), (b) ordinal-α vs alternatives for a 4-class ordered scale, (c) forced-choice vs allowing Abstain.

Permanent archive: https://doi.org/10.5281/zenodo.20344847