I started reading this literature while building an AI scientist (Can an AI Scientist Start With a Dataset Instead of a Goal) that scores hypotheses by how much a test result moves the model’s belief. But beliefs need calibration: a weather forecaster who says “70% chance of rain” is calibrated if it rains on about 70 of every 100 such days. The forecaster does not have to be clever; a forecaster who says 30% every single day in a place where it rains 30% of the time is perfectly calibrated and completely useless. Being useful requires a second property, usually called discrimination or resolution: the confident days and the doubtful days have to be different days. Almost every disagreement about whether language models “know what they know” comes down to which of these two properties someone is measuring.
The measurement is simple, which is why there is so much of it
Take a benchmark where answers can be checked. Ask the model a question, ask it for a confidence, record whether it was right. Repeat a few thousand times. Then bucket the answers by stated confidence and compare each bucket to its accuracy. The average gap is the expected calibration error. The Brier score punishes being both wrong and confident. And AUROC asks the discrimination question: if I pick one right answer and one wrong answer at random, how often is the right one given more confidence?
The first surprise in this literature is that base models were already decent at this. Kadavath et al. (2022) showed in 2022 that if you take a large pretrained model and ask it to answer “Is this claim true?”, the probability it assigns to the token “True” is close to calibrated, and gets better with scale. The second surprise is that fine-tuning for chat damages this. The probabilities of an assistant post-trained for helpfulness are pushed toward confident-sounding text.
That damage produced the most quoted result in this area: for aligned models, the number the model says is better than the number its own token probabilities imply. Tian et al. (2023) called the paper “Just Ask for Calibration”, and reported roughly half the calibration error from simply asking.
Asking once is worse than asking several times
Stated confidence has an obvious failure mode. Models pick round, flattering numbers. They say 90% a lot, 80% sometimes, and almost never 35%. On SimpleQA (2024), OpenAI’s factuality benchmark, every model tested overstated its confidence. But the same paper points at the alternative: sample the same question 100 times and count how often the model repeats its most common answer. Semantic entropy (2023) uses clustering answers that mean the same thing before measuring disagreement, which stops “Paris” and “It’s Paris” from counting as a conflict.
How much better is it? In Xiong et al. (2023), averaged over five tasks, chain-of-thought verbal confidence reaches 0.56 AUROC, barely above chance. Sampling five answers and measuring agreement reaches 0.73. Combining the two, sampling several answers and averaging the confidence stated with each, reaches 0.745. That ordering keeps reappearing: stated alone is weak, sampled is much better, the hybrid is best. A small-model study (2026) of small instruct models found verbal confidence at 0.554 AUROC, essentially noise, while ten samples of the same model reached 0.999 on the same questions — on the authors’ narrow short-answer benchmark, so read it as evidence that sampling helps a lot in that setup, not that sampling solves calibration in general.
Two ways of asking that work better than asking harder
The first is comparison. People are bad at absolute probability judgments and much better at ranking, and models seem to share the trait. Shrivastava, Kumar and Liang (2025) ask “which of these two questions are you more confident about?”, run the answers through a chess-style rating system, and beat both absolute stated confidence and self-consistency across five models and fourteen tasks.
The second is making the model spend its confidence. A model asked “how sure are you that X?” will happily say 80%, and if you then ask about a contradictory Y it may say 80% again. DINCO (2025) exploits this: have the model generate incompatible alternatives, ask about each one separately in a fresh context, then normalize so the answers sum to one. On the paper’s QA evaluations, ten calls of that beat a hundred calls of plain self-consistency, and the resulting scores are far less piled up at the extremes.
The wording matters a lot
One study of 17 prompt variants across 11 models (2024) found reliability “strongly depends on how the model is asked”, with small and large models preferring opposite styles. Another found that confidence is unstable across paraphrases (2024) that mean the same thing. When models are asked to use phrases like “likely” or “probably”, most map them to numbers in a roughly human way (2024). But the mapping moves depending on whether the statement happens to be true, so prior knowledge leaks into what is supposed to be a fixed scale. Human interpretations of those same words vary with context and culture (2024) too, so a “likely” handed from a model to a person is two translations deep.
Benchmarks with known answers are the easy case
All of the above needs questions someone has already answered. The harder, and more interesting, case is a claim whose truth is not yet settled: will this trial succeed, will this experiment replicate, will this event happen? A 12-model median forecast (2024) turned out to be statistically indistinguishable from a crowd of 925 human forecasters, though the models shared an acquiescence bias, predicting “yes” more often than events warranted. On ForecastBench (2024), which only uses questions that resolve after the models were trained, expert human forecasters still win. And an aggregation study (2026) shows that contamination inflated the apparent gap between frontier and small models from 8.9% to 35.8%.
What to do if you need a number
If you want a usable confidence from a model you can only send text to, the recipe from the papers above is roughly this. Ask for a probability rather than a word. Ask more than once and use the agreement between samples, which is where most of the signal lives. Combine the sampled signal with the stated one rather than choosing. If you can afford the calls, ask comparatively, or make the model divide its confidence among alternatives. Then recalibrate against whatever labelled data you have, because the raw number will be too high. And check discrimination separately from calibration, so a scaling bug does not look like knowledge and a confident constant does not look like skill.
The case I care about most is the one nobody has solved: a claim nobody has tested yet, where there is no answer to score against until someone runs the experiment.
@article{magas2026calibration,
author = {Magas, Dmitrii},
title = {When an AI Says It's 90\% Sure, How Would You Check?},
year = {2026},
month = {09},
howpublished = {\url{https://eamag.me}},
url = {https://eamag.me/2026/AI-Calibration}
}