Ask a large language model to total a grocery list and it may answer with perfect confidence and be wrong. The field’s name for this is hallucination, and the remedies on offer were the blunt ones: a bigger model, or more training on worked solutions. A recent lecture at Stanford laid out a different remedy, and the results are hard to wave away. Train a second model to do one narrow job, judging whether the first model’s solution is correct, then let it sift a pile of draft answers for the one that holds up. On the benchmark the field uses for grade-school math, that arrangement matched a model thirty times larger. The newest checkers do not even wait for the final answer. They grade every step, they learn from labels that are cheap and noisy rather than gold, and they are then compressed into a single model small enough to run beside the solver it polices.
The gap between solving and checking
Language models bluff through multi-step arithmetic with total confidence, so the field built a benchmark of grade-school word problems to catch them at it.
The failure mode is not hesitation. Asked a multi-step math problem, a model will often produce a clean, confident, wrong solution, each line plausible, the arithmetic broken somewhere in the middle. The authors of the field’s standard benchmark said it formally in 2021: state-of-the-art models still struggle to robustly perform multi-step mathematical reasoning.
LLMs can hallucinate and confidently present wrong solutions to the users.
The opening this creates has a name: the generation-verification gap. Sample a model many times on one problem and the correct solution is often somewhere in the pile, even when the model could not produce it reliably on demand. The task stops being knowledge and becomes sorting: find the right answer among the drafts. That is a job for a second model.
A second model, paid to doubt
The verifier’s whole job is one number: the probability that a solution is correct. Trained well, it turns a pile of drafts into an answer.
The method, set out in 2021 by Karl Cobbe and his co-authors at OpenAI, is mechanical. Sample the generator many times on the same problem, train a verifier to score each candidate, and keep the candidate with the highest score. The verifier’s output is a single number: the probability that the solution in front of it is correct. The generator does the thinking; the verifier does the judging.
The payoff was startling. Against ordinary finetuning, meaning extra training on worked-out solutions, the verifier arrangement bought the equivalent of a thirty-fold increase in model size. And where finetuning’s gains flatten, verification scales more effectively with increased data. A bigger pile of drafts keeps paying.
The ablations in this line of work circle one design question: how fine-grained should the doubt be? A verifier can read a whole solution and score it once, score it sentence by sentence, or score it token by token, a token being the fragment of a word the model actually reads. Each option gets run as its own experiment. Before granularity, though, comes a blunter question of budget: how smart does the judge need to be?
Spend the compute on the solver
Hold the budget fixed and a big solver with a small judge beats the reverse. The no-training alternative, majority voting, has a ceiling.
Instinct says the judge should be at least as smart as the solver. The experiments run the comparison directly: hold the total compute fixed, then trade generator size against verifier size. The result is lopsided.
Using a large generator with a small verifier performs significantly better than using a small generator with a large verifier.
The rival method needs no training at all. Majority voting samples the model many times and returns the answer that shows up most. It works, until it doesn’t. The failure arrives after around fifty samples per problem: past that point, extra drafts stop adding correct answers. A verifier’s advantage is that it reads the drafts instead of counting them.
Put the findings together and a budget doctrine emerges. Spend on the generator, sample it generously, and let a small, cheap verifier do the sorting. Verification converts inexpensive sampling into accuracy that would otherwise require a much larger model. What it does not do, yet, is look inside the reasoning: the verifier reads a finished solution and cannot say where it went wrong.
Grade the steps, not the answer
Outcome supervision grades only the final answer. Process supervision grades every step, and finding out whether that matters took eight hundred thousand labels.
A verifier that scores finished solutions is an outcome reward model, ORM: it sees the answer, never the path. A process reward model, PRM, scores each step instead, so it can point to the exact line where a sound argument turns. Step-level grades also teach. Fed back as training rewards, through PPO, the standard reinforcement-learning algorithm that pushes a model toward whatever scores higher, they shape how the model reasons, not just what it concludes.
The price of the informative kind is counted in labels. For the 2023 process-supervision paper by Lightman and colleagues, human annotators graded individual steps, producing the PRM800K dataset: 800,000 step-level labels over 12,000 math problems. To keep the bill survivable, the team labeled by active learning, meaning the system picks the steps where a human grade would teach the most, rather than labeling at random. That choice made the effort 2.6 times more data-efficient than random sampling.
Process supervision is more effective than outcome supervision, but outcome reward also scales with more data.
Process supervision took the head-to-head: step-level feedback beats answer-level feedback at the same budget. But the hedge attached to that verdict does real work. Outcome rewards keep improving as data grows, and answer labels are nearly free, since answers can be checked mechanically. Step labels cannot, unless someone manufactures them. The next two designs do exactly that, from opposite directions.
A committee of weak graders
Two papers attack the cost of step labels from opposite ends: manufacture them automatically, or pool checkers too weak to trust alone.
The first escape from the labeling bill came from Wang and colleagues in 2023. Their Math-Shepherd system produces step-level labels automatically, taking the human grader out of the loop and bridging expensive supervision to cheap. The labels it makes are noisier than gold. The wager of the newest work is that noisy, properly pooled, is enough.
Weaver, from Saad-Falcon and colleagues in 2025, takes that premise literally. Its verifiers are weak on purpose, and no single one is trusted; pooled, the ensemble grades solutions well enough to measurably shrink the generation-verification gap. Then comes the step that makes it practical: distillation. One compact model is trained to imitate the committee’s judgments, so the crowd’s discernment runs at a single model’s price.
The closing results point one way: the verification strategies post real performance gains and deliver more accuracy per unit of compute than simply scaling the solver. What stays open is the contest between the two kinds of supervision, which the data has not settled: outcome rewards scale with data, and answer labels will always be cheaper than step labels, so the process grader must keep earning its bill. The thing to watch is the size of the judge. Every stage of this story has shrunk it, and the gap it polices has shrunk with it.