A score can look impressive while leaving the practical question unanswered: can I rely on this result? Aakanksha Chowdhery’s Stanford CS329A lecture of November 17, 2025 provides the starting point for this reading. Three evaluations help separate the questions. METR relates task duration to a stated success rate. GDPval compares deliverables with professional work. DeepScholar-Bench examines research synthesis, retrieval, and citation support. Their scores concern different tasks and do not form a single scale of capability. Before handing work to an agent, ask which requirement the test measures and which it leaves open.
Keeping score for systems that act
The old scoreboards graded single answers. Agents have to be graded on what they finish, the quality of the work, and whether anyone can check it.
A multiple-choice score alone cannot show whether a system will complete a task that involves tools. An agent may browse, write code, run it, and revise the result. For that kind of assignment, evaluation needs to examine the outcome and the steps that matter to it.
This reading compares three evaluations. METR relates completion to human task duration at a stated success rate. GDPval compares deliverables with professional work. DeepScholar-Bench examines research synthesis and its sources. These perspectives help frame different questions about a task; none supplies a universal score.
The stopwatch and the success rate
METR relates success on its tasks to human completion time. Its fitted horizon always needs a stated success threshold.
METR’s March 2025 study examines multistep software and reasoning tasks, using completion times from people with relevant expertise. It fits success curves against those human durations. The time horizon is read at a chosen success threshold. Its value depends on the tasks and human baselines used; it does not describe every occupation.
On that axis, the curve is steep enough to invite extrapolation. In the historical models examined by METR’s March 2025 study, the 50%-success horizon doubled roughly every seven months, an estimate reported by the research team. Follow the line forward and it leaves the workday behind, then starts eating the work week. Whether that trend continues remains uncertain.
A model that succeeds half the time is different from one that succeeds nineteen times in twenty. The success threshold therefore belongs beside the time horizon. A result at 50% does not tell us the horizon at 95%. These are questions about task completion; GDPval asks a separate question about the quality of the resulting work.
The paycheck test
GDPval, built at OpenAI, asks a blunter question than any exam: is the output good enough to bill for?
GDPval compares deliverable quality against professional work. Economic relevance describes the tasks; the score is not a dollar valuation. GDPval assembles 1,320 tasks drawn from real occupations, the deliverables people are actually paid to produce, with 220 of them released as an open gold subset for anyone to inspect. A task here is not a puzzle; it can be a memo, spreadsheet, or design. A benchmark result does not guarantee that a client would accept it without editing. And the grading question, stated out loud, is substitution.
Good enough is a judgment call, so the benchmark leans on human graders who compare model output against professional work. Their consistency carries its own unlovely statistic: inter-rater agreement, the degree to which two graders judging the same output reach the same verdict. Agreement measures consistency between graders. It does not, by itself, establish that their judgment is correct or that the benchmark measures the intended capability.
One number from this benchmark matters more than any win rate, and it is not a score but a slip: what happens to the score when the instructions get thin.
The cost of the thin brief
GDPval tested shorter instructions in one GPT-5 experiment. That experiment tests sensitivity to the information supplied.
GDPval’s appendix A.2.7 tests GPT-5 with shorter prompts on an earlier gold subset. Human grading found weaker performance with less context. This was a specific experiment, not a general loss rate for all agents.
People acquire context through meetings, experience, and conversation. That takes time. An agent may need explicit instructions or access to that information. Comparing its output with human work therefore requires asking who prepared the context and whether that preparation is included in the cost.
A benchmark that ignores the briefing is measuring the wrong worker, and the third ruler changes the subject accordingly: not what the model produces, but whether anyone can check it.
Show your sources
DeepScholar-Bench evaluates a concrete assignment: a related-work section with sources that support its claims.
DeepScholar-Bench, developed by researchers at Stanford and UC Berkeley, asks systems to write related-work sections for academic papers. That is a specific research-synthesis task. Its automated evaluation separates synthesis, retrieval quality, and verifiability. Finding a relevant source and using it to support a claim are different requirements.
DeepScholar’s results do not show that four out of five reports failed an audit. In the paper’s first version (August 2025), no evaluated system exceeded 0.19 on every metric; several individual scores were higher. A revised version (February 2026) reports different figures for its systems. Those are separate measures, not a single report pass rate.
Verifiability is what turns a language model from a raconteur into an instrument. A research agent that invents citations is worse than no agent at all, because it launders error into authority; the audit is the product. The practical question is whether the cited evidence supports the claim in front of the reader.
What to watch on the dashboard
These evaluations offer different views of performance. The useful question is which view fits the task.
The wager underneath all three rulers is that the era of the single score is over. A model’s standing now has three coordinates: which task durations it can handle at a stated success threshold, how its deliverables compare with professional work, and whether its claims survive checking. The benchmarks examine different tasks and populations. Their scores do not establish a common ranking of progress. A dashboard, unlike a leaderboard, tells you where the bottleneck lives.
there remains the possibility of substantial model error.METR · 19 March 2025
This warning concerns the forecasting model: extending a historical curve into the future may be wrong. It is not a measured error rate for every AI agent.
Three questions stay open. The first is whether the doubling holds as horizons stretch past the workday, because long curves have a way of bending. The second is whether reliability can be trained upward as fast as reach, or whether it is a different kind of problem entirely. The third is whether anyone builds the missing fourth ruler, one that prices the briefing itself, the context work people still contribute. Before relying on an agent, the reader needs to choose which requirement matters for the task and check whether the evaluation actually tests it.