Ask a language model to fix a real bug from a public software repository, and on a single attempt it succeeds about sixteen times in a hundred. Let it try 250 times, and it fixes more than half — a better score than the strongest model in the world manages in one try. Nothing about the model changed between attempts. It simply got more of them. That result sits near the center of the opening lecture of a course on large language models taught by Aakanksha Chowdhery and Azalia Mirhoseini, two researchers whose own work helped produce it. Their account of the field is that its engine has been replaced. For a decade, progress came from training ever larger models on ever larger datasets. Now it comes increasingly from the computation a model is allowed to spend on an answer after training ends. Size still matters. But the newest gains are bought at inference time, one sample, one second, one chain of thought at a time.
Two guides from inside the lab
A course on large language models opens with two of the people who helped make them large.
The course is about large language models, and the lecturers speak with unusual authority: several of the results they present are their own. The opening lecture moves through the field the way its builders remember it, as a sequence of findings that each changed what was possible. There is a course to run, with grading, a project and the usual logistics; the lecturers get to all of that at the end. First comes the story.
Their vantage is empirical. In their account, progress in language AI has come less from clever ideas than from curves: measured relationships between what goes in (parameters, data, computation) and what comes out. That lens governs everything that follows. A capability matters once it can be plotted; a technique matters once it moves the plot.
The story begins with the straightest line in modern machine learning.
The first scaling law
The field’s first engine was a 2020 finding: a model’s error falls on a smooth curve as size, data and compute grow.
A language model is trained to predict the next word, and its quality is scored as test loss: the average error on text it has never seen. In 2020, Kaplan and colleagues measured that error across many model sizes and found it falls as three things grow together — parameters, training data, compute — along a curve regular enough to extrapolate.
Predictability turned ambition into engineering. A lab could run small experiments, fit the curve, and forecast the performance of a model a hundred times larger before paying for the run. Training the largest models stopped being a leap and became a budget.
As you scale up the number of parameters in a model, they get better.
A loss curve predicts how well a model will guess the next word. It says nothing about what the model will suddenly be able to do. The surprises began there.
The abilities nobody trained in
Beyond a certain size, models do things their smaller versions cannot, and that nobody programmed in.
Give a large model a task it was never trained for, with two or three worked examples written into the prompt, and it often just does the task. This is few-shot learning: no retraining, no new parameters, only context. Small models mostly cannot do it. Beyond a certain size, they can.
As the models become larger, not only do we have this kind of predictive scaling-laws property… new kinds of behaviors appear in the model.
The most consequential of the new behaviors came from changing the question rather than the model. Asked directly for the answer to a math problem, models often guess. Asked to work through it step by step, they solve problems that standard prompting cannot. The result traces to a 2022 paper by Wei and colleagues, and it gave the field a new verb: eliciting.
Emergence delivered capability. It did not deliver obedience. A model that can do almost anything still has to be taught to do the thing you asked for.
Teaching the model manners
A base model completes text; an assistant follows instructions. The distance between them is alignment.
The first fix is instruction tuning. After pretraining, the model is fine-tuned on pairings of instructions with good responses, across many kinds of tasks. Chung and colleagues scaled that recipe, and the resulting models follow instructions they have never seen phrased before. Systems built for conversation, like the LaMDA model of Thoppilan and colleagues, come out of this pipeline.
The second fix is reinforcement learning from human feedback, or RLHF. People rank the model’s answers; the model is tuned until its answers rank well. Together, instruction tuning and RLHF are critical for aligning models with human preferences and instructions.
Alignment made models helpful and pleasant to use. It did not make them smarter. The next leap came from somewhere other than training: from trying more than once.
The second axis
Training used to be the expensive part. The new idea is to spend serious computation at the moment of answering.
Every model carries two budgets. Training is the famous one; inference, the computation spent each time the model answers, used to be the cheap one. That division has broken down. Give a trained model more time, more attempts or more room to work, and its performance rises without a single parameter changing.
The most visible form is the agent that acts in the world. Three systems show the range. Shao and colleagues built one that drafts Wikipedia-style articles from scratch. Lu and colleagues’ AI Scientist attempts open-ended scientific research. Yang and colleagues’ SWE-agent fixes real software bugs by driving a programmer’s tools itself.
The simplest form of the new axis needs no tools and no loop. It needs only a question, a grader, and 250 tries.
Ask the model 250 times
Coverage, the share of problems solved at least once, keeps rising with the number of attempts.
The study is Large Language Monkeys, a 2024 paper by Bradley Brown and a team that includes the two lecturers. The method is blunt: sample the model many times on the same problem, and count whether any sample succeeds. On SWE-bench Lite, a benchmark of real issues from public software repositories, a mid-sized coding model solves 15.9 percent of problems with one attempt — and 56 percent with 250. The strongest models in the world, given one try each, manage 43.
It seems like the models already know a whole lot more than what you get out of them when you just ask them once.
Nor is the pattern confined to code. On the GSM8K and MATH sets of math word problems, Llama-3 models reach 95 percent coverage (problems solved at least once) when allowed 10,000 samples. And the growth has the same law-like shape as the training curves: coverage rises with sample count along something close to a straight line on the right axes. An inference-time scaling law.
The catch is the one the typewriting monkeys always had. Coverage counts a problem as solved when any attempt succeeds, but somebody has to know which one. Math and code come with graders: proofs check, tests pass or fail. Where no automatic checker exists, the paper itself flags the task of picking the right answer out of the pile as unsolved.
Models that think longer
OpenAI’s o1 turned thinking time into a straight line on a graph, and the field has been redrawing it since.
OpenAI’s o1 is the reasoning model built on that idea: reinforcement learning trains it to produce a long internal chain of thought before it answers. The company’s report shows a log-linear relationship between test-time compute and accuracy. The more computation spent thinking, the higher the score. OpenAI’s published figures put o1 in the 89th percentile on Codeforces, a competitive-programming site, and at a 74 percent average pass rate on the AIME, a competition mathematics exam, with one attempt per problem.
Since then the approach has become a crowded field. After DeepSeek released its reasoning model R1 in January 2025, researchers published a wave of test-time techniques: tokens that tell a model to wait and keep thinking, rounds of written self-feedback, backtracking through reasoning trees. One project, s1, reached competitive performance with just 1,000 curated training examples carrying reasoning traces. DeepSeek’s own researchers, reviewing their path to R1, dismissed two much-discussed techniques — rewarding each reasoning step, and tree-search planning — as “unsuccessful attempts”.
How far the reinforcement-learning recipe itself scales remains, by OpenAI’s own account, under investigation. The deeper limit is the one the monkeys left behind. Repeated sampling and long reasoning pay best where answers can be checked automatically, and most human questions have no checker. So watch the verifiers. If models learn to grade their own answers in the domains that lack test suites, the second axis opens further. Until then, the straight lines bend where the checking runs out.