What the AI coding studies actually measured

A coding-productivity claim needs two dates. One is when the paper appeared. The other is when people actually used the tools being studied. Treating the publication year as the tool’s vintage can make an old experiment look like a current verdict, especially when the surrounding products change quickly.

It also needs a denominator. More completed tasks, less time per task and a higher self-reported value of work are different outcomes. All can matter. They cannot be exchanged freely in a debate simply because each is expressed as a percentage or a multiplier.

The research sequence below is a useful corrective to both enthusiasm and dismissal. It does not produce one number that a manager can paste into a staffing plan. It shows why a serious local question has to specify the work, the tool, the comparison and the result worth measuring.

2025 publication, earlier company experiments

A paper by Cui and colleagues, available through MIT, pooled three field experiments involving 4,867 developers at Microsoft, Accenture and an unnamed Fortune 100 company. Access to a coding assistant was randomized. The preferred estimate was a 26.08% increase in weekly completed tasks for tool users, with a standard error of 10.3 percentage points.

The number is more specific than a claim that every developer became a quarter more productive. The estimate used an instrumental-variable approach because access and actual use were different. The paper also describes noisy individual experiments and differing rollout arrangements. Its tool suggested code completions; a present-day multi-agent development workflow is a different intervention. The Microsoft Research publication page dates its listing to June 2025, while the linked draft is dated February 2025.

There is an important distinction between an offer and its uptake. Randomly offering a tool can support a causal comparison, but some eligible people may barely use it. Estimating the effect of access answers one question. Estimating an effect for users under additional assumptions answers another. A summary that omits the distinction can make a careful experiment sound like a different experiment.

Completed development tasks also need interpretation. They are observable workplace output, not a direct measure of customer value or of all future maintenance costs. A team can legitimately care about completion while also asking whether the completed work meets its requirements. Adding those questions does not invalidate the original outcome; it prevents the outcome from carrying claims beyond its scope.

July 2025: a different setting produces a slowdown

METR’s early-2025 open-source developer study recruited sixteen experienced contributors working on repositories they knew well. Across 246 real issues, tasks were assigned to allow or disallow AI. The tools were those available during the early-2025 study period. The reported result was 19% longer completion time when AI was allowed.

The setting matters: experienced maintainers, familiar codebases and the standards of work intended for those repositories. The authors explicitly warn against treating the finding as proof that AI fails to speed up most developers or cannot improve later. Their observed result also differed from participants’ impressions of speed. That contrast supports caution about intuition as a measurement instrument, rather than a conclusion that developers’ experiences have no value.

This result need not cancel the company experiments. Familiarity with a repository can change which parts of a task are easy for a person and difficult for an assistant. Tool capabilities, work organization and the outcome being counted differ as well. A disagreement in results becomes more informative when the comparisons are unpacked instead of put into opposing columns labeled good and bad.

Consider an expressly hypothetical developer who can implement a small familiar change in twenty minutes. An assistant might draft it in two minutes but require twenty-five minutes of inspection and correction. The same assistant might save substantial time on unfamiliar scaffolding that would otherwise take much longer. These invented durations explain a possible mechanism; they do not estimate either study’s underlying effects.

The review burden is especially important because generated code arrives looking complete. A person can spend time discovering which assumptions it made before deciding what to retain. That work may be worthwhile, and it may improve with better context or tools. It still belongs in the task’s completion time if the intended outcome is reviewed, usable code rather than a first draft.

February 2026: even the experiment’s participant pool has changed

METR’s February 2026 update described a follow-up begun in August 2025 and judged its signal unreliable for the current productivity effect. Developers were increasingly unwilling to participate without AI, and some withheld tasks they did not want assigned to the no-AI condition. A lower compensation rate and difficulty timing concurrent agent work complicated interpretation further.

The update reports raw estimates leaning toward faster completion in the later sample, with substantial uncertainty, and argues that selection may omit people and tasks expecting large benefits. It does not establish a clean universal replacement for the earlier slowdown. The change in the ability to recruit a comparison group is itself consequential evidence about how the work environment evolved.

This is a subtle problem for a rapidly adopted tool. Suppose the people who expect the largest benefit are least willing to give it up for an experiment. The remaining participants may be perfectly honest and the assignment perfectly random within their group. The experiment can still provide an unrepresentative picture of the broader population because entry into that group was selective.

Randomization does not automatically repair selection into the sample. It helps compare conditions among the people and tasks included. Generalizing beyond them requires an argument about who is missing and why that matters. The appropriate response to the selection problem is neither to ignore the experiment nor to replace its result with an optimistic guess about the missing participants.

The same issue can arise inside a company pilot. Volunteers may be the most enthusiastic users, or they may be people with enough spare capacity to participate. The busiest team may decline, while particularly suitable tasks are either deliberately submitted or deliberately withheld. Recording how the pilot population was assembled is therefore part of understanding its result.

May 2026: ask a different question, get a different kind of evidence

METR’s May 2026 technical-worker survey examined self-reported impacts among 349 respondents. It supplies evidence about reported experience and perceived changes in the value of work. It is not a randomized timing experiment. The report itself discusses reasons to be skeptical about the magnitude of self-reported benefits.

Placed together, the studies describe different objects. The comparison below keeps the outcome beside the people and work observed.

Study and setting	Outcome recorded	How to read the result
Cui and colleagues: 4,867 company developers	Weekly completed tasks	The preferred estimate is 26.08% more tasks for tool users; it concerns the assistant and company workflows studied.
METR: 16 experienced contributors, 246 issues in familiar repositories	Time to finish real issues	Allowing early-2025 AI tools increased completion time by 19% in this experiment.
METR follow-up, reported February 2026	Task completion time	The researchers judged the signal unreliable because selection and concurrent work complicated the comparison.
METR survey: 349 technical workers, February–April 2026	Self-reported change in work value	The answers describe perceived benefits and hypothetical alternatives, rather than a randomized timing result.

The first two rows can disagree without contradicting one another: they change the population, intervention and measured outcome. The later rows add evidence about adoption and experience. Averaging the headline numbers would erase precisely the differences a team needs to understand.

The comparison a team can actually make

Start with the decision the team faces. It might be whether to provide a tool, which tasks to route through it or how to change review arrangements. Those decisions call for related but different evidence. Measuring enthusiasm for access cannot determine which review process is adequate, and measuring one code-generation task cannot establish the value of an organization-wide rollout.

Define a unit of work whose completion is meaningful. For an existing service, that could include the requested behavior, appropriate review and necessary documentation. For a throwaway prototype, the acceptable standard may be different. The distinction should be deliberate and visible, so that the prototype’s apparent speed is not sold as evidence about maintaining a production system.

Measure the work around generation. Time spent preparing context, inspecting output, testing behavior and repairing mistakes can move between people rather than disappear. If an assistant makes one developer faster while increasing another person’s review load, the team’s net effect is a separate question. A local pilot should follow the part of the workflow relevant to its decision.

It should also distinguish elapsed time from active attention. A task that runs for ten minutes while the developer completes something else has a different cost from ten minutes of continuous supervision. Concurrent agents make simple stopwatch accounting harder. Counting every simultaneous minute once per agent can inflate effort, while ignoring supervision entirely can understate it.

Task mix needs a record. If developers start attempting larger or more speculative changes, average completion time can rise even while the tool is useful. If they split work into smaller tickets, completed-task counts can rise without an equivalent increase in delivered value. Stable definitions and a description of changing work prevent a metric from becoming detached from the thing it is supposed to represent.

Quality requires its own observation rather than a moral assumption about either humans or machines. A code change can pass an immediate test and still be difficult to maintain; it can also be clearer and better documented than the available alternative. The right comparison looks at the actual artifacts and later consequences within a suitable window. It does not infer quality directly from how the code was produced.

What should survive the next tool release?

An old effect estimate can become less relevant without the original research becoming worthless. Its design, scope and explanation of possible mechanisms can still help frame a new test. A later result should be dated just as carefully; novelty is not a guarantee of representativeness, and a fresh model name does not erase persistent workflow constraints.

Keep the local configuration alongside the result: tool and model versions where known, permissions, task categories and review expectations. That makes a future comparison possible. If several parts of the workflow change at once, attribute the result to the combined change rather than announcing that the model alone caused it.

Learning can change the comparison over time, too. A new user may spend the first sessions discovering how to supply context or reject an unsuitable suggestion. An experienced user may reorganize tasks around the tool’s strengths. A pilot should state whether it is measuring first-use friction, an established practice or a transition between them. The answer can affect training and rollout decisions even when the final tool subscription is the same. Comparing an expert human workflow with an unfamiliar AI workflow answers a particular question, rather than every question about long-term use.

The evidence so far supports a more useful sentence than either AI slows developers down or AI makes them a fixed percentage faster. Different interventions have produced different measured outcomes in different settings, and adoption can change the population that experiments are able to observe. A team making a consequential decision should ask which comparison resembles its own work—and what it would need to measure before trusting the resemblance.