The fastest draft can leave the slowest review queue.
AI coding can accelerate some work and add verification effort to other work. Measure the complete path from a task to a reviewed, deployed change. Earlier research does not establish a universal 2026 slowdown or a universal speedup. The practical objective is more accepted useful work without creating an unmanageable review, security or maintenance burden.
What changed—and what the sources establish
METR’s early-2025 randomized study involved 16 experienced open-source developers completing 246 tasks in repositories they knew well. AI access made tasks take 19% longer in that study setting. It was not a measurement of every developer, every task or October 2026 tools. METR early-2025 developer productivity experiment.
In its February 2026 update, METR explains that task/developer selection and measuring concurrent agent use make later estimates difficult to interpret. It believes newer tools likely help more, but describes its later data as weak evidence for the size of that improvement. METR 2026 experiment-design update.
DORA’s 2025 report examines AI-assisted development alongside organizational capabilities. Its framing of AI as an amplifier supports examining testing, delivery and team systems rather than attributing every outcome to the model alone. DORA 2025 AI-assisted software development report.
Choose the approach that fits the constraint
Match assistance to how clearly the task and acceptance conditions can be expressed. These suggested choices are an engineering evaluation framework, not results from a new benchmark.
| Situation | Start here | Tradeoff / acceptance check |
|---|---|---|
| Small bounded transformation | AI draft with focused regression check | Verify semantics, not just formatting |
| Unfamiliar architecture change | Design and codebase investigation first | Review dependencies and ownership before editing |
| Security-sensitive boundary | Specialist review and adversarial checks | Require authorization, data isolation and failure behavior |
| Large autonomous patch | Split into reviewable increments | Measure integration effort and rollback readiness |
Read the productivity evidence without flattening it
The early METR experiment matters because it compared actual task completion under randomized access to AI rather than relying only on developer impressions. Its scope also matters: experienced contributors working in mature repositories have substantial context that an assistant must reconstruct. A tool that produces plausible code can consume time when the developer must inspect suggestions, correct assumptions and reconcile them with project conventions.
The later update is equally important. When developers prefer AI for particular tasks and are reluctant to accept a no-AI condition, the sample can change. Concurrent agents also blur active work and elapsed time. Do not cite the early result as proof that today’s assistant slows everyone, or cite a later point estimate as a settled productivity gain. State the population, task mix, period and uncertainty.
Instrument the whole delivery chain
Define an accepted change before measuring speed: requirements satisfied, relevant checks passed, reviewer concerns resolved and the release observable. Track author time, review time, waiting time, integration corrections and post-release defects separately. Generated lines are an output of the tool, not a unit of business value. A large patch can make the author appear productive while consuming the reviewer’s entire afternoon.
Use task categories that your team recognizes: bug repair, test design, routine migration, documentation, incident diagnosis and feature implementation. Compare similar tasks and account for experience with the codebase. Keep security and reliability outcomes visible beside throughput. Do not quietly count a reopened ticket as another successful AI-assisted task; connect rework to the original change so the economics remain interpretable.
Make verification cheaper before making generation broader
Give the assistant a bounded objective, the relevant architecture boundary and an acceptance example. A focused test should fail on the original defect and pass after the repair. For changes with state or money, include failure recovery and authorization checks. Ask for a small diff and a concise rationale that lets a reviewer understand the behavior before inspecting every implementation detail.
Automate dependable checks: types, formatting, dependency boundaries and meaningful regression tests. Add test coverage where it protects a real behavior rather than mirroring the implementation. Require an independent review path for consequential changes. A second model’s agreement is not equivalent to runtime evidence. The release decision needs actual test output, code review and monitoring that can detect a regression after deployment.
Worked example: illustrative assumptions, not a reported result
Illustrative task economics: without assistance, implementation takes 120 minutes and review takes 30, totaling 150 active minutes. With AI, drafting takes 60 minutes, but correction takes 35 and review takes 45, totaling 140. The net improvement is 10 minutes, or about 6.7%, not the 50% reduction suggested by comparing drafting alone.
If an escaped defect later adds 90 minutes of incident and repair work, total attributable effort becomes 230 minutes. This is an invented example, not a measured effect size. It shows why a team needs a common definition of completion and a way to attach later rework to the originating change. Record both active work and elapsed lead time; neither should silently substitute for the other.
Failure modes to catch before scaling
- Review saturation: agents produce changes faster than maintainers can inspect them. Cap work in progress and reduce diff size before adding more agents.
- Passing tests with wrong behavior: generated tests repeat the implementation assumption. Validate acceptance against the original requirement and a meaningful negative case.
- Benchmark substitution: a vendor coding score becomes a team productivity forecast. Evaluate the actual repository, task mix, review requirements and release process.
Software quality is a portfolio of behaviors, not a single metric. Faster delivery with more incidents is a different result from faster delivery with stable service outcomes. Discuss the tradeoff explicitly.
A concrete next step
Select a small set of recurring tasks. Record implementation, review and correction effort under your current workflow, then evaluate AI assistance using the same acceptance standard and track reopened work for a defined follow-up window.
| Owner | Engineering lead owns acceptance; reviewers record integration effort |
|---|---|
| Unit | Accepted change with linked requirement and release evidence |
| Guardrails | Escaped defects, security findings and review backlog |
| Decision | Expand only for task categories with supported net benefit |
Keep the protocol simple enough that developers actually use it. Capture a task’s start, accepted state and rework instead of recording every keystroke. Report sample size and task mix. If the data is sparse or selection changes, say so and resist turning a local observation into a universal productivity claim.
Practical questions
Did METR prove AI makes all developers slower?
No. Its early-2025 study found a 19% slowdown among 16 experienced open-source developers on 246 tasks in their own mature repositories. That finding has a defined population, tool period and task setting.
Did METR’s 2026 update prove a universal speedup?
No. METR describes substantial selection and measurement problems and says the later data is weak evidence for the size of improvement. It is a reason to update the interpretation, not replace one universal claim with another.
What should replace lines-of-code productivity metrics?
Measure accepted useful changes, including implementation, review, correction and attributable post-release work. Track lead time and quality separately so a faster draft cannot hide an expensive integration.
Should every AI-generated change receive more tests?
Use meaningful verification appropriate to the risk. A test that repeats an implementation assumption adds little assurance. Prioritize behavioral acceptance, security boundaries, failure recovery and relevant regressions.
Primary sources checked 11 October 2026
- METR early-2025 developer productivity experiment — The study involved 16 experienced open-source developers, 246 tasks and found tasks took 19% longer with early-2025 AI tools in that setting.
- METR 2026 experiment-design update — METR says later selection effects and concurrent-agent time measurement make later speedup estimates weak evidence.
- DORA 2025 AI-assisted software development report — The report examines AI-assisted development in organizational context and treats AI as an amplifier.
The calculations in this guide are illustrative planning examples. Product documentation establishes supported features and conditions; it does not establish a measured return for your organization. Recheck account availability and release notes when implementing the recommendation.
Continue with Agent execution reliability and Multi-agent coordination.
Research Path