Faster Is Not the Same as More Capable

A workflow takes forty minutes instead of an hour. The result is clean, the queue moves, and the value appears legible. Time saved can be a useful productivity measure, especially when quality and downstream costs are also visible. It becomes dangerous only when it is asked to answer questions it never measured.

Did the person become better at the underlying work? Could they perform it when the tool is unavailable? Did they learn to recognize a novel failure, or did the system’s fluent framing make error less visible? Speed contains none of those answers by itself.

The distinction matters because an organization can become more productive today while weakening a capability it will need later. It can also become faster while people learn more quickly through better feedback. Both trajectories are possible. Leaders need measures that can tell them apart.

METR ran a randomized controlled trial from February through June 2025 with 16 experienced open-source developers completing 246 tasks in mature repositories they knew well. Tasks were randomly assigned to AI-allowed or AI-disallowed conditions. With the early-2025 tools used in the study, allowing AI increased completion time by 19 percent.

The perception gap was as striking as the result. Before beginning, participants expected AI to reduce completion time by 24 percent; after the study, they estimated that it had reduced time by 20 percent. In this narrow setting, experienced people misread the direction of the productivity effect.

This does not show that AI generally slows software development. The sample was small, the work involved familiar and mature repositories, the tools reflect an earlier capability period, and METR explicitly cautioned against broad generalization. It does show why self-reported speed and benchmark capability should not substitute for direct measurement in the actual work.

In February 2026, METR reported that its newer experiment could not provide a reliable current estimate. Developers increasingly declined to participate because they did not want to work without AI; compensation changed; and concurrent agent use made time measurement harder. The raw results suggested some speedup, and METR judged it likely that early-2026 tools were more helpful than the earlier systems, but described the evidence for the size of that improvement as very weak.

A tidy article would preserve the memorable slowdown and move on. A useful one keeps the update beside it. Tool capability changed, work habits changed, and the people willing to accept random assignment changed. The measurement problem moved with the technology.

That is the larger management lesson. AI productivity is not a permanent property of “knowledge work.” It is an empirical result produced by a particular system, person, task, context, and point in time. Measurement needs a review date.

Quality asks whether the output is accurate, useful, and fit for consequence. Learning asks whether assistance improved the person’s underlying mental model or merely completed the present task. Retention asks what remains after time passes or the assistance is removed. Transfer asks whether the capability travels to an altered situation.

Oversight adds another test. A person may produce excellent work with AI while becoming less able to recognize when the AI is wrong. That capability can matter even if unassisted production is rarely required, because review and escalation depend on an independent basis for doubt.

These measures can move in different directions. Assistance might improve speed and immediate quality while leaving later unaided performance unchanged. It might slow an expert today while building facility with tools that pay off later. It might raise novice performance and narrow a skill gap. One productivity number cannot adjudicate among these possibilities.

Medical research offers an early warning from a setting where unaided recognition can carry high consequence. In one observational colonoscopy study, unassisted adenoma detection declined after regular exposure to AI-assisted detection. A 2026 scoping review found deskilling evidence scarce but present across several clinical domains.

Those results do not transfer automatically to software, finance, operations, or executive work. Clinical tasks have distinct training, perceptual demands, feedback, and regulatory conditions. The studies also do not show that productivity gains themselves caused skill loss.

They support a narrower proposition: where retained unaided capability matters, it deserves its own longitudinal measure. The cost of that measurement should be proportional to consequence. Not every assisted spreadsheet needs a manual-proficiency regime; a critical diagnostic or escalation capability may.

Measure the assisted task first: completion time, quality, error correction, and total downstream effort. Then create a separate opportunity to perform unaided, ideally after a delay and on a case that varies the surface details. Compare error recognition as well as final answers.

For oversight roles, seed plausible errors and incomplete evidence. Ask the person to form an initial view before seeing the AI recommendation. Track whether they notice when the system crosses a decision boundary and whether escalation improves the outcome. Repeated measures can show whether capability is stable, improving, or decaying.

The aim is not to preserve manual work ceremonially. If the organization no longer needs unaided production, it may choose not to maintain it. That should be a conscious risk decision made after identifying the recovery and oversight capability that would disappear with it.

Successor development contains the same measurement trap in human form. A successor makes a strong call while the incumbent supplies context, introduces the right stakeholder, and quietly corrects the frame. The result is good. The evidence about independent readiness remains incomplete.

The Passage separates supervised practice from progressively independent authority. Historical cases, altered scenarios, real decisions, and explicit escalation boundaries can help a board see what the successor carries alone and what still depends on the incumbent. This is an analogy in measurement, not a claim that human learning and AI assistance operate through the same mechanism.

Faster work may be genuinely valuable. So may stronger assisted performance. The disciplined move is to name which result has been demonstrated—and resist spending evidence about today’s output as though it proved tomorrow’s capability.