The AI Validation Bottleneck

For decades, the scarce resource in software was the capacity to write code. Sprint planning, headcount models, and vendor contracts were all built around that assumption. For teams running AI agents at scale, it no longer holds. The hard part now is deciding whether code is correct, architecturally sound, and safe to release.

Generation capacity grew quickly. Review capacity did not. It is still the same senior engineers, reading at the same pace, now facing far more volume.

The cost shows up as rework. In the 2025 Stack Overflow Developer Survey, 45% of developers named 'AI solutions that are almost right, but not quite' as their top frustration, and 66% said they spend more time fixing that code. The work did not go away. It moved downstream, to a place a velocity dashboard does not show.

Steve points to a randomized controlled trial METR ran in early 2025. Sixteen experienced open source developers worked 246 real issues in repositories they already knew well. They expected AI to make them 24% faster. Measured against control, they took 19% longer. Afterward, they still believed AI had sped them up by about 20%.

The sample is small, and METR says so. In a February 2026 update, METR noted that developers are likely sped up more by today's tools, although its follow-up data was too unreliable to size the effect. The lesson Steve draws does not depend on the slowdown figure. Skilled engineers, working on code they understood deeply, misjudged their own productivity by roughly 40 points. Self-reported gains are not evidence.

Temporal's survey shows a similar split inside a single group: 85.5% of respondents trust agent output at least somewhat, while 41.1% report running into agent-related issues daily or more often.

Sonatafy calls this the AI Validation Gap: AI reaches production without evaluation, measurement, or a feedback loop anyone trusts. It is easy to think of it only as a product risk, such as a hallucination that reaches a customer. Steve's point is that the same gap now sits inside the delivery system itself, where it is much harder to see, because the output looks like ordinary code.

AI did not invent a new way for software projects to fail. It removed the friction that used to slow the familiar failures down.

None of these require buying a new platform. They require treating verification as a funded function instead of an assumption.

1. Measure instability, not only velocity. Track change failure rate, rework rate, and time to restore next to throughput. A scoreboard that only counts output cannot report a loss.
2. Put a number on review capacity. Know how many pull requests a senior engineer can meaningfully review in a week. That number exists whether anyone records it or not, and it is the real ceiling on delivery.
3. Ask for delivery data, not sentiment. If the only support for a productivity gain is that the team feels faster, it is not evidence. Ask what moved in the delivery data.
4. Name an owner for verification. When verification belongs to everyone, it belongs to no one. Give one person the budget and the authority to hold a release.

The leverage from AI tooling comes from expanding review, testing, and evaluation capacity at the same time as generation capacity. That is what lets a leader answer the only question that matters with data instead of conviction: did it work?

Steve's conclusion is that the fix is not technical. It is an accountability decision: who owns proof, how much capacity they get, and whether they are allowed to say no.

Read the full column on Forbes