← Blog

Why You’re Not Seeing the AI Coding ROI You Were Promised, and Why You Probably Won’t

There is a lot of hype surrounding AI and it’s role in software development. You see it every day. In it there is a lot of misinformation and confusion. One person says it 10x their productivity, and others say it doesn’t… and chances are you aren’t seeing that 10x on your team. There’s a reason for that… and today I’m going to explain it.

Today I come not to praise AI coding tools, nor to bury them.

They work. I use them. They can make some parts of software development dramatically faster.

But there’s a mismatch between what AI coding benchmarks measure and what most software engineers actually do. And companies are making staffing, budgeting, and productivity decisions based on numbers that were never measuring their problem in the first place.

The problem isn’t that AI can’t write code faster.

It’s that writing code probably wasn’t what was slowing you down.

The power mixer problem

Think about making bread.

Suppose mixing the dough by hand takes 20 minutes. You buy a power mixer that cuts that to 10.

The mixer has made that part of the job dramatically faster, really. It works exactly as advertised.

But the bread doesn’t come out of the oven dramatically sooner.

You still have to measure the ingredients. The dough still has to rise. You still have to shape it. It still has to proof. The oven still has to preheat. The bread still has to bake.

And no matter how powerful you make the mixer, it doesn’t make the dough rise any faster.

That’s the problem with a lot of the claims about AI coding productivity.

AI can make writing code dramatically faster. But writing code is only one step in producing working software.

Stakeholders need to be listened to. Requirements still have to be understood. Existing systems still have to be understood. Architectural decisions still have to be made. Generated code still has to be reviewed and tested. Bugs still have to be diagnosed. Production still has to be operated. Customers still have to be supported.

If writing code takes 25% of the process, making that step dramatically faster doesn’t make the whole process dramatically faster.

You bought a much better mixer.

But your company sells bread.

The claim vs. the job

You’ve heard the pitch in one form or another: AI coding assistants make developers dramatically more productive.

One of the most commonly cited numbers traces back to a 2023 GitHub/Microsoft/Accenture study. Developers using Copilot completed a programming task 55.8% faster than a control group.

That’s a statistically significant result.

It’s also worth noticing exactly what the task was: implement a single HTTP server in JavaScript, from scratch, as quickly as possible, with the success criteria defined in advance.

I’m not criticizing the study. It measured exactly what it set out to measure.

But look at the task.

Greenfield. Narrow. Unambiguous. No existing codebase. No customer. No production system. No stakeholder changing the requirements halfway through. No mystery bug that only happens on Tuesdays. No architectural history to understand.

Everything except writing the code had already been handled.

Most engineers, most of the time, aren’t doing that job.

In other words, the study measured how much faster the mixer mixed.

It didn’t measure how long it took to make the bread.

A randomized controlled trial in the real world

In July 2025, the nonprofit research group METR tried the experiment on harder ground.

They recruited 16 experienced developers working in large, mature open-source repositories averaging more than 22,000 GitHub stars and over a million lines of code. These were projects the developers had personally maintained for years.

Across 246 real bug fixes, features, and refactors, tasks were randomly assigned to be completed with or without AI tools.

Beforehand, the developers predicted AI would make them about 24% faster.

Afterward, they believed it had made them about 20% faster.

The measured result?

They were 19% slower.

That number needs an important qualification. METR revisited the result in February 2026 and identified a selection problem: developers who strongly preferred AI were increasingly unwilling to participate in the “no AI” condition. A redesigned follow-up found a much smaller, close-to-neutral effect.

But something else in the original result may be even more interesting.

The developers couldn’t reliably tell whether AI had made them faster.

They felt faster.

That distinction matters enormously if companies are measuring AI productivity by asking developers whether AI makes them more productive.

Perceived acceleration isn’t ROI.

When coding gets faster but delivery doesn’t

Faros AI measured real usage across more than 10,000 developers on 1,255 teams.

Teams with high AI adoption completed 21% more tasks and merged pull requests at nearly twice the rate.

Sounds like the win everyone has been promising.

Except code review time increased 91%, nearly double the original time.

Bug counts increased 9%. Each of which required time to fix.

The result? Organization-wide delivery metrics stayed basically flat.

That’s the problem in miniature.

The coding step got faster. The system around it didn’t.

Or, back at our bakery, imagine that the new mixer lets you prepare dough twice as quickly, but your oven can still only bake ten loaves an hour.

You don’t necessarily make more bread.

You make a bigger pile of dough waiting for the oven.

That’s what a 91% increase in code review time alongside dramatically increased PR throughput looks like. The constraint moved downstream.

There’s also another cost that’s easy to miss.

Code isn’t just output. Code is inventory.

The AI can generate 500 lines in a minute. Those 500 lines now belong to your organization.

Someone needs to understand them six months from now. Someone has to debug them when they fail. Someone has to update them when an API changes. Someone has to assess their security implications. Someone gets paged at 2 AM when they behave differently in production than they did in the test environment.

Which is to say that while AI has radically reduced the marginal cost of creating code, it has not equivalently reduced the cost of owning code.

Generating more of it faster isn’t automatically productivity.

Coding was never most of the job

This is the part management tends to miss, regardless of AI.

It’s easy to picture software engineering as:

Engineer sits down → engineer writes code → software appears.

That mental model has never matched reality.

Microsoft Research’s 2025 “Time-Warp” study looked at how developers actually spend their workweek. Developers reported spending roughly 11% of their time coding, 9% debugging, 12% in meetings, 6% on architecture and design, and 5% on pull requests and code review.

Interestingly, developers wanted to spend more time coding, around 20%.

Forrester put actual coding at roughly 24% of developers' time.

A Sonar survey put writing or improving code at about 32%, with another 35% consumed by maintenance, testing, and security response.

Different samples. Different methodologies.

Same basic answer.

Writing code is somewhere around a fifth to a third of the engineering job.

While code is the visible artifact produced, it is actually the result of a much larger process.

Someone has to talk to the stakeholders.

Someone has to translate what they want into what should be built.

Someone has to understand where it goes in the existing system.

Someone has to figure out what “correct” means.

Someone has to resolve ambiguous requirements.

Someone has to debug the production problem.

Someone has to handle customer issues.

Someone has to make the architectural tradeoffs.

Someone has to decide whether the thing you’re asking the AI to build is a good or bad idea in the first place.

AI coding tools can dramatically accelerate one slice of that process. But that slice was never the whole job.

This all feels a little familiar.

The software industry has run a version of this experiment already.

During the offshoring boom in the 1990s and early 2000s, companies realized that implementation could be performed somewhere less expensive.

And sometimes it worked, usually when the work was well-specified, isolated, and low-ambiguity. The hype engine was the same: Massive delivery gain, lower cost.

Many companies, however, learned the hard way that if you didn’t have extremely clean and clear docs and processes, and the on-shore talent to manage them, the promised improvements never materialized.

Turns out the business didn’t actually know exactly what it wanted. The specification was incomplete. The interfaces weren’t stable. The people implementing the software lacked context that had never been written down because nobody realized it needed to be.

The problem wasn’t necessarily that the produced code was bad, though often it was.

The problem was that implementation couldn’t actually be separated from context and judgment as cleanly as everyone assumed.

AI changes the economics and latency of the implementation resource dramatically, but it doesn’t eliminate that underlying constraint.

Someone still has to know what the software should do.

The power mixer problem, in numbers

Let’s put some numbers on it.

Let’s take the higher number and suppose writing code accounts for 25% of an engineer’s work. That’s 10 coding hours in a 40 hour work week.

Now suppose AI makes that work 50% faster.

That seems like an enormous improvement.

But “50% faster” doesn’t mean the work takes half as long. Work performed 50% faster takes about two-thirds as much time. So our hypothetical 10 hours of coding becomes roughly 6.7 hours.

We’ve saved 3.4 hours out of a 40-hour body of work.

That’s an 8.5% improvement in total engineering time from a tool that made coding 50% faster.

And that’s the best case.

It assumes none of the gain gets consumed by additional prompting, reviewing, correcting, testing, or understanding generated code.

If half of the gain gets reabsorbed elsewhere in the engineering process, our spectacular “50% faster” AI coding improvement has produced something closer to a 4% improvement in total engineering throughput.

There’s nothing wrong with 4%.

Four percent at scale can be worth a fortune.

But it’s very different from the productivity story being told on TikTok… and a far cry from what companies think they’re going to get when they.

The mixer got dramatically faster.

The bread didn’t.

Your bottleneck determines your ROI

This is the important part.

Imagine two companies using exactly the same AI coding tool.

Company A, has large engineering team, with dedicated roles and mature staffing. Engineers receive clear requirements. Interfaces are stable. Tests are comprehensive. Product management handles scope. Infrastructure is mature. Observability is excellent. Security expertise is readily available. Customer support triages problems and only a fraction of them reach engineering.

In this company, Engineers really do spend a significant portion of their time writing code, and implementation really is a major constraint.

Make implementation dramatically more efficient and the organization gets a substantial benefit.

Now consider Company B. A more typical company where the engineers are meeting with stakeholders, performing support functions, managing operations, etc.

The engineer receives a support ticket saying:

“Payroll is wrong for some customers when they run it after changing an employee’s deduction, but only sometimes.”

Before writing anything, the engineer has to reproduce the problem, understand the payroll rules, trace the existing implementation, determine which customers are affected, talk to support, figure out what “correct” means, assess whether changing it will break historical behavior, design the fix, and decide how to safely deploy it.

Then they write 30 lines of code.

An AI that writes those 30 lines five times faster isn’t going to transform Company B.

Not because the AI is bad.

Because writing those 30 lines was never the bottleneck.

That’s why the same tool can produce dramatically different ROI at different companies.

AI coding productivity isn’t primarily a property of the model.

It’s a property of the system into which you put the model.

Why smaller companies are particularly exposed

Large engineering organizations can spend headcount and infrastructure to remove work surrounding implementation.

They can specialize roles. They can build extensive internal tooling. They can maintain dedicated security, operations, support, product, platform, and reliability functions.

That isn’t to say that engineers at large companies “just code.”

But it does mean the organization has more opportunities to move non-implementation work away from individual engineers and amortize it across many people.

Small and mid-sized companies usually can’t.

The engineer is often the person who triages the support ticket, talks to the stakeholder, designs the feature, understands the old system, watches production, investigates the outage, and writes the code.

Frequently in the same afternoon.

If coding was already only a minority of that person’s job, dramatically accelerating coding only compresses a minority of the job.

The rest of the work doesn’t disappear.

The dough still has to rise.

Measure the bakery, not the mixer

None of this means AI coding tools aren’t useful.

They are.

They’re excellent for scaffolding, boilerplate, exploration, first drafts, unfamiliar APIs, writing documentation, writing tests, repetitive transformations, and well-bounded implementation work.

The mistake is taking a productivity measurement from one stage of software production and applying it to the entire engineering organization.

If you’re an engineer, don’t expect a tool that makes one coding task 50% faster to make your week 50% easier. Use it where it removes real work, and remember that generated code becomes code you and your team own.

If you’re leadership, don’t staff against a productivity number measured on a job your engineers don’t actually perform.

Measure what happens in your organization.

Did lead time fall?

Did features reach customers faster?

Did defects fall?

Did engineers close customer problems faster?

Did operational load decrease?

Did the same team produce more working software that mattered?

Not more lines of code.

Not more pull requests.

Not more tasks marked complete.

Those are measures of activity inside the system.

The business cares about the output of the system.

AI coding tools can make implementation astonishingly cheap.

But making one stage of a process faster only improves the whole system when that stage was constraining the system in the first place.

If mixing wasn’t what determined how many loaves your bakery could produce, buying a mixer that’s twice as fast won’t double your output.

And if writing the code wasn’t what determined how quickly your company could deliver working software, making coding twice as fast won’t double that either.

That’s why you’re not seeing the AI coding ROI you were promised.

And unless you fix the actual bottleneck, you’re probably not going to.

If you would like help optimizing your development processes, Ionzero can help.

References:

5. Forrester Developer Time & Predictions

2026 Software Development Predictions: Predictions 2026: Software Development Goes From Jamming To A Full Orchestra (Forrester Research Blog)
Enterprise Predictions Hub: 2026 Predictions Guide For Technology & Security (Forrester Research)

6. Sonar Developer Workday Survey

Official Website & Research Hub: Sonar Developer Research & Reports