The real impact of AI in software
- 22 min read
There used to be a time when showing off your many lines of code as a sign of engineering prowess would make you a laughing stock. The trend was quite the opposite - achieving the same result with fewer lines of code was considered peak performance. Both ideas were stupid. It was common knowledge that counting lines of code, for any purpose, was not only useless, it was harmful. And here we are, in 2026, using precisely this metric as an indication of productivity. Is this lead in the water? Did we finally succumb to brain rot?
I’m of course talking about how we choose to measure the impact of AI in software. It’s really rather frustrating. The story that emerges in the media, especially the social media, just doesn’t align with my lived reality, and it’s not even close. Am I really that weird? Are all the people I work with really that weird? Am I the one with brain damage? Luckily we still live in the era of science, so I’ve decided to do some spelunking and get to the bottom of what’s really happening.
The existing research
There is seemingly a lot of research into impacts of AI on software engineering work. Finding research that’s trustworthy is another matter. It’s as if whoever is sponsoring or reporting on it puts their own spin on the results, so that the same data leads to diverging conclusions. That is, if they even used valid data to begin with.
Let’s have a look at a few reports, each coming from a different source with a different agenda, with their own data collection process and methodology. I’ve picked those that seem to have somewhat reasonable sample size and lack an obvious, unmitigated selection bias.
First, the DORA report - from last year because this year’s report didn’t come out yet. They’ve been doing this every year, collecting self-reported data from a variety of companies. If they really used their methodology as described, this data should be solid. And here is the first problem. The report is very narrative-heavy. It’s like they don’t want the data to speak for itself. That’s because the data is confusing.
On one hand, people report an increase in individual productivity (more than 80% of all respondents) but the team performance is only mildly affected. There is an increase in throughput (DORA measures it by lead time for changes, deployment frequency, and failed deployment recovery time). At the same time, the instability also increased (measured by change fail rate and rework rate). We can’t really see the raw data that forms both conclusions in the report, instead we’re presented with a wall of text and a lot of bar charts for various adjacent topics. I assume they just took the previous year’s data and only present us with the aggregate comparison. Fine, I’ll take it.
So we have an increase in throughput, mostly on the individual level, simultaneous increase in instability, and a minor increase in team-level performance. So is the AI usage net positive? That’s an open question. DORA authors call it “stubborn results” which immediately puts me on alert, because it suggests they are trying to fit the results into a pre-existing story. Which is probably fair considering one of the authors also co-wrote Vibe Coding and is promoting it. While the narrative is questionable, I trust the general findings because the sample selection is solid and the underlying metrics reasonable. It’s all self-reported though, so this is more a reflection of our industry’s collective state of mind.
Then there is CircleCI’s report from this year. They’ve analyzed the data from their users and seem to have used decent methodology. This makes the data trustworthy.
Data shows a 59% increase in throughput - and by throughput they mean the number of daily workflows run. The workflows also run faster - median duration is faster by 18% compared to last year. But the time to recovery is also up by 13% (or even 25% on feature branches), continuing the trend since 2022. Then there’s the success rate which is down compared to last year - the attempts at merging changes into production code bases now fail 30% of the time.
Ooof. So we have more builds, more merges, which implies more code, possibly more automation because it all runs faster, but the average quality is worse. I see this and I immediately think - churn. This is code churn. But that’s not what CircleCI calls it. They call it productivity and hail it as great results. What’s more, they claim the gains are concentrated at the top 25% of teams and call it a performance gap. This is an excellent example of how to spin a narrative that contradicts the data. I actually like it because it shows you can’t ever trust the narrative without seeing what went into constructing it. This report appears to drive FOMO among those teams who aren’t “using AI to its full potential” while selling a solution to the very obvious problem of low quality. The results are presented as great news, while to me, they’re anything but. It’s especially funny that they say it themselves:
“For a team pushing 5 changes to the main branch per day, going from a 90% success rate to 70% is the difference between one showstopping breakage every two days to 1.5 every single day (a 3x increase). At just 60 minutes recovery time per failure, you're looking at an additional 250 hours in debugging and locked deployments every year. And that's at a relatively modest scale. Teams pushing 500 changes per day would lose the equivalent of 12 full-time engineers.”
So we see more code generated, and more failures overall. Notice that this echoes the data from the DORA report. We still can’t say whether AI usage in software development is net positive based on this data.
Then there is a report from Faros, a company selling AI telemetry tools. I expect this to be heavily biased on multiple dimensions. First, selection bias - much like CircleCI, this report is based on data from this product’s users. These are only companies that are heavily investing in AI tools (otherwise they wouldn’t pay for extras like telemetry), so the results favor responses that should be positive towards AI outcomes. On the other hand, it’s in Faros’ interest that there is a problem, because they are selling a solution. I expect the narrative to be skewed towards highlighting the issues. This tension between selection bias (positive towards AI) and the company's agenda (highlighting problems with AI) is what makes this report worth looking at.
This report explicitly calls out code churn: “the ratio of lines deleted to lines added in a given quarter, has increased 861% with high AI adoption”. They list high throughput - measured as developer task completion (+33.7%), epics completion (+66.2%), PRs merged per developer (+16.2%). At the same time, they observe 11% fewer deployments per week. The percentages represent a “change in metric values between the two quarters of lowest AI adoption and the two quarters of highest AI adoption within the observation period”. This makes the data valuable, we’re comparing apples to apples.
As expected based on Faros’ agenda, the report paints a rather bleak picture of AI adoption - code churn, quality decrease, and interestingly, workflow slow down (which is counter to what CircleCI reports, but I have to assume they both mean different things by “workflow”). They don’t mention many positives, which once again makes it hard to determine the full picture just from this report.
Ok, so where does this leave us? All three sources agree on one thing - there’s more code generated, but the quality is worse. There is another conclusion, either exposed through data or implied through the narrative - the overall benefit is not visible on a systemic level even if we’re trying very hard to find it.
There are more reports of this kind out there, but they all expose the same pattern. There is something that the authors tend to view as a contradiction - more work is getting done, but we fail to see the positive outcomes on the holistic level. The explanations given for it differ depending on the authors’ agenda. Most common explanation is that humans aren’t able to keep up, often with an implication that those pesky humans slow everything down and if it wasn’t for them, all would be perfect. I don’t like it. I don’t care if the data shows a contradiction - that only makes me curious. Presenting a single narrative that explains the contradiction without doing more research and without calling that narrative a hypothesis is a shitty move. So I trust the data but don’t trust the narrative in any of those cases.
Explaining the contradictions
The data shows there’s more code generated, more work done (by some definition of “work”) but that doesn’t translate to high quality, stable code delivering value for the business. I have two hypotheses immediately forming here. One - the AI in the form of LLMs simply isn’t the right technology for making good software. Two - AI is encouraging hyper-local optimizations and most people don’t realize this is bad.
Sadly, I don’t have a team of researchers to devise proper experiments to validate these hypotheses.
Instead, I’m going to do a very unscientific thing and use anecdotal evidence. Or rather, do a very scientific thing and perform observations in situ. See how easy it is to frame the same situation in a very different light, depending on what story you want to tell?
I talk to a lot of people. I talk to our customers and prospective customers. I speak internationally at tech conferences and talk to attendees. I have friends who work in tech. Seriously, that’s a lot of people. A wide variety of companies are represented in this set as well. There are pragmatic small businesses, taking only calculated risks. There are hope-fuelled startups, using every opportunity to get ahead. There are ancient enterprises that discovered Agile just a few years ago. There are hyperscalers, each a unique and very special snowflake. Then there are all sorts of medium and large businesses at varying stages of prosperity. All that, from different countries. I think that’s a pretty great set to do some qualitative analysis.
So what do people say?
From what I’ve gathered, the vast majority of developers interact with LLMs over a chat interface to get suggestions on their code. This is very similar to how they used to use Stack Overflow way back when - ask a question, copy/paste the answer. There is very little usage of agentic coding at work, which is consistent with the DORA report. On the other hand, both my data and DORA report are several months to a year old, which could be argued is ancient history when it comes to AI. Still, people experiment with agents. And when I say experiment, I mean it - they tinker, do pilot projects, apply it to some half-neglected workflow that no one cared enough about in the past. Other than hyperscalers (who have the resources) and very early stage startups (who don’t have the resources, pardon, engineers), I haven’t encountered a software company that’s using agents at any meaningful scale.
This is important, because when we talk about “using AI” for coding, chances are we mean very different things. Many people say they use it, while all they do is occasionally ask the “chat” (which is whatever provider their company pays for) to write a function or a class, or click tab to autocomplete a line of code. That’s very different from running Claude or Copilot or Codex or C-whatever agentic harness to autonomously make important architecture decisions while writing or re-writing features end to end.
With that out of the way, what is the impact of AI that people talk about? Almost everyone whose role involves hands-on coding says they feel more productive. Feel being the important keyword. In this case, I’m not interested in feelings, I want to know what those people accomplish, and that’s where I usually drive the conversation.
From all the people I talked to, there is a set that consistently has genuine success stories. Not just people feeling productive, not some fluff, but actual cases of projects delivered ahead of schedule or with fewer people, or even enabled in the first place. Those success stories would be amazing if not for the fact that people in this set have one thing in common - they work at one of the hyperscalers. This means access to internal, custom models trained on the massive corpus of internal data and unfettered access to powerful hardware running those models “for free” (yes, I know it’s technically not free, but those teams don’t have to account for their hardware usage in their budget). In one case, the team in question trained their own foundational model not on English language, but on the language of their specific types of logfiles. This is not merely fine-tuning, it’s creating a bespoke, custom-fit model from scratch. None of these luxuries - neither the “free” hardware nor custom models, not even the massive amount of data to train the models on - are available to us plebeians. People working at hyperscalers tend to be oblivious to this gap.
What’s more, their success stories tend to relate to problems that only hyperscalers have - either due to organizational red-tape or due to the scale, or something else that’s only true for that one specific company. This means none of this can be applied at companies that aren’t hyperscalers. Consequently, while those stories are interesting, I’ve learned to completely ignore them.
Then there are all the people claiming they feel way more productive, sometimes estimating 10 times productivity increase or some other huge shift. However, I have yet to encounter a single person who can explain what “productive” really means in this situation, or how they even know they’re more productive. Digging deeper, it always ends up as some form of measuring the output - number of features, tickets, PRs, but not things that are shipped in front of the customers. This is consistent with all the reports I’ve outlined in the beginning - we see more activity, more output, but not necessarily better outcomes. I still think it’s good when people feel more productive, it’s great for morale. What’s even better is when useful outcomes are delivered by those happy engineers. I’m still waiting to hear that story from someone who doesn’t work for a hyperscaler.
Finally there are people who I call tinkerers. They create little utilities for themselves, tailored to those very narrow use cases that would be mildly annoying if unsolved. Small automation here and there, minor convenience apps that speed things up a bit. Nothing fancy or commercially suitable, or even usable at scale, but still making things just a tad bit better for one or few people. Interestingly, those people do this at home a lot, but not nearly as much at work. Exploring the why yields some curious insights.
One reason why people don’t tinker at work as much as they do privately is the lack of opportunities. And by opportunities I mean the existence of small, fairly simple problems that beg to be solved with that one evening of vibecoding. The quick wins simply aren’t always there.
The second reason is more interesting. People happy with the quality of LLM generated code for their personal little projects point to the insufficient quality as the reason for not being able to deliver similar results at work. They go from “LLMs write great code!” to “LLMs generate hot garbage!” in the course of a single conversation. It’s almost as if the question of quality is one that gets answered with “it depends”... That’s why it’s important to always interview the specifics when asking people general questions. What’s good enough or even great for personal use, may be woefully inadequate for enterprise scale. I see this pattern a lot and having worked as a software tester in the past, I’m not surprised.
With all that said, the vast majority of people I talk to about AI aren’t in any of these groups. They don’t have a lot to say on the topic at all. They don’t seem to have strong opinions. Mind you, these are the same people who will argue for hours for their side of monolith vs microservices or serverless vs Kubernetes debacles. But when it comes to AI, they may acknowledge “things change” or repeat some hypey fluff you hear in the media and other than that, they are very quiet.
It’s like they don’t really know what to do with AI, where to apply it or how to plug it into their daily reality. They bounce off the existing tooling, or at least what their company has sanctioned. They don’t play with LLMs outside of work. And whatever they do at work either isn’t significant enough or isn’t clear whether it’s “good” or “bad” to force these usually opinionated people to form an opinion.
There are a few things at play here. One: there is always an adoption curve which lags behind the cutting edge. With new trends and technologies, there is usually a chasm between the adventurous early adopters and the majority of future users. But this is something else. We’ve had LLMs for around 4 years now, that’s ancient history for tech. The chasm would have normally shrunk by now. The chasm theory doesn’t explain what we see. It certainly feels like in the case of AI, the success stories we hear out there are not merely across the chasm, they come from a different planet.
The sense I get when talking to this silent majority of not-quite-users of AI in software is that they hesitate to share what they really think, especially if their experience is very far from what they perceive as the mainstream, but what really is the curated storyline presented in the media and at conferences. It doesn’t help that the prevailing narrative implies that to use AI you need some special black magic skills (prompt engineering, vibe engineering, harness engineering, doesn’t it feel like we’re simply inserting “engineering” after random words to make it sound more serious?) Like that, it’s easy to believe that if it doesn’t work for you, it’s a skill issue and you must be prompting it wrong or aren’t smart enough. And who wants to admit they’re not smart or incompetent?
These are the conversations that are most difficult to get to the actual crux of the issue because of that hesitation. But I’ve done enough user research over the years and plenty of behavioral interviewing, so I know how to get people to talk. Doesn’t always work, but when it does, it’s eye opening. I believe this group, the majority voicing no opinions on AI, is where we find a lot of the answers to explain the contradictions we see from the three reports I’ve covered in the beginning.
By far, the most common story is that people don’t know how to use AI in practice. Dismissing it with a “prompt engineering is a skill, you have to learn how to do it properly, yada yada” may blind you to the fact that these are all smart people. They have access to the same materials you do - articles, tutorials, trainings - and they go to tech conferences each year, so are really open and willing to learn new stuff. Yet, for some reason, the existing interfaces and tooling surrounding the AI just don't work for them. I’d love to do some proper in-depth research on those at some point, but meanwhile I see themes of: confusion over the current state of work, quality of the output, concerns over security and privacy, and poor developer experience.
The other common story is that LLMs don’t seem to be helping with activities that take the most time, and so their impact is not significant. Code reviews, testing, clarifying requirements, waiting for feedback from the stakeholders, waiting for that other team’s part, discussing the design options, designing the system, fleshing out the architecture… you know, like the 90% of all the things that make up software development, other than coding itself.
Notice that those two cases create a self-reinforcing feedback loop. If you try using LLMs and the impact is small, you’re not motivated to spend more time trying to improve your LLM skills, which in turn may block you from finding out ways to apply those tools for higher impact activities. This feedback loop appears to exist even at companies that “strongly encourage” (read: force) people to use LLMs as much as possible. It’s almost as if the results of knowledge work depend on intrinsic motivation, who would have thought?
To recap all this, we have two fairly small groups of people I ignore when evaluating impact of AI on software making: engineers at hyperscalers and those who point to outputs not the outcomes as success. Then there is a small group that’s able to achieve meaningful outcomes for their side-projects but can’t replicate them consistently on an enterprise scale. Finally, there is the silent majority with no opinions on AI who don’t use it much, and whatever they do use it for isn’t worth talking about.
One could argue it’s still early, the technology is changing rapidly and LLMs are getting better. But it’s been 4 years, as I’ve said before, that’s a very very long time in tech. I strongly suspect what we’re seeing is not what we’re calling it. It’s not a revolution. It’s a slow, maybe fundamental, but still a slow change. AI is a normal technology.
Let’s get back to my two hypotheses: AI in its current form is not the right tool for making software, or AI creates local optimizations that are bad or neutral for the larger system. We can clearly see evidence supporting the latter. I don’t think we can say much about proving or disproving the former hypothesis though, it seems situational. But talking to the people with no strong opinions on AI, I see something else that appears to be the crux of the issue.
Raising the floor, not the ceiling
I wrote about this before. The LLMs are great at getting us to a mediocre, or slightly below average result really fast. If we start from zero, this is great. AI speeds us up tremendously. If all we want is something rather average, we hit the jackpot. But as soon as we’re looking for something of greater quality, or even something unique, LLMs not only stop being helpful, they quickly turn into hindrance. This is the whole reason for the contradiction in various pieces of research on the matter - yes, LLMs make us more productive and yes, LLMs don’t produce significantly better outcomes. Those two statements apply to different parts of the work though.
The biggest issue I see is that while the AI in the form of LLMs is great at small wins and getting us to the average fast (raising the floor of what we can achieve), it is sold to us as an ultimate shortcut for everything, as if it can do the work of an expert. And it’s so easy to get fooled.
If you’re not an expert in something but merely a layperson, the output of an LLM, being average, will seem like expert’s work to you. That’s because if you’re not the expert, you’re oblivious to the nuances and intricacies of a given field. What’s more, LLMs output their text in a manner radiating confidence, they mimic the language of experts. Us being humans tend to pick up on that and use it as we would with other humans - as heuristics to evaluate the trustworthiness and competence of the person speaking. Except in case of LLMs, these heuristics are just empty words, they don’t map to actual competence or trustworthiness. Our naturally evolved instincts fail us here.
Meanwhile, experts tend to see right through it, because, well, they’re the experts. They can recognize little inconsistencies, mistakes, or outright fabrications. They don’t rely on the language as heuristic, they understand the subject matter. That’s why most professionals will agree that LLMs don’t quite work for this thing that is their utmost, deepest expertise, but LLMs are wonderful for all those other things.
So we have this general purpose tool that can only output the mediocre - what do you think happens when you try to use this tool believing it can do the work of an expert? This is what we see in the software industry at the moment and it’s not pretty.
Sometimes it feels as if our whole industry has permanently succumbed to FOMO doused in unicorn juice, opting for emotional narratives and cherry-picking data to fake their legitimacy. It feels like we’re no longer interested in science or even logic. But then I remember it’s only what gets shouted in the media, traditional and social ones, where the most shocking statement gets the attention. Reasonable industry leaders are everywhere, but even if they talk, they don’t get amplified much.