Not long ago I was advising a CTO whose team had shipped a product under competitive pressure. They had used AI tooling heavily throughout, and the result looked credible: a working system, a live demo, a technical story that held up in a pitch room. Then I asked what happened to data integrity if a dependency went away mid-write. The conversation stalled. Nobody in the room could answer, and nobody had asked before. No one had done anything wrong in the conventional sense; they had used tools that really did reduce execution friction. But nothing about the system they ended up with could be examined, because nothing about it had been reasoned through. A system nobody can explain is not ready to be operated.
None of this is new, and none of it is specific to AI. Cloud let teams provision faster than they could keep track of what they were running. Microservices let teams decompose and redeploy faster than anyone could hold the resulting dependency graph in their head. Every platform transition I have worked through has had the same shape: it lowers the cost of producing a working system and raises the cost of understanding one. AI is the current iteration, and it is moving faster than the ones before it.
Code Production Was Never the Hardest Part
For most mature engineering organizations, typing code stopped being the bottleneck a long time ago. The constraints live elsewhere: designing systems that hold together, integrating change without breaking what already works, reviewing deeply enough to catch what matters, and getting to production before the requirements move. AI has increased the rate at which change enters that pipeline without widening any part of it. Teams experience the first half as productivity. The second half arrives later.
Easier code generation does not make an organization better at absorbing change. Review queues lengthen and test infrastructure saturates, and the fact that AI generates tests too does not help much, because writing tests was never the hard part. Reasoning about what they actually cover is. Senior engineers end up spending their time validating work that looks clean on the surface and whose behavior under load nobody has thought about. Output goes up. So does the cost of operating what that output produced, though that cost arrives on a delay and lands somewhere else in the budget.
The Illusion of Architectural Competence
The opening story is a calibration failure. When execution friction drops, the signals leaders use to judge their own technical maturity stop being reliable. Building something that looks production-ready no longer says much about whether you can operate it.
A different team I worked with built a custom decryption mechanism for their API gateway using Envoy proxy. It required a new underlying stack to hit the performance target. Everything tested correctly and the benchmark numbers were good. In production, under real traffic, the deployment ran out of file descriptors, first taking down Envoy, then cascading to the underlying stack and the DPDK distributor, and resulting in a full service denial. The load, stress, and shadow testing had all been done, and done carefully. None of it had been run against the connection patterns real users actually produced. What failed was not the test coverage. It was the team's model of how the thing would behave once it left the lab. The system did exactly what they built. Nobody had understood what they built.
Research on AI in high-skill knowledge work calls this a jagged technological frontier: the boundary inside which AI reliably improves performance, and outside which it degrades performance without announcing that it has. A system that looks credible tells you nothing about which side of that line you are on. That is the part that is new. Broken systems used to look broken.
Change Throughput vs. Assimilation Capacity
The frame I keep coming back to is assimilation capacity. Every organization has a finite ability to understand new system behavior, spread that understanding across teams, and bring less experienced engineers along while doing it. Change throughput used to be constrained enough that those processes kept up. AI-assisted development is pulling the two rates apart.
When change outruns the capacity to internalize its consequences, what accumulates is not defects but fragility. Ownership boundaries blur, documentation drifts from reality, and incidents take longer to resolve because the number of people who know how the system behaves under stress keeps shrinking.
The bill for this arrives as capital allocation, not as a technical problem. Organizations already pay it: longer incident recovery, duplicated systems, infrastructure provisioned for headroom nobody modeled, and senior engineering hours spent validating work that should have been understood before it was written. That money buys the operation of complexity introduced faster than anyone could absorb it, and none of it shows up as a line item with that name on it.
Pressure Across Three Planes
None of this stays inside team process. It shows up in infrastructure behavior across three planes. On the control plane, deployment velocity turns into coordination instability: retries amplify, and failure domains stop being something anyone can reason about ahead of time. On the data plane, encryption overhead and placement decisions interact at scale in ways a small test environment will never show you, and the cost curve bends. The artifact plane is the one that persists. Faster experimentation means more copies of long-lived strategic assets, and every copy is something that has to be secured, governed, and eventually retired by someone who may not know it exists.
The most effective intervention I have found is simple: require that any pull request containing AI-assisted code include a human-written explanation of what the change does, what its failure modes are, and why this approach was chosen over alternatives. Reviewers are then expected to assess whether that explanation reflects understanding or surface familiarity. It does not slow development much, but it moves where the thinking happens. The engineer who submitted the code has had to reason about it, and the reviewer is checking for comprehension, not just correctness. That distinction matters when the thing being reviewed is a system that will need to be operated, debugged, and evolved by someone other than the person who generated it.
Organizations do not need to slow down what engineers produce. They need to control how that output crosses team boundaries and lands in the rest of the system. That much is an internal problem, and solvable internally. The exposure collecting on those same boundaries is not. It is also being shaped by an adversary base that keeps getting more capable, by regulators with less patience than they had five years ago, and by cryptographic assumptions carrying a known expiration date.
The Emerging Constraint Surface
AI changed the economics of building software. It did not change the consequences of operating it. What bounds an organization now is not model capability or compute budget but how fast it can turn accelerated experimentation into understanding that survives the person who did the experimenting.
While that gap is open, exposure collects in the places that are hardest to look at: the trust assumptions sitting inside system boundaries, the cryptographic posture around assets that will outlive it, and the knowledge of how the whole thing behaves under stress that currently lives in three people's heads. Every platform transition I have worked through has done some version of this, from the server room to the cloud, from the monolith to the service mesh, from chroot and cgroups to Kubernetes. This one is faster, and the assets on the table are worth more.
How that exposure compounds across control, data, and artifact planes is the subject of the next piece in this series.
