In the previous piece in this series, on engineering judgment, I described an engineer who spent months training a neural network to decide whether a network address fell inside a given block. It is a question a few integer operations answer exactly and instantly, every time. He reached for a heavyweight statistical tool to solve a problem with a known deterministic answer, and the result was slower, more expensive, and less reliable than a few lines of code. That was one person’s misjudgment, the kind another person in the room can catch and correct.
What I want to examine here is the same shape produced not by a person but by the tool itself. When you ask a model to choose an approach to a hard problem, it reaches for the heavyweight, well-published, consensus method the way that engineer did, except it does so systematically. A person’s mistake can be corrected once. A model’s bias has to be corrected every time.
A language model does not return the best answer to your problem. It returns the most probable continuation given everything it was trained on, which is a popularity-weighted average of how problems like yours have been answered before. When your problem sits near the center of that distribution, the average is excellent, and the model is a genuinely useful collaborator. When your problem sits in the tail, the average is confidently wrong, and it is wrong in the most plausible-sounding way available.
Underneath everything that follows is one fact about these systems. A model is built to agree with the recorded past. Its objective, at bottom, is to return the continuation the existing body of human writing would most likely produce, so its answer is only ever as good as the fit between your problem and that record. When the two agree, the model is superb. When they diverge, the divergence takes one of two forms. Either the record is generic, written about everyone else’s problem rather than yours, or it is stale, settled before the field moved. Those two failures look identical coming out of the model, and they call for different defenses. Telling them apart is the rest of this piece.
Why the Model Pulls Toward the Center
This is not a flaw in any particular model. It is a property of how these systems are built, and it has two compounding sources. The first is the pretraining objective itself. A model is trained to predict the next token, which rewards what is common over what is merely correct. The second is preference tuning. The reinforcement-learning stage that makes a model helpful and safe also narrows its output toward what evaluators expect to see, an effect documented as mode or preference collapse, in which less common responses are driven out even when they are valid.
The two effects point the same direction. Pretraining centers the model on the frequency of the training data. Preference tuning centers it again on the expectations of human raters. By the time you are talking to it, the model has been compressed toward the consensus twice, and compressed hardest away from exactly the rare, specific, or contrarian answers that mark out expert judgment.
Consensus Is Generic
The first failure is that the consensus is generic. It is an average taken over everyone else’s problems, aimed at the median case, and your system is not the median case. A model can be perfectly current and still be wrong for you in this way, because the issue is not that its information is out of date. The issue is that your specific constraints invert the tradeoff the consensus assumes.
I ran into the hard edge of this on a project of my own: a spacecraft physics simulation engine I have been building for more than twenty years, now owned by Arbor. I developed the upper layers AI-first, deliberately, to see how far the tooling could carry real work. On the simulation and integration code it carried far. On the guidance problem it failed in a way that was not random but structural.
The task was low-thrust trajectory guidance: how a solar-sail or ion-thruster spacecraft should steer to move between orbits. Every model I put the problem to reached for the same answer. Train a reinforcement-learning agent, reward it for reaching the target orbit, let it discover a policy. I asked for a different approach, repeatedly and explicitly, and each time the model acknowledged the instruction and drifted back, sometimes inside the same response. Reinforcement learning dominates the recent literature on autonomous spacecraft guidance, and that dominance was a center of mass the models could not move off of.
The approach that worked was older than the literature the models were drawing from. I built the guidance around Q-law, a Lyapunov-based feedback control law whose minimum-time transfers approach a result Theodore Edelbaum published in 1961. It is deterministic. It is cheap to evaluate. It carries no training loop and no learned policy that has to be trusted on inputs it never saw. It outperformed every reinforcement-learning policy I trained, by margins that were not close.
No model proposed it. Not because the mathematics is obscure, since it sits in the open literature, but because it is not where the mass of the recent literature sits. A control law from the 1960s is in the tail of the distribution. Reinforcement learning is the peak. The model goes to the peak every time, and here the peak was the wrong place to be.
Notice the difference between this and the engineer who trained a network to compute what plain arithmetic computes exactly. He made a mistake. The models did not make a mistake. They produced the field’s consensus exactly as designed, which is the harder problem, because a mistake is corrected once and a bias reasserts itself on every prompt.
Consensus Is Stale
The second failure is different in kind. The consensus the model carries is frozen at the moment its training data ends. It is a snapshot, and it is handed to you without a date on it.
I hit the small version of this constantly, and it is more annoyance than hazard. I will ask a model to set up a test matrix for a Python project, and it reaches for 3.12 and 3.13 and stops, leaving out 3.14, because 3.14 did not exist when its training data ended. It does not suggest checking whether a newer release has landed. It hands me the matrix that was current at its cutoff, as though the cutoff were the present. On a version list this is trivial to catch, because I know 3.14 shipped, and that is exactly what makes it a useful illustration. You can see the staleness here only because you happen to hold the missing fact. In the domains where you do not hold it, the same omission is invisible, and it arrives with the same confidence.
Fields move. A technique that was speculative becomes standard. A default that was best practice becomes an anti-pattern once its failure modes are understood. A library’s idiomatic usage changes across a major version. A security assumption everyone relied on is shown to be weaker than believed. When that movement happens after the training cutoff, the model cannot see it. It will answer a question the field has already reconsidered by giving you the view from before the reconsideration, with exactly the fluency and confidence it would bring to a current one.
This is more dangerous than plain ignorance, because the answer does not look wrong. It is not gibberish and it is not visibly out of date. It is yesterday’s correct answer, wearing today’s confidence. A model does not merely fail to see that the ground has moved. It erases the evidence that it moved at all, answering in the same even voice it uses for everything else. Nothing in the reply tells you that history has shifted underneath it. To the model, a consensus that still holds and one that has since been overturned are the same high-probability continuation.
The faster a field is moving, the wider this gap opens. In areas where practice is consolidating month over month, the security posture around a new class of system, the right way to operate a framework that is six months old, the current estimate of a threat that looked distant a year ago, a model’s confident answer can sit a full generation of practice behind what a current practitioner would say.
Consensus Is Usually Right, Which Is the Trap
Neither of these is an argument against using models, and neither is a claim that the consensus is usually wrong. Most of the time the consensus is right. That is what makes it the consensus, and it is why a model that reaches for it is a genuinely useful collaborator on the large majority of problems that sit near the center and change slowly. I use models this way every day. The convergence I am describing is a feature far more often than it is a failure.
The trap is narrower and sharper than a general warning not to trust the model. It is that the model gives you no signal about which case you are in. It hands you the consensus answer for a problem at the center and a problem in the tail with identical confidence, and for a field that has moved and one that has not with identical confidence. The fluency is the same. The certainty is the same. The only thing that differs is whether the answer is right, and that is exactly the thing the model cannot tell you, because telling you would require the contextual judgment it does not have.
There is a way to get the tail answer out of a model, and it is worth being honest about it. You can steer the model there, as I eventually did, by naming the approach and forcing it to engage. But that works only if you already know the approach exists. The model did not supply the judgment that Q-law was the answer. I supplied it, and used the model to help implement the decision once it was already made. The model amplified a judgment I brought to it. It did not originate one.
What This Means for Engineering Leadership
The leadership implication follows directly. Treat a model’s recommendation as a compressed map of the field’s consensus as it stood at the training cutoff, which is what it is, and never as a judgment about your specific problem today, which it cannot be. Those are different artifacts, and confusing them is the failure mode.
A practical test: if the model’s recommendation is the one a strong recent graduate would give after reading the ten most-cited papers on the topic, you have retrieved the consensus. That is valuable, and it is also not yet an answer. The answer requires someone who knows where your problem departs from the median case, and whether the field has moved since the model last looked.
That points at two kinds of engineer a model cannot replace, and both sit outside the center it occupies. The first carries unfashionable, domain-specific, or historical knowledge, the old technique or the adjacent field or the constraint everyone else forgot, and catches the model when the consensus is generic. The second is current with where practice has actually moved, and catches the model when the consensus is stale. As models absorb more of the routine center, these people do not become less valuable. They become the point.
This is the other half of the argument the previous piece made. When AI compresses the cost of building almost anything, the scarce and decisive act is no longer implementation. It is choosing the right approach, and the right approach is precisely the one the model is least equipped to propose. Cheap execution does not diminish judgment. It concentrates the remaining value in it.
A model is extraordinarily good at telling you what the field believes. It cannot tell you whether the field is wrong for your problem or simply behind it, because in its voice the mistaken answer and the outdated answer sound exactly like the right one. That distinction is what engineering judgment is for, and it is the one thing no amount of fluent, confident output will decide for you.
