Considered Intelligence

How AI-coded systems turn fixes into fossils

Michael Witbrock and Codex (GPT-5.6 Sol Ultra) written in the context of a frustrating programming session.

The most dangerous code written by AI is not obviously bad code.

Bad code is visible. It fails tests, throws exceptions, performs poorly or can’t be maintained. The more dangerous output is a clean, documented, reviewed and thoroughly tested patch that prevents one observed failure by quietly eliminating valid behaviours, alternative implementations and future routes to improvement.

A failure is reported. A coding agent analyses it, constructs a plausible causal story and repairs the nearest relevant code. It adds a validator, contract, state, retry, adapter or fallback. It writes tests showing that the failure is gone and updates the documentation to explain the new mechanism.

The ticket closes.

Has the system improved? Or has one uncertain diagnosis just become permanent architecture?

We call this the reliability ratchet: a process in which every visible failure adds machinery, while almost nothing creates comparable pressure to remove it. The system becomes more locally defensible and more globally constrained.

That is non-progress—not inactivity, but energetic work that makes future work harder and may remove all reasonable paths to achieving the intended system..

An old pathology at a new speed

Human software teams have always accumulated workarounds, stale abstractions and tests that preserve yesterday’s implementation. Coding agents did not invent technical debt. They change its economics.

A human team might take days to turn an incident into a diagnosis, an implementation, a test suite and a design explanation. A coding agent can produce the entire package in minutes. Because all the pieces are coherent, the result looks unusually rigorous.

But the test is often not independent evidence for the diagnosis. It is the diagnosis repeated in executable form. The documentation repeats it in prose. The contract embeds it in an interface. The implementation makes it operational.

Four artefacts agree because one conjecture generated all four.

This creates a distinctive form of epistemic debt. The code may be locally excellent, the tests comprehensive and the documentation clear. What has not been established is that the explanation of the failure was true—or that the repair preserves the wider range of behaviour the system ought to support.

AI coding lets a weak causal theory acquire institutional authority with unprecedented speed.

How the ratchet works

The cycle is usually:

Observation → conjecture → mechanism → regression test → inherited law.

A coding agent receives a bounded task: make this failure stop happening. It inherits the existing architecture, tests and repository instructions and thinks they are largely intentional – that they have resulted from a coherent plan. Questioning that architecture appears out of scope; modifying the nearest code appears responsible.

So one race condition becomes a mandatory global ordering. One timeout produces a universal retry layer. One malformed payload makes an optional field compulsory. One integration failure creates a permanent fallback. One ambiguous request becomes an intent classifier.

Any of those changes might be right; the mistake is moving from a single observation to a deterministic structure without either first establishing the failure class or weighing the effects against the overall purpose of the software..

The next coding agent encounters the new mechanism and its tests as authority. Removing them looks dangerous. Working around them looks safe. When a neighbouring case fails, another mechanism is added outside the first.

Nothing in this process is locally irrational. That is precisely the problem.

A guardrail that recorded its own defeat

This ratchet is exceptionally difficult to avoid. In the large system maintained substantially with coding agents (including both of the authors) that provided the initial context for this analysis, a test was introduced to restrain the growth of a central coordination module. Its accepted baseline was 46,106 lines. A genuine deletion brought the module down to 45,355.

Then development resumed. As new features and repairs enlarged the file, the baseline was repeatedly refreshed until 49,717 lines counted as compliant. The test even printed the command required to approve the larger number.

Line count itself was not the diagnosis; large files can sometimes be justified. The revealing fact was that a supposed anti-growth mechanism repeatedly certified growth. The guardrail had not changed the development pressure. It had converted accumulation into a notarised exception.

The repository possessed the right principle and an automated check. The executable environment still rewarded closing the current task.

A conjecture is not an invariant

A trace rarely determines its own explanation. The observed failure may have been caused by the nearest branch, or by missing context, a false abstraction, an obsolete stage, an upstream race, an incorrect requirement—or the fact that the entire mechanism no longer earns its place.

The coding agent’s diagnosis is a hypothesis. It should remain revisable until neighbouring evidence supports it.

Software is uncomfortable with hypotheses. It prefers types, states and assertions. That preference is valuable when the underlying property is genuinely invariant:

• One user must not access another user’s private data.
• A payment must not be charged twice.
• A destructive action requires proper authority.
• A committed record must remain internally consistent.

Those requirements can be stated independently of the incident that revealed them. Other decisions are contingent: which component should handle a request, which recovery should be tried first, whether one intermediate state must exist, and whether today’s workaround will remain useful after the surrounding system changes. And perhaps most importantly, whether today’s workaround will interfere with a current or future vital system function that just happened not to have been exercised during the test that drove the fix.

Hardening a contingent diagnosis into an invariant does not remove uncertainty. It launders uncertainty into structure.

In AI systems, user intention makes this especially obvious. Intention is normally a working hypothesis assembled from language, context, available actions and the consequences of being wrong. It should be revised as evidence arrives. Turning it immediately into an immutable classification or expected-outcome contract may make the system more explicit while making it less intelligent.

But the same mistake occurs throughout ordinary software. A causal theory is not made true by giving it a schema.

Tests as an accidental constitution

Tests are indispensable. But tests do not acquire authority merely by existing.

A test may protect an externally meaningful requirement: data integrity, authorisation, idempotency, API compatibility or an observable user outcome. Or it may instead protect an accident: the exact helper called, stage traversed, internal label assigned, cache consulted, message emitted or sequence followed by yesterday’s successful repair.

Those are not equivalent.

When implementation-specific tests are treated as enduring requirements, the test suite becomes an accidental constitution. Future agents are told, in executable form, that correctness means reproducing an approved internal history, not satisfying the system’s purpose.

This is particularly powerful for coding agents. Tests are concrete, local and machine-verifiable. Lost opportunities are none of those things. Persistent instructions to coding agents are not successful in causing them to view tests as revisable, or deletable,

The valid input wrongly rejected by a test leaves no stack trace. The simpler architecture never attempted produces no failing test. The recovery strategy eliminated by a validator generates no incident. The cost of making future changes harder appears days or, in projects slowed by the capability ratchet, months later, distributed across unrelated tickets.

Every visible failure can produce another permanent artefact. Missing alternatives leave almost no evidence that they ever existed. The ratchet is asymmetric.

Why capable agents can conceal a bad architecture

A sufficiently capable coding agent can navigate an astonishingly complicated codebase. It can discover obscure contracts, satisfy brittle tests, thread another field through twelve layers and produce a plausible explanation for all of it.

That capability can conceal architectural decline.

The fact that an agent can successfully modify the maze does not show that the maze is justified. It may show only that the agent is good enough to compensate for it.

As coding models improve, this problem may worsen before it improves. Stronger agents can keep increasingly elaborate systems operational, delaying the moment when humans are forced to confront the structure itself. Meanwhile, the product may become slower, less adaptable and more expensive even as the rate of closed tickets rises.

As we’ll explore in a future post, these abilities also have recently been enabling capable agents to find composite cyber-vulnerabilities in complex systems; this does not mean that they will be as capable at planning and implementing function-preserving mitigations.

Why more rules will not save us

Most repositories and software engineering guides already contain excellent principles: prefer simple designs, avoid special cases, preserve compatibility, do not overfit, add abstractions only when justified, and remove obsolete machinery.

Then the development environment rewards the opposite.

Issue trackers fragment systemic failures into local units of closure. Continuous integration rewards preserving the visible suite. Review templates reward explaining additions more readily than demonstrating that an existing mechanism should disappear. Repository instructions accumulate the lessons of incidents until agents spend increasing amounts of context interpreting accumulated fear.

When prose and executable authority conflict, executable authority wins.

Adding a mandatory fourteen-field “repair theory contract” would merely reproduce the pathology at the process level. The answer is not more paperwork describing simplicity. The selection pressure must change.

Changing what counts as progress

A better development environment would make several distinctions operational.

Separate observation, inference and invariant

Record what happened separately from the proposed explanation. Ask what neighbouring observations would contradict that explanation. Do not let the first plausible diagnosis arrive pre-packaged as the permanent contract.

Diagnose a failure class before prescribing a route

Test nearby inputs, alternative sequences, different timings and plausible competing causes. The aim is not exhaustive proof, but enough variation to discover whether the proposed repair generalises or merely recognises the incident.

Test outcomes while allowing implementation freedom

Where the internal path is not itself a requirement, test the observable result and prohibited effects. Accept multiple valid implementations. Characterisation tests for legacy machinery may still be useful, but they should not silently govern its replacement.

Compare additions with the simplest credible alternative

Before adding a new layer, compare it with removing or bypassing an existing one. A no-new-mechanism baseline is often more revealing than a more elaborate competing design. The baseline will not always win; its purpose is to make complexity earn its place.

Use evidence the implementation agent has not already absorbed

Neighbouring and held-out cases can expose repairs that merely encode visible examples. Where appropriate, use an independent reviewer or protected evaluation set that initially reports outcomes and failure classes rather than handing the implementer every case. A “hidden” suite readable by the coding agent is not a holdout.

Give deletion equal status with addition

A task should be closable by proving that a stage, adapter, fallback, rule or test is unnecessary. Temporary mechanisms should have removal conditions. When a replacement succeeds, delete what it replaced. Permanent dual paths are how the next tower begins.

Whether this development environment can be achieved within current dev platforms and with current coding agents remains to be seen.

A shared theory of purpose

Large systems do need shared intent; they do not need an encyclopaedic contract describing every permitted implementation.

They need a compact, revisable account of the jobs the system exists to perform, the outcomes that matter, the states that are genuinely unacceptable, the constraints that are hard, the mechanisms that are merely current choices, and the evidence that would justify changing or removing them.

Without that shared theory, the test suite becomes the de facto product definition and the issue tracker becomes the architecture. The purpose of such guidance is not to dictate every repair. It is to give coding agents enough context and authority to recognise when the right action is subtraction.

The standard of belief

A coherent explanation is not enough. Coding agents are extremely good at coherent explanations.

For any substantial new mechanism, the harder questions are:

• What class of failures does this prevent?
• What valid behaviours or future designs might it exclude?
• What evidence would cause us to remove it?

If there is no serious answer to the last question, the mechanism is unlikely to remain an engineering decision. It will become a fossil.

What to look for in your own repository

The reliability ratchet is a hypothesis other teams can test. Look for:

• Local regression rates falling while change latency and validation burden rise.
• Baselines that are repeatedly waived or refreshed upward.
• Tests that prescribe internal routes rather than observable outcomes.
• Temporary fallbacks with no removal conditions.
• Agents that spend more effort satisfying accumulated structure than solving the underlying problem.

None of these observations alone proves that a system is over-engineered. Together, they suggest that local reliability may be purchased by quietly narrowing the future.

Coding agents are capable of extraordinary work. The answer is neither to constrain them less indiscriminately nor to trust them without any hard boundaries.

It is to constrain the right things, and to preserve enough room for a better explanation, an unanticipated solution, or the discovery that yesterday’s successful fix should no longer exist.

Intelligence needs boundaries. It also needs room.