The document was mine, it was a week old, and it was wrong.

It was a critical analysis of the system I am building — a good one, written carefully, the kind of thing that gets linked to when somebody asks what state the work is in. Re-reading it before citing it, I found it was silent on six of the seven components added since it was written, and that an integration inventory beside it listed sixteen where there were now thirty-six.

Neither of those is a bug. Nothing broke. Both documents were accurate on the day they were saved and had been quietly decaying ever since, in the specific way that documents decay: not by becoming obviously false, which someone would notice, but by becoming incomplete while remaining fluent.

What made it worth writing about is that this is the same failure the system underneath those documents exists to prevent. That system's whole argument is that an assertion about what a machine did has to be bound to evidence produced at the time, because an assertion maintained separately from the thing it describes will drift. I had built that argument into the substrate and not into the prose describing it.

So I went back through the whole review cycle looking for the shape rather than for bugs. Six defects. Not one was a coding error.

The six

FIGURE 1 · THE DEFECT LEDGER Every one of these was found while the suite was green. WHAT IT SAID WHAT WAS TRUE THE TESTS WHAT CAUGHT IT An error message named a command-line flag the flag did not exist green a test that reads the binary’s own output A published recall figure for the classifier that configuration was not the one running green running the benchmark against the live configuration The packaged app installs a tray icon the tray never appeared in a packaged build green installing the package and launching it A signing document promised an ignore rule the rule was not there — for the two file types that matter green a test that reads the ignore file, not the prose about it A config comment: “this does not affect Windows” it applied to every platform, including Windows green installing on Windows and reading the install path A shortcut sheet described a keyboard route the route had been removed; the sheet described a ghost green a test that reads the keybindings and counts Six defects. Six green suites. Not one of them was a coding error. Every failure lived in a sentence, a flag name or a configuration comment — artefacts a test runner never reads. Prose has no compiler. Neither does a configuration comment, a shortcut sheet, or a promise in a signing document.

Take them one at a time, because the interest is in how ordinary each one is.

An error message named a flag that did not exist. The binary, on failing, told the operator to re-run with a particular option. The option had been renamed. Everything about the code path was correct — it detected the condition, it exited non-zero, it printed a helpful message — and the help was a lie. A user following it precisely would have got an unrecognised-argument error and concluded the tool was broken.

A published recall figure was quoted for a configuration that was not running. The classifier's documented performance came from a specific model at a specific quantisation and a specific context length. What was actually loaded had drifted from one of those. The number was real, published by someone else, and not a measurement of the thing in front of me. This is the most dangerous defect in the list, because it is the one a reader is least equipped to check.

A tray icon that never appeared in a packaged build. It appeared when run from an unpacked directory, which is how it was tested for months. Packaging changed a path resolution, the tray silently did not install, and every test that exercised the tray logic continued to pass because the tray logic was fine. What was broken was the relationship between the logic and the package, and no unit of that relationship had a test.

A signing document promised an ignore rule that was not there. It said the repository ignored the two file types a purchased code-signing certificate actually arrives as. It did not. The consequence of that gap is a private key committed to version control — which is not a defect that degrades performance, it is a defect that ends a signing identity. And it lived in a sentence.

A configuration comment asserted that a setting did not affect Windows. The setting was declared at the top level, which applies to every platform. So Windows took a fix written for Linux, installed itself into a directory named for the Linux package, and produced an uninstaller named for it too, while the system's own program list showed the correct product name. The comment directly above the setting said Windows was unaffected. That comment is why nobody looked.

A shortcut sheet described a screen that had stopped existing. The keyboard sheet documented a route to a destination that had been removed in a redesign. The sheet was not a stale draft in a folder — it was rendered in the running product, in front of the user, describing a product that had changed underneath it.

Six failures. Six green suites. And the suite is not thin: six hundred and seventy-eight Rust tests, forty-four in the console, sixty-eight in the desktop application, six hundred and seven in the machine-learning package, with continuous integration, coverage and bill-of-materials jobs all passing.

I am not citing those numbers as an achievement. I am citing them as the setup for the only sentence in this piece that matters.

Why no amount of coverage would have helped

The instinct on reading a defect list is to ask what test was missing. It is the wrong question here, and understanding why it is wrong is the whole argument.

A test asserts something about the behaviour of code. To catch any of the six above, a test would have to assert something about an artefact that is not code: the text of an error message measured against the set of real flags; a model's loaded configuration measured against a figure in a document; a packaged installer's behaviour measured against a claim about its behaviour; the contents of an ignore file measured against a promise in a different file; a configuration comment measured against the configuration it sits above; a rendered help sheet measured against the routes that exist.

None of those artefacts has a compiler. That is the property they share, and it is the whole property. Code has something that reads it and objects when it is internally inconsistent. Prose does not. A configuration comment does not. A packaging manifest is parsed but never checked against the sentence describing it. Every one of these six defects survived because it lived in a place where being wrong has no mechanical consequence.

This is not an argument that the tests were pointless. The tests were doing their job, and their job is a different job. It is an argument that a green suite is evidence about one class of artefact, and that a team reading a green suite as evidence about the whole system is making an inference the suite does not support.

The general form of this argument is the reason the substrate exists, and it is worth stating once here without arguing it, because it belongs to a different piece. A security control asserted in a document is in the same position as these six defects: it is a sentence with no compiler. The whole design consequence is that an assertion about what a system is permitted to do has to be enforced somewhere a claim cannot reach — and that when it cannot be, the artefact should say so rather than imply otherwise. That is the subject of the companion piece on what a receipt is entitled to claim.

The countermeasures, and what makes them a class

Every one of the following exists because an earlier defect got through. None of them tests behaviour.

FIGURE 2 · THE COUNTERMEASURE CLASS Tests that read the artefacts a compiler never sees. WHAT IT READS WHAT IT ASSERTS THE CLASS IT CLOSES The binary’s own output help text, error strings every flag and command the binary names is real documentation describing a product that stopped existing has caught six changes — all of them mine Both guard implementations each reads the other’s source they agree knob for knob two implementations of one rule, drifting apart The console’s keybindings the shortcut sheet documents the same count a help document outliving the feature it describes The ignore file itself not the prose about it the promised rules are present a document promising a configuration nobody made Every CI action reference resolved at its pinned revision the declared runtime is current, across the whole repository a warning that reports only the jobs that happened to run Each was written because an earlier defect got through. None of them tests behaviour.
  1. A test that reads the binary's own help and error output, and requires that every flag and command the binary names in its own output actually exists. It has caught six changes since it was written. All six were mine, which is the correct result and also a slightly uncomfortable one.
  2. A test in which two implementations of the same guard — one in Rust, one in Python — read each other's source and must agree knob for knob. Two authoritative implementations of one rule is the condition under which drift is not merely possible but scheduled.
  3. A test that reads the console's keybindings and asserts the shortcut sheet documents the same count. It cannot tell you the sheet is well written. It can tell you the sheet describes a product that still exists.
  4. A test that reads the repository's ignore file rather than the prose describing it. The prose was the thing that was wrong; reading the prose would have confirmed the error.
  5. A sweep that resolves every continuous-integration action reference to its own manifest at the pinned revision and checks the declared runtime — rather than trusting the platform's deprecation warnings, which report only the jobs that happened to run.

The last of those five is the one I would generalise hardest, because the reasoning behind it recurs. Three times in two commits, a platform's deprecation annotation described the jobs that had happened to run rather than the repository. Two workflows had not been exercised, so their contents never warned; one reference sat inside a composite action nobody edits. Each fix cleared what the annotation named, went green, and the annotation moved to something else — which is the behaviour of an inventory being revealed one item at a time by a process nobody designed to be an inventory. The fix that ended it was a sweep rather than a reading.

Two failures in the checking rather than in the code

The cycle ended by producing two more defects, and both were about the gates rather than about what the gates were checking.

FIGURE 3 · GATE COVERAGE A gate run by habit covers the language you were thinking about. FIFTY-FIVE COMMITS Rust format + lint ran before every commit Node · console suite ran before every commit Node · desktop suite ran before every commit Python format ran before none of them — while Python was being edited AND WHERE THE CHECK COULD LOOK python/…/ inside the pattern · covered python/…/ inside the pattern · covered python/…/ inside the pattern · covered ml/ outside the pattern · invisible 186 lines of logic landed in the one directory the gate could not see — beside launchers of 28 lines, in a directory whose own readme calls that “an ungoverned code surface inside a governance substrate.” CI could not have caught it. Reading the readme did. Run the gate for every language you touched — then check the gate can see where you put the file.

The first: one continuous-integration step went red on a formatting check for a Python file. The formatting itself was one wrapped function call — thirty seconds of work. The interesting part is how it reached the main branch at all. The Rust format and lint checks and both Node suites ran before every one of fifty-five commits. The Python check ran before none of them. During a period in which Python was being actively edited.

Nobody decided that. It is what happens when gates are run by habit: you run the checks for the languages you are thinking about, and you are thinking about the language you consider the project to be written in. A four-language repository run this way is a one-language repository with three unguarded surfaces.

The second is worse, and it is the one I would want a reader to take away. Running the sweep locally showed that a file written the same day was also unformatted, and the gate had said nothing at all — because the check globs a path pattern, and the directory that file was in does not match it.

The gate could not see where the file was. A hundred and eighty-six lines of logic had accumulated in that directory, beside launcher scripts of twenty-eight lines. The directory's own readme forbids exactly that, in its own words, calling it an ungoverned code surface inside a governance substrate. Continuous integration could not have caught it — the file was invisible to the check that would have flagged it. The only thing that caught it was reading the readme sitting next to it.

The implementation moved into the gated package with eight tests, one of which asserts that the launcher stays a launcher. But the lesson is not about that directory. It is that a coverage gap and a coverage failure look identical from the outside: both produce silence. A gate that runs and passes and a gate that never ran are the same colour on a dashboard.

The one no countermeasure covers

Everything above has a fix, which makes it comfortable material. This part does not, and it is the reason the piece is worth writing.

While correcting the false claim in the signing document — the ignore rule that was promised and absent — the correction nearly introduced a new false claim. Writing up the corrected status, I was about to record that continuous integration had never built one of the platforms. It had built all four, two days earlier.

It felt like tightening the document. That is the part I cannot design around. Every instinct said this was the careful move: I had just been burned by a document overclaiming, so I was correcting in the direction of claiming less, and claiming less felt safe. A document that is stale in the direction of understating is still a document nobody can trust — and it is much harder to catch, because the reflex that produces it is the same reflex that produces good corrections.

None of the five countermeasures reaches it. A test that reads the binary's output cannot check a sentence about what continuous integration did last Tuesday. The only thing that caught it was checking the claim against a different document before committing it — which is a habit, not a mechanism, and habits are what this entire piece is arguing you cannot rely on.

I do not have a good answer. The closest I have is a weak one and I will state it as weak: claims about what happened, as distinct from claims about what a thing does, should carry the artefact that shows it — the run identifier, the digest, the timestamp. Not because that prevents the error, but because it makes the error checkable by someone who is not me. That is a smaller claim than a countermeasure and it is what the evidence supports.

What would falsify this

Two things, and they are worth stating because a report about one's own work is the easiest kind of writing to grade generously.

The first: if a team can show a test suite that reliably catches this class — assertions about prose, packaging and configuration comments, caught by ordinary testing rather than by artefact-reading checks — then my claim that the suite is structurally incapable is too strong, and the correct conclusion is that my suite was in the wrong shape rather than that suites are. I have not seen it. I would want to.

The second is about generality. Six defects in one review cycle of one system, found by the person who wrote the defects, is not a study. It is a sample of one with an obvious observer effect: I found these because I went looking for a shape I had already decided was interesting, which is the condition under which people find shapes. What I would defend is narrower than the piece might imply — that this class exists, that it is invisible to the suite, and that artefact-reading checks catch instances of it. Not that it is the dominant defect class in software, which I have no basis to claim.

The provenance for everything above is deliberately boring: test counts from running the suites, with the Rust figure re-verified from a completely clean build directory; install results from the actual installer on the actual operating system; the confinement and coverage findings from running the thing rather than reading about it. None of the six defects in this report was reachable from an artefact. Every one of them was found by running, installing, or reading a file that sat next to the file being changed.