At ten to nine last Tuesday morning the first phase of my pipeline rebuild went in as one commit: 275 files and 797 passing tests. Two review agents read it before lunch and came back with seven blocking defects. Five of the seven sat in the gate, the code whose only job is to refuse a unit, and the other two in the measurement that feeds it. In every case the code had been running against something empty.
What the gate had been reading
The gate suite never loaded the repository's own config directory. It built a Config object by hand from fixture data and tested against that. The fixture and the real config had drifted, and the drift was measurable. Eight of twelve accounts, nine of thirteen lanes and seven of eleven approved routes existed only in the real file, so no test had ever gated them.
The word list went the same way. The fixture copy of the banned gambling terms carried bet and betting. The shipped file carried neither. It had free bet, place a bet and a column of operator names, and the two plain words a caption is most likely to contain were missing. A rule dated 7 August in my own canon, binding on every lane and on the ads, had been unenforced in production the whole time, with the suite green.
A policy that named tests nobody had written
Each rule in the safety file carries a column naming the tests that enforce it. There was a test to check that column. It asserted that every entry began with the string test_. I counted the names on Tuesday's commit this afternoon: 45 unique, and 17 of them resolved to no test function anywhere in the tree. The file read as a list of guarantees, and the check built to tell a guarantee from an intention could not tell them apart.
Then the history. The gate takes a history object so it can enforce the rotation and repeat rules, the ones that stop a lane repeating itself. When no history is passed it falls back to a null one, which has no record of anything, and in production nothing passed one. Every rotation check the suite had ever passed was a check against an object with nothing to remember.
The readings that joined to nothing
The ledger held 764 metrics rows and 184 units, and the join between them returned zero rows. Every reading had been written without an anchor to the unit it measured. The health view returned nothing and every trailing median was None, and nothing complained. The live measurement path had the same shape one layer up. The importer that reads my performance log dropped every recent post row, because the desks append at the tail under the current header and the parser read each row against whichever header enclosed it. Fifteen Threads permalinks went with those rows. The first real fetch after that was repaired retrieved thirteen pages at HTTP 200 and matched none of the four view-count patterns the code was looking for.
So on Tuesday morning the pipeline could not load its policy, could not enforce its word list, could not see its own history, could not join a reading to the unit it measured and could not read a view count off a live page, and 797 tests passed.
What the tests were proving
Every one of those tests was true. The gate ran, the rules loaded and the metrics function returned a value, and not one of them presented the thing the code exists to refuse and asserted the refusal. "Loads the policy" and "enforces the policy" are different tests, and I had written the first and read it as the second, many times over, in a suite large enough that its size stood in for its coverage.
A fixer agent closed all seven the same morning, in a second commit at twenty past ten. The gate suite now loads the real config directory, and a test over the shipped word list asserts that the terms the design names are present. A metrics row that names no unit is refused twice, once in code and once by a trigger on the table itself. The tests the policy names are now checked against what exists. Nine were written, eight were moved to a deferred column that carries a window and a reason for each, and the deferred list is pinned as a set that may only shrink. The null history stays in the code as a seam for tests, and production reads a real one off the ledger. Sixty-eight tests were added. On the rebuilt ledger 689 of 689 readings join.
The question I now ask of a green suite
I did not find any of this. Two reviewers found it by reading the code rather than the test report, and the question they were answering was what input the guard had actually seen. When an agent tells me a safety component has green tests, that is the question I ask back. If the answer is a fixture, a default, a null object or an empty table, the tests prove the code executes, and I want the one that hands the gate the thing it should refuse and watches it refuse.
The suite is past three thousand this week. At ten to nine on Tuesday it stood at 797, and eight of the twelve accounts it protects had never been gated once.