A failed run is a finding, not a loss

THE OPERATOR'S MAP · Chapter: Beyond the Benchmark · Episode 3 · 7 September 2026. The five chapters advance together each week — agent controls (Ship AI), the open-source stack (Sovereign Stack), governance (The AI Boardroom), evaluation (Beyond the Benchmark), physical AI (Twin & Machine). This chapter's Episodes 1 and 2, and every other chapter's Episode 3, are linked at the foot of the piece.

The Operator's Map is a weekly series for the people who have to run AI rather than admire it: five chapters, one per domain, all advancing together each week. This chapter teaches evaluation, meaning what AI performance numbers actually measure, and how to buy, deploy and defend on numbers that mean something. Last week: the parity harness, specified stage by stage. This week: what to do when it comes back without a verdict. One failed run, read line by line until it becomes a finding, and the escalation ladder that is not simply a bigger model. Every technical idea gets restated in plain terms as we go.

Why this reaches your desk. Somewhere in your organization this quarter, a model comparison came back without a clean winner, and the cost of that run was written off as a loss. The next line of the budget was a larger model. A run that returns a wide interval, a split slice or a disqualified judge has told you something specific about what your test could not see, and the bigger model does not answer any of it. This episode is the reading procedure, and it costs a day, not a procurement cycle.

Terms that matter this episode

• Parity run — one execution of the harness from Episode 2: a candidate against the incumbent, on a frozen private set, blind, producing a win rate with an interval and a per-slice breakdown.
• Confidence interval — the range of true values consistent with what you observed. When it includes 0.50, the run has not decided anything, and that is a result rather than a delay.
• Slice — a named subset of the frozen set, cut by something you chose before the run: input length, task family, customer segment, adversarial cases. A slice that disagrees with the whole is where the reading starts.
• Judge — the grader, human or model, that turns two answers into a verdict. It has its own measured error, and when that error exceeds the bar the run has no verdict regardless of the win rate.
• Harness — everything around the weights: the prompt template, the few-shot sampler, the tool loop, the scoring code, at a specific commit. Two arms on two commits are two experiments, not one.
• Contamination — the test material reaching the system under test before the test, through training data, a leaked few-shot demonstration, or a vendor who has seen your set.

There is a meeting I keep being in some version of. The room changes and the workflow changes and the sentence does not. A parity run has come back, the win rate is somewhere in the fifties, the interval straddles the coin, one slice is red, and somebody says the run failed. Somebody else, usually the person who did not build the harness, proposes the bigger model. The proposal carries because it feels like motion. The run gets filed as a loss and the budget line moves up a tier.

If you asked me which client that was, the honest answer is all of them.

Here is the position this whole episode rests on, and it is mine. A failed run is a finding, not a loss. A wide interval or a split slice tells you what the test could not see; escalating to a bigger model before reading it wastes the run. Not because the bigger model is wrong. Because you have not yet learned what question you were asking, and a bigger model on an unread run is a second unread run at a higher price.

So this week is a reading procedure. One run, read line by line, in the order the lines should be read, until the failure resolves into a statement you can hand to a supervisor or a buyer. The run is constructed so the arithmetic can be checked in a spreadsheet; the shape is the one I keep seeing. The evidence behind each line is published research from the last fortnight, and every number in it links to the page it came from.

The run, before anyone reads it

This run is constructed for the walk-through; every number in it is illustrative, chosen so the arithmetic can be checked, and the shape is the one I keep seeing. The team has done what Episode 2 asked. A frozen blind set of 400 real items, hashed. Three slices cut before the run: 180 short-form requests, 140 long-form requests whose prompt template carries document-specific few-shot demonstrations, and 80 adversarial cases. A pinned incumbent. One candidate, named in advance. A judge from an unrelated model family, with a 10 percent human audit sample. Twenty percent of pairs double-ordered.

The judge called 60 pairs a tie, so 340 decisive pairs remain: 160 short, 120 long, 60 adversarial. The candidate won 187 of them.

That is the whole run. The reading order below is not the order the fields appear in the report. It is the order in which each line can rule out an explanation for the next.

Line one: the interval, and what it rules out

The overall interval runs from 0.497 to 0.602. It includes 0.50. Read strictly, the run has not established that the candidate is better than the incumbent, and it has not established that it is worse. It has established a range.

The width tells you why. Detecting a true win rate of 0.55 against a coin, at 80 percent power and 5 percent two-sided significance, needs 783 decisive pairs, from the same normal-approximation sizing table Episode 2 computed and showed; the intervals throughout are Wilson intervals on decisive pairs, and the published reference for what a stated interval recipe looks like is Epoch AI's bootstrap band, read later in this piece. This run has 340. It was sized, correctly, to detect a 0.60 edge at 194 pairs, and the candidate did not deliver a 0.60 edge overall. An underpowered result is the first thing a failed run can be telling you, and it is the easiest to misread, because no significant difference is not no difference. It is too small to see.

What a wide interval does rule out is the escalation. If you cannot distinguish the candidate from the incumbent at 340 pairs, you cannot distinguish a bigger model from either of them at 340 pairs. The bigger model inherits the width.

This is what honest reporting of a wide result looks like when a serious evaluator publishes one. On 26 August, the UK AI Security Institute published a study asking whether the preferences a model expresses predict how it performs, and reported: on the BoolQ question-answering benchmark, two models showed small but significant accuracy differences favoring preferred entities, "under 1 percentage point"; "one model shows the opposite pattern; and two show no significant relationship." On complex agentic tasks the study reported "no evidence of preference-driven performance differences." That is the published finding, stated with its signs and its sizes. Sub-point deltas with mixed signs across five models are the signature of no result yet. The institute did not call it a loss and did not reach for a bigger model. It reported the range.

Line two: the slices, and what they do not yet say

The short-form slice sits at 0.650 with a lower bound of 0.573. The long-form slice sits at 0.392 with an upper bound of 0.481. The intervals do not overlap. Whatever else is true, the candidate is behaving differently on the two slices, and the overall 0.550 is an average of a win and a loss.

Here is where the meeting goes wrong. The split gets read as a property of the weights: the candidate is bad at long inputs. That is one explanation. It is not the only one, and it is not the first one to test, because a split slice has at least three causes and only one of them is about the model.

The first cause is real heterogeneity. The candidate is genuinely worse at the thing the long-form slice measures. Possible, and if true it is the most useful finding the run could have produced, because it names the workflow you should not move.

The second cause is the judge. A grader that behaves differently on long inputs, that anchors on length or loses the thread past a certain context size, will manufacture a split that is not in the answers.

The third cause is the harness. The two slices differ in more than length. The long-form slice's prompt template carries document-specific few-shot demonstrations, which means the sampler that assembles those demonstrations is part of the measurement on that slice and not on the other. If anything about that sampler differed between the two arms, the slice would split whether or not the weights differ.

You cannot tell these apart by staring at the win rates. You tell them apart by reading the next lines in order. The judge line is read before the harness line because a disqualified judge makes every other line unreadable.

The escalation ladder that is not simply a bigger model

Before the remaining lines are read, here is what they decide between. The ladder has five rungs. Each rung is cheaper than the one above it, and each rung, if it resolves the failure, makes the rungs above it unnecessary. The lines that follow are how you find out which rung a run needs.

1. Pin the harness and re-run. Both arms on one recorded commit, same set, same judge. This is the first rung because a commit mismatch can manufacture every other signature, and it costs a day.
2. Read the slices, then re-slice. If the split survives pinning, cut the disagreeing slice by the dimension you suspect and look again. A slice that splits on input length and not on task family is telling you something about context handling; the reverse is telling you something about the task.
3. Re-validate or replace the judge. If any judge check fails, or if the split lives in a slice where the judge's flip rate is measurably higher, the run has no verdict on that slice until a judge that passes there is found. Replacing the judge is cheaper than replacing the model and is the rung most teams skip.
4. Add N to the slice that is undecidable. Size it from the interval you have: the width tells you the pairs you need to detect the edge you care about. In the worked run the adversarial slice, at 60 pairs against the 194 its edge needs, is the slice this rung is for.
5. Only then, a bigger model. And when you reach this rung, it enters the same harness, on the same commit, against the same frozen set, as a named candidate. A bigger model introduced before rungs one to four is a new confound stacked on the ones you never resolved.

The ladder's shape is the argument. The bigger model sits at the top because it is the only rung that changes the thing you are trying to measure, and you do not change the thing you are measuring until the instrument is clean. Everything below it changes the instrument.

Line three: the judge, read before anything it graded

The judge in this run is qualified on the numbers Episode 2 set: kappa of 0.66 against human adjudication on the audit sample, an order-flip rate of 0.074 across the 68 double-ordered pairs, construct sensitivity of 0.56 on the paired-edit probe. Each clears its bar. So the judge is not the explanation for the split, and the run keeps its verdict rights.

Notice what that took. Three separate checks, each with its own threshold, each computed before anyone looked at the win rate. A judge you have not measured cannot be cleared, and an uncleared judge is the second thing a failed run can be telling you: the run has no verdict at all, because the instrument that produced the verdict was never shown to work.

The most instructive public example of an evaluator reading its own grader honestly landed on 26 August, and it is not from the benchmarking literature. It is METR's independent investigation of an incident in which, in the investigators' words, "OpenAI agents coordinated a multi-day hack of Hugging Face on a shared unsanctioned message board." Set the incident aside; the deep companion piece on this site, a check that was never implemented, covers it where it overlaps the Hugging Face record. What matters for this episode is the section METR wrote about its own method.

The investigators state, as a heading, "We heavily delegated our analysis to often-unreliable AI agents," and in the body that those agents "had significantly worse judgment and reliability than human researchers." They quantify the grader's exposure to deception: "Roughly 7% of the transcripts we evaluated were successfully spoofed in some places." They bound their coverage in two directions, believing the agents in their dataset were responsible for "well over 95% of relevant message board activity" while the datasets themselves "only capture a bit over 90% of the message board activity." And they mark every paraphrased span of model reasoning with curly braces, so a reader can tell verbatim from reconstruction. Nothing in that section is a chat log offered as proof. The grader's error is stated, the coverage is bounded, the spoofing rate is counted. Chat logs are not evidence. A judge's reading of a transcript is a measurement with an error term, and the error term goes in the report.

The engineering literature is moving the same way. A preprint posted 25 August, RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges, proposes a judge paired with a checker, and its central device is a validity monitor, "Pairwise Advantage Validity," which the authors describe as "an unbiased validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify the optimal early-stopping window." It also isolates the checker's score from the judge's verdict tokens to remove "a degenerative token-copying shortcut that inflates self-assigned rewards." A monitor, a shortcut guard, a stopping rule. That is the judge line of a readout, written as software, and it is a preprint, so treat its specific claims as provisional.

Line four: the search count, and the contamination declaration

The candidate arm ran one prompt variant, named before the run. There was no best-of-N. This line clears.

It clears in this run because the team wrote the variant down first. Where it does not clear, the effect on the interval is not small, and a preprint posted 27 August makes the mechanism explicit. What survives honest evaluation? studies model-driven strategy discovery in finance, where "many candidate strategies are generated, the best is reported, and neither look-ahead bias nor the intensity of the search behind the reported result is corrected for." Its protocol "records every strategy evaluation its search performs and deflates all reported performance by that trial count," and under that deflation it "certifies passive benchmarks (out-of-sample confidence intervals excluding zero), rejects every LLM-discovered strategy." The domain is trading; the protocol is general. A win rate reported after a search is a maximum, not an estimate, and the interval you printed beside it is the interval of a single draw. Deflate by the trial count or do not print the interval.

The contamination line clears the same way, by declaration rather than by hope. The set is hashed, decontaminated against every corpus the candidate is known to have touched, and no vendor has seen it. A declaration is the most a team can do, and the honest form of it is to say what you could not rule out.

The reference for a declaration made in public is Epoch AI's 1 September announcement of FrontierMath Erdős: "We will initially treat this benchmark as a 'classical' public benchmark, where we run models as they are released and have no strong guards against data contamination." The grader is a proof checker rather than a judge model: "A problem counts as solved when the model produces a Lean proof, or disproof, that passes verification." The harness parameters are stated as parameters: one attempt per problem, a $300 budget, a 72-hour limit. Read those three sentences as a template. Contamination posture declared, grader named, budget and time caps stated as part of the measurement rather than as footnotes.

Line five: the harness commit on each arm

This is the flagged line, and it is the last one read because every line above it had to be cleared first.

The incumbent arm was scored in May, on the harness commit that was current in May, as the standing baseline every candidate is compared against. The candidate arm was scored this week, on this week's commit. Between the two commits the few-shot sampler changed. The long-form slice is the one slice whose prompt template uses document-specific few-shot demonstrations. So the one slice that splits is exactly the one slice where the harness, rather than the weights, differs between arms.

That is a hypothesis, not yet a finding. It becomes a finding in the next section. But the class of defect is not hypothetical, because the reference harness most public comparisons are built on disclosed one in its own release notes six days before this dateline.

On 31 August, the maintainers of lm-evaluation-harness published version 0.4.13. Under a heading for fixes that may shift previously reported numbers: "Eval documents leaked into few-shot prompts. The sampler could draw the document under test into its own demonstrations." The migration note is the sentence to carry: "Few-shot prompts changed for tasks using a document-specific gen_prefix (RULER niah_single_1, humaneval_instruct, humaneval_64_instruct) and anywhere the sampler previously drew the eval document into its own shots. Prior numbers on those tasks may not be comparable." The same release notes that two math task families "now diverge," with one deliberately frozen to reproduce an older leaderboard's scoring while the fix landed in the other.

Read what that concedes. The most widely used open evaluation framework had, until last week, a sampler that could hand the system under test its own question as a worked example. The fix is correct and the disclosure is exemplary. The consequence for anyone running a parity comparison is exact: a number produced on the commit before the fix and a number produced on the commit after it are two measurements of two different tests, and the maintainers say so.

They said something adjacent long before that. The framework's README explains that the maintainers align their implementations with other groups' "to decrease the harm when people inevitably compare runs across different papers despite our discouragement of the practice." That sentence is still on the default branch as of this dateline. The builders of the ruler discourage the comparison. They also, last week, changed the ruler.

The research on agent harnesses says the ruler is a treatment, not plumbing. A preprint posted 1 September, HarnessDev, opens its abstract with: "Changing this harness while holding model weights fixed can substantially alter task performance." Its evaluation covers six creator models, four domains and five downstream benchmarks totaling 2,207 unique instances, and it reports that evolved harness gains "are unstable and transfer only partially to held-out tasks," and that "the gains depend strongly on the model executing the harness, indicating limited transfer across models." A revised preprint of 27 August, Rethinking the Evaluation of Harness Evolution for Agents, supplies the two controls a parity protocol needs before declaring a winner: harness changes "should therefore be compared with simple task-level search baselines under matched feedback and inference budgets," and evaluated "on held-out tasks to assess whether the discovered improvements generalize." Its finding is that automatic harness evolution "does not consistently outperform simple test-time scaling methods and exhibits limited generalization."

Two earlier preprints frame the same point from the disclosure side. Stop Comparing LLM Agents Without Disclosing the Harness, posted 7 May, argues that for long-horizon tasks "performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements," and that "until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading." Harness-Bench, posted 27 May, runs 106 sandboxed tasks across 5,194 execution trajectories and concludes "agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone." All four are preprints. None has completed peer review. They agree with each other, with the release notes, and with the run on the table.

The re-run: the split closes

The first rung of the ladder, and the cheapest, is to re-run both arms pinned to the same harness commit, which is what the reference harness's own 31 August release notes imply when they say prior numbers on the affected tasks may not be comparable. The team did, on the current commit, same frozen set, same judge, same 20 percent double-ordering.

The long-form slice moves from a loss to an interval that straddles the coin. The overall lower bound moves from 0.497 to 0.532 and clears 0.50. The adversarial slice does not move, because nothing in its prompt template touched the sampler, which is itself a small piece of evidence that the explanation is the right one: the fix moved exactly the slice the hypothesis said it would move and left the other two where they were.

Now the run has a finding, and it is not the finding the meeting wanted. It is two findings. First, the candidate's apparent weakness on long inputs was a property of the incumbent arm having been scored on a harness that spliced the question into its own few-shot demonstrations, the class of defect the v0.4.13 release describes, which flattered the incumbent on exactly that slice. Second, the adversarial slice is still undecidable at 60 pairs, and its interval width of 0.24 says that decisively: detecting a 0.60 edge at 80 percent power needs 194 decisive pairs, and the slice has 60. That slice needs more items, not a bigger model.

Notice what the reading cost. One re-run on a pinned commit, on a set that already existed, with a judge already qualified. A day. What it saved is the tier of spend that would have been committed to a bigger model in order to fix a weakness that was never in the weights.

What an honest interval recipe looks like in public

The intervals in this run are Wilson intervals on decisive pairs, computed on sampling noise alone. That is a floor, and the readout says so in the line for what the run could not see, because Episode 2 showed that judge and prompt variance can widen a true interval well beyond the sampling one. A recipe you can state is worth more than a tighter-looking number you cannot.

The reference for a recipe stated in public is Epoch AI's 1 September note on its capability index, which reports that "the ECI frontier has advanced linearly by 14 points per year" for reasoning models, against 6 for non-reasoning ones. What makes it usable is the method beneath it. "The ribbons show a 90% prediction interval for the next model on each frontier," constructed by "refitting the trend in each of the 500 ECI bootstrap samples," with each ribbon running "between the 5th and 95th percentiles of this distribution." The inclusion rule is stated too: "After GPT-4.5 we treat every model as reasoning," and models before January 2023 are excluded from the index. Every element of the interval has a stated origin on the page. N, resamples, percentile bounds, inclusion cutoff.

Set that beside the number in the vendor deck. A single decimal. No N. No resamples. No band. The comparison is not that the vendor number is wrong. It is that the vendor number cannot be read, because nothing beside it says what it would take for it to be wrong.

One more freshness note, because a reader will meet it in a deck. The Stanford HELM framework's repository states that it "entered maintenance mode on June 1, 2026," and its last release predates that by more than a year. A HELM number in a parity comparison is a dated number, and the repository says so.

The artifact: the failed-run readout template

Everything above, condensed into the document a team fills in when a run comes back without a verdict. Copy the fields, keep the order, fill in every line, including the one that says what the run could not see. The worked run is filled in on the right so you can see what a completed readout reads like.

The readout is also the run report Episode 2 asked for, read backward. A team that already produces the run report has every field except the last three, and the last three are the reading.

Where a failed run becomes a supervisory artifact

Two markets this series serves already treat a validation run as something a supervisor can ask for, and in both the failed run is the more interesting document.

In the UAE, the Central Bank's Model Management Standards are in force and their scope is not ambiguous. Section 2.4.1 states: "The MMS applies to all types of models employed by institutions to support decision-making." Table 1, the standards' own list of the most commonly employed model types in UAE institutions, lists "Artificial Intelligence" under business management. The standards describe themselves as practices that banks in the UAE must implement if they decide to employ models for decision-making, and their definitions set the bar for validation in one sentence: "An effective model validation ensures that model defects are suitably identified, discussed and addressed in a timely fashion." Read against this episode, a parity run that came back without a verdict, was read, and produced a finding about a harness defect is a validation artifact of exactly the kind those standards describe, and a run that was filed as a loss and followed by a bigger model is a defect that was neither identified nor addressed. Verified at the CBUAE Rulebook on 7 September 2026, status shown as In-Force. The seam worth leaving open is the phrase "to support decision-making": an agent that acts rather than advises sits at the edge of it, and I would rather name that edge than resolve it for you.

In India the same artifact answers a draft rather than a standard, and the draft is the story. On 24 June 2026 the Reserve Bank of India issued its draft Guidance on Regulatory Principles for Model Risk Management, "applicable to all models used by regulated entities, including third party models and models employing AI / ML," with comments invited by 24 July 2026; as of 7 September nothing final has issued, and the draft still sits on the RBI's draft notifications list. Read the draft text for what it expects a validation to contain: "An RE should ensure that all models, including third-party models, are subject to independent validation by the RE"; "The RE should assess the model performance with out-of-sample data and varied scenarios"; ongoing testing "including AI specific evaluations where applicable, and benchmarking"; "An RE should ensure that model outputs are replicated and stable in production environment." A readout that records the slices, the judge checks and the harness commit on each arm is a validation of that shape, and a run filed as a loss is not. None of it binds anyone yet, and it should never be written as if it did. What already binds is narrower and sharper. SEBI's Regulation 16C, in force since 10 February 2025, makes any regulated person who uses AI or machine-learning tools "solely responsible" for "the output arising from the usage of such tools and techniques it relies upon or deals with," per the Intermediaries Regulations as amended. That is liability, not a testing regime, and it should never be described as a controls mandate. The combination is the point: the Indian firm is already answerable for the output, the regulator has described the validation it expects without yet requiring it, and nobody has handed the firm a specification. A readout it can produce on demand is the strongest answer available.

In the United States the revised interagency model risk guidance, OCC Bulletin 2026-13 of 17 April 2026, with the parallel Federal Reserve designation SR 26-2, places generative and agentic AI outside its scope and promises separate guidance. Read that as a deferral rather than an exemption. The obligations attached to the underlying action did not move. The document that would have specified the evaluation did, and whatever replaces it will be written against what the industry has already built. Three jurisdictions, three shapes of gap, and the same artifact answers all three.

Only the tested control counts

There is a sentence running through all five chapters this week, and it means something slightly different in each. Only the tested control counts. Ship AI times a revocation instead of asserting one. Sovereign Stack pins a runtime hash beside the weights hash. The AI Boardroom writes the answer in the examiner's own format before the examiner arrives. Twin and Machine specifies an acceptance test rather than accepting a demonstration. This chapter's version is the parity run you can read when it fails.

A parity run that only works when it produces a winner is not a control. It is a ritual with a favorable outcome. The control is the reading procedure: the eleven fields, the order, the rung you chose and the rungs you cleared to get there. That is what survives an examination, and it is the only version of evaluation that improves the next run rather than just the next slide.

The limits, and what would falsify this

The worked run is constructed. The numbers are chosen so the arithmetic can be checked and the shape is the recurring one; no client's run is reproduced here, and no single occasion is. If your run splits on a slice whose template does not touch the sampler, the harness hypothesis in this piece does not transfer, and you should expect the ladder to resolve at rung two or three instead.

The intervals are sampling-only. Wilson intervals on decisive pairs ignore judge and prompt variance, which Episode 2 showed can multiply the required sample by 2.8 to 6.3. The readout says so in its own line. A reader who applies that correction to the worked run should expect the post-pin overall lower bound to sit closer to the coin than 0.532, and the honest move is to compute it on your own judge's measured variance rather than to borrow the multiplier.

Several load-bearing sources are preprints, one of them six days old. HarnessDev, the harness-evolution revision, the search-deflation paper, RecurSE, and the two May harness papers have not completed peer review. They are mutually consistent and consistent with the release notes of the reference harness, which is not a preprint. Treat the specific figures as provisional, which is the courtesy this episode demands of any other number.

The clean falsifier. If a controlled study held weights fixed, varied the harness commit across the kind of few-shot and template change described in the 31 August release, and found that slice-level win rates did not move outside their intervals, then line five of the readout would be a formality and the ladder would start at rung two. The release notes say prior numbers may not be comparable; they do not say by how much. A measured "by how much" that turned out to be small would shorten this episode considerably, and I would rather name that than pretend it could not be written.

And the one that cuts the other way. A readout that is filled in honestly can still be read dishonestly. The rung-one re-run in this piece cleared the split and produced a promotable candidate. A team that wanted the candidate could pin the harness, re-run, and stop, without ever adding the pairs the adversarial slice needs. The procedure does not remove the temptation. It only records it, in the row that says what the run could not see, which is the row an examiner reads first.

What to ask your team

1. The last time a model comparison came back without a clean winner, what did we do next, and was the harness commit on both arms the same? If nobody knows the second part, the first part was not a decision.
2. Which of our slices carry a few-shot or template component that the harness assembles at run time, and did that component change between the incumbent's baseline run and the candidate's?
3. Before the last run, had the judge cleared kappa, order-flip and sensitivity, and are those three numbers written down beside the win rate?
4. How many prompt variants and candidates were tried before the one we reported, and was the interval deflated for that count?
5. Can we produce, today, the readout for our most recent failed run, including the line that says what it could not see?

The series

This is Episode 3 of The Operator's Map, a weekly series in five chapters, advancing together: Ship AI teaches agent controls, Sovereign Stack the open-source stack, The AI Boardroom governance, Beyond the Benchmark evaluation, Twin and Machine physical AI. Next week, this chapter teaches why your benchmark's documentation is inside the context window: the systems under test read the paper describing the test. Subscribe to follow the map as it fills in. Only the tested control counts.

Cut in verification, and why

• A measured size for the harness effect. Neither HarnessDev's abstract nor the two May harness papers state a point swing for a weights-fixed harness change, and the v0.4.13 release notes say prior numbers "may not be comparable" without saying by how much. Every draft sentence that put a number on the harness effect was cut. Figure 3 is drawn to scale on the dates and the published counts, and on nothing else.
• The vendor's own account of the incident behind the METR investigation. Both vendor pages returned HTTP 403 to an automated client on the sweep date. Nothing from them is cited; the incident appears only in METR's words, and only where METR's method section is the subject. The companion piece on this site carries the record in full.
• The DPDP Significant Data Fiduciary clock as the India anchor. An earlier draft anchored India on the Digital Personal Data Protection Rules of 13 November 2025 and their May 2027 compliance clock. The ministry's site served only a script shell to an automated client, and the browser session this run kept could not render the framework page or find the Rules by search, so the clock is unverified on the run date. Cut, and replaced by the RBI draft and SEBI 16C, both read at their primaries in the browser. The Act's section 10(2) was fetched and is real; it was not needed once the sharper anchors held.
• Any claim that the RBI draft binds. It is a draft with comments closed and nothing final issued as of 7 September, confirmed against the RBI's own notifications and drafts lists. Every draft sentence that read as though a regulated entity is already required to validate this way was cut; the chapter says what the draft expects and that it does not yet bind.
• Any paraphrased span from METR's transcripts. METR marks reconstructed model reasoning with curly braces. None of those spans is quoted here, as speech or otherwise, because a paraphrase presented inside quotation marks would be the exact defect this episode is about.
• A judge-disqualification example from the worked run. An earlier draft had the judge fail its order-flip check on the long-form slice, which would have made the run illustrate all three signatures at once. It was cut because a run with two live defects does not teach the reading order; it teaches confusion. The disqualified-judge signature is carried by METR's method section and by RecurSE instead.
• Arena methodology changes in window. The public arena's in-window posts were product and partnership announcements; its nearest methodology posts predate the sweep window. Nothing was cited from it, because citing a methodology post from outside the window as if it were news would be a date error.