The leaderboard's own instructions say the score moves when you change the batch size

THE OPERATOR'S MAP · Chapter: Beyond the Benchmark · Episode 2 · 28 August 2026. The five chapters advance together each week — agent controls (Ship AI), the open-source stack (Sovereign Stack), governance (The AI Boardroom), evaluation (Beyond the Benchmark), physical AI (Twin & Machine). This chapter's Episode 1, and every other chapter's Episode 2, are linked at the foot of the piece.

The Operator's Map is a weekly series for the people who have to run AI rather than admire it — five chapters, one per domain, all advancing together each week. This chapter teaches evaluation: what AI performance numbers actually measure, and how to buy, deploy and defend on numbers that mean something. On Tuesday: why the leaderboard measures the model and never measured whether the agent is safe. Today: the instrument that measures your workflow instead, specified stage by stage, so a competent team can build it from this page. Every technical idea gets restated in plain terms as we go.

Why this reaches your desk. Every model decision your company made this year rests on a number somebody read off a public ranking. That number was produced on someone else's tasks, graded by someone else's grader, and published without the range of values it is actually consistent with. This episode specifies the instrument that produces a number you can defend to a supervisor, a buyer or a board, and it is buildable in a fortnight by two engineers.

Terms that matter this episode

• Parity harness — a private evaluation rig that scores a candidate model against the one you already run, on your own tasks, with the identity of both hidden from the grader.
• Blind set — the held-out tasks the harness scores on. Frozen before the first run, never trained against, never sent to a vendor.
• Paired comparison — the same input answered by both systems and judged side by side. The pairing is what removes task difficulty from the measurement.
• Confidence interval — the range of true values consistent with what you observed. The single number is a summary of the interval, never a substitute for it.
• Statistical power — the probability your run detects a real difference of a stated size. An underpowered run mostly returns no result, which is not the same finding as no difference.
• Promotion gate — the written rule that decides whether a candidate replaces the incumbent. Written before the run, or it is not a gate.

Start with the instrument already on your team's slide, and read its own instructions.

The public rankings most open-weight comparisons are quoted from are produced with a shared evaluation framework, and that framework's maintainers wrote down what they think of the way people use it. From the lm-evaluation-harness documentation, explaining why they align their implementations with other groups': they do it "to decrease the harm when people inevitably compare runs across different papers despite our discouragement of the practice."

The people who built the ruler have asked you, in writing, not to do the thing your slide does with it.

Then read the leaderboard's own reproduction section. To reproduce the published figures you are told to install a specific fork, "as our PRs are not all merged in it at the moment," and then this, verbatim, under a bold Note: "You can expect results to vary slightly for different batch sizes because of padding." That sits on the leaderboard's about page, written by the people who publish the scores.

Sit with what that sentence concedes. The score is not a property of the model alone. It is a property of the model, the harness version, the prompt template, the few-shot count, and a memory-management decision about how sequences get padded into a batch. Change the last one and the number moves. The number moves, and the ranking is published to two decimal places with no interval printed beside it.

Two models a fifth of a point apart on that board are not distinguishable by that board. Nothing in the published artifact says they are, and its own documentation says they might not be.

The interval that was never printed

Here is the discipline the rest of this episode is built on, and it fits in one rule: never quote a point estimate without its interval.

Take a concrete case. A team evaluates a candidate model on a screening task, runs eight examples, gets seven of them right, and reports 87.5 percent recall. It goes in a deck.

Now compute what that observation is actually consistent with. Seven out of eight, at 95 percent confidence, using the Wilson score interval:

• Point estimate: 7 ÷ 8 = 0.875
• 95 percent Wilson interval: 0.529 to 0.978
• Interval width: 0.449

The true recall could be 53 percent. It could be 98 percent. The deck says 87.5 percent, and the deck is not lying. It is reporting the midpoint of a range that runs from barely better than a coin to nearly perfect, and dropping the range. That is not a rounding problem. That is the entire finding, deleted.

The arithmetic is reproducible. The Wilson interval for a proportion is

center = (p̂ + z²/2n) / (1 + z²/n)

half-width = (z / (1 + z²/n)) · √( p̂(1−p̂)/n + z²/4n² )

with p̂ = 0.875, n = 8, z = 1.96. Put those in and you get 0.5291 and 0.9776. Anyone can check it, which is the reason for showing it rather than asserting it.

Widen this out and it stops being a beginner's error. An April 2026 study of measurement error in evaluation pipelines, Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking, reports that "naive standard errors are 40 – 60% smaller than the TEE-corrected SE" once the variance contributed by judge choice, temperature and prompt design is accounted for. The same work reports that naive 95 percent intervals lose coverage as the sample grows, while corrected intervals hold at 95 percent.

Read that second clause twice. Collecting more data does not fix this class of error. It tightens the interval around a number that was displaced by something other than sampling. More evidence, more confidently wrong.

Translate the correction into the only currency that matters here. If the true standard error is between 1.67 and 2.5 times the naive one, then variance is between 2.8 and 6.3 times larger, and the number of examples you need is between 2.8 and 6.3 times larger too. Every sample-size figure below is therefore a floor, and the table says so where it sits.

The instrument, stage by stage

What follows is a full specification. Eight stages. At each one I name the specific wrong conclusion that stage exists to prevent, because a stage whose failure mode you cannot name is a stage that gets quietly dropped when the schedule tightens, and the dropped stage is always the one that produces the wrong answer.

Stage 1 — Capture tasks from traffic you already serve

Build it. Sample real requests from the workflow you intend to change. Not synthetic tasks, not the vendor's demo set, not what the team imagines the work looks like. Stratify by the dimensions you know matter: request type, input length, customer segment, and time of arrival. Strip or tokenize anything that cannot leave your boundary, and for a regulated Indian entity or a Gulf subsidiary that boundary is a legal one rather than a preference, so it gets resolved at stage one instead of at stage six.

Target 400 to 1,000 captured items before filtering. Expect to lose roughly a third to redaction and near-duplication.

The wrong conclusion this prevents: the model is good at our work. Without capture from real traffic you have measured the model on a distribution somebody else chose. A public benchmark's distribution is a design decision its authors made for their own purposes, and it carries no obligation to resemble yours. Every claim about how a model handles your work that is derived from a public score is an extrapolation across a gap nobody has measured.

Stage 2 — Freeze the blind set, and decontaminate it

Build it. From the captured pool, draw the held-out set. Split by source and by time, not at random: hold out a later period, so the set is not a shuffled sibling of anything used for prompt development or tuning. Decontaminate it, with n-gram and embedding-level deduplication against any corpus that touched a candidate. Then freeze it before the first run, and record a hash of the frozen file in the run log.

Size it at 300 or more items for judged tasks, and 1,000 or more if you are computing classifier metrics such as recall or F1, where the interval at a given n is much wider. Reserve 20 percent for adversarial and edge slices deliberately, and report those slices separately, because a mean over a set containing hard cases hides them rather than measuring them.

Never send the frozen set to a vendor. A set a vendor has seen is a training set.

The wrong conclusion this prevents: the model can do this. Without freezing and decontamination you are measuring memorization and calling it capability. That failure is not hypothetical even for mature public instruments: this chapter's Episode 1 covered an audit that found a majority of the hardest unsolved instances in a widely trusted agentic benchmark resting on tests too narrow or too broad to distinguish a solution from a non-solution. A frozen, private, decontaminated set is the only version of this you control.

Stage 3 — Pin the baseline, with a version and a date

Build it. The comparison is against the system you actually run today, in the harness you actually ship: same tools, same retrieval, same system prompt, same temperature. Record it as a triple of model, version and date. Re-pin quarterly.

If you are choosing among several candidates rather than testing one, name the entire candidate list before the run and record it. This is the stage that feels like bureaucracy and is not.

The wrong conclusion this prevents: we improved. Beating a stale baseline is a calendar artifact, not a result. And a more expensive error hides one level above it: if you evaluate candidates until one wins and then report that one, you have run a hidden multiple comparison, and the winner's margin is partly the maximum of a noisy draw.

The arithmetic is unforgiving. Testing k independent candidates at the 5 percent level, the probability that at least one clears the bar by chance alone is 1 − 0.95^k:

• 1 candidate — 5.0 percent
• 3 candidates — 14.3 percent
• 5 candidates — 22.6 percent
• 10 candidates — 40.1 percent

Ten quiet attempts and it is close to a coin flip that something looks significant with no underlying difference at all.

The same mechanism, run at scale and undisclosed, is what the April 2025 study The Leaderboard Illusion documented on a major public arena: providers testing many private variants before release and disclosing selectively, with one case reported at 27 private variants tested ahead of a single launch. That study identifies the providers by name; this piece describes them by class, because it is third-party research rather than vendor self-disclosure, and it is the mechanism that transfers rather than the roster.

The mechanism deserves its own arithmetic. If scoring noise has standard deviation σ and you take the best of k draws, the expected reported score sits above the true one by roughly the expected maximum of k standard normal draws:

• best of 3 — about 0.85σ above the truth
• best of 5 — about 1.16σ
• best of 10 — about 1.55σ
• best of 27 — about 2.00σ

Two standard deviations of apparent improvement, produced entirely by the reporting rule. No training run required.

Naming the candidate list before the run removes this from your own process. It costs one line in a document.

Stage 4 — Blind the comparison, properly

Build it. For every item in the frozen set, generate the incumbent's answer and the candidate's answer. Then, before anything is scored:

1. Strip identity. Remove model names, version strings, provider markers, and any formatting fingerprint that identifies the source: trailing sign-offs, characteristic list styles, disclaimer boilerplate. Normalize whitespace.
2. Strip history. No prior scores, no revision numbers, no attempt counters, nothing in any field the grader can see that says which one is new.
3. Randomize order. Independently, per item, decide which answer appears first. Record the mapping in a file the grader never reads.
4. Score both orders on a subset. For at least 20 percent of items, present the same pair in both orders and count how often the verdict flips. That flip rate is your position-bias measurement, and it goes in the report.

The wrong conclusion this prevents: the grader preferred the better answer. It may have preferred the answer it was told was newer.

This is measured, recently and at scale. In Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence, submitted 26 August 2026, the authors ran 185,271 successful evaluations out of 192,000 attempted, across eight models and three prompt conditions. Seven of the eight showed statistically significant anchoring. Effect size reached Cohen's d of 0.71. The anchored fields were mundane metadata: revision, attempt, prior score. And the operational number is the one to keep. Anchored metadata blocked 48 percent of error corrections and reversed 10.18 percent of judgments that had been correct.

Half of the corrections a grader would otherwise have made, suppressed by a version number sitting in the context.

Position bias is documented separately. The June 2026 study Reliability without Validity — 21 judges from nine providers, 118 runs, roughly 541,000 individual judgments — reports test-retest reliability above 0.95 coexisting with position bias above 0.10 in two production-deployed judges. Its authors name that pair a consistency–bias paradox: a grader can be almost perfectly repeatable and still systematically wrong in the same direction every time.

Stage 5 — Validate the grader before you trust a single score

Build it. The judge is an instrument, and an unvalidated instrument produces assertions rather than measurements. Four things, all reported:

1. Independence. The judge comes from a different model family than either arm, and from a different family than anything used to generate training data for either arm.
2. Chance-corrected agreement. Take a human-adjudicated audit sample of at least 10 percent of the frozen set. Report Cohen's kappa, not raw agreement.
3. Construct sensitivity. Build a small paired-edit probe. Take items and make two kinds of edit: one that preserves what you are trying to measure, one that genuinely changes it. Report invariance (verdict unchanged under the preserving edit) and sensitivity (verdict changed under the changing edit). Both numbers, always.
4. Order-flip rate, carried up from stage 4.

The wrong conclusion this prevents: our judge agrees with humans 80 percent of the time, which is as good as humans agree with each other, so it is fine.

That sentence is the most quoted defense of automated grading in the field, and it descends from a real result. The 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena reported strong judges "achieving over 80% agreement, the same level of agreement between humans," and named position, verbosity and self-enhancement biases in the same abstract. The finding was honest. What happened to it afterward was not: the agreement number traveled and the caveats did not.

Two corrections have since landed on it, and both are arithmetic you can run yourself.

Correction one: raw agreement is not chance-corrected. Reliability without Validity states the problem plainly — exact-match agreement "does not correct for chance and systematically overstates discriminative ability" — and measures the gap at 33 to 41 percentage points of kappa deflation on the canonical benchmark. Work the mechanism. Suppose your judge agrees with a human on 80 of 100 pairs, and suppose both judge and human pick the first-listed answer about 70 percent of the time. Expected agreement by chance alone is 0.7 × 0.7 + 0.3 × 0.3 = 0.58. Kappa is (0.80 − 0.58) ÷ (1 − 0.58) = 0.52. Eighty percent agreement, and a little over half of the available agreement above chance. Not worthless. Not what 80 percent sounded like.

Correction two: the validation set may be easier than it looks. In A Judge Should Know What Changed: Construct Validity for LLM-as-a-Judge Evaluation, submitted 25 August 2026, the authors audited five public label sets and found that surface-only predictors reproduce 55 to 67 percent of the labels in paired mode, including 67.4 percent of the human votes on the benchmark that agreement figure comes from. Two-thirds of those human preferences are recoverable from style alone, without reading the content.

Redo the boast against that floor. If a surface-only predictor reaches 67.4 percent and your judge reaches 80 percent, the share of the available headroom it captured is (0.80 − 0.674) ÷ (1 − 0.674) = 0.387. Thirty-nine percent of the distance from reads the formatting to reads the answer.

The same paper supplies the number that makes the whole category legible. Across 7 judges and 4 domains, at matched invariance of 0.90 or better, judges averaged invariance S = 0.945 and construct sensitivity R = 0.319. They almost never change their verdict when nothing important changed, and they change it less than a third of the time when something important did. The paper also separates the two kinds of change it tested: sensitivity to a shift in scope ran at 0.383, sensitivity to a shift in strength at 0.262, a gap of 0.121 with the same sign for all seven judges.

There is a rule inside that finding worth naming: report both invariance and sensitivity, never a single grader score. It is the same rule as never a point estimate without its interval, applied one level up.

One honest complication, because it cuts against the easy version of this story. That same 541,000-judgment study found verbosity bias small, below 0.011, across its cohort under a single pairwise rubric. The folk wisdom that graders simply prefer longer answers did not survive its own measurement at that scale. Keep length normalization in the rubric as hygiene, but the headline defect is not verbosity. It is insensitivity.

A second, independent probe of the same joint landed on 24 August 2026. The Limits of Automatic Evaluation of Creativity in Large Language Models collected human evaluations of human-written and machine-written short stories across 11 dimensions and reports that automated judges "exhibit a systematic preference for AI-generated stories," while widely used automatic metrics showed "near-zero alignment with human judgments." On open-ended work, the grader has a taste, and the taste is for the kind of text the grader is made of.

Stage 6 — Size the run before you run it

Build it. Decide, before the first generation, three things: the smallest difference that would change your decision, the confidence level, and the power. Then compute how many paired comparisons that requires, and go get that many.

The comparison is a paired win rate against 0.5. For a two-sided test at the 5 percent level, the required number of decisive pairs, computed from the normal approximation for a proportion against 0.5, is:

Read the first row and the last row together, because the whole discipline is in the ratio. Proving a clear win takes 20 comparisons. Proving a narrow one takes 783. Thirty-nine times the work to establish a difference one third the size. That ratio is why teams who "just try a few examples" always come back with a verdict and never with a measurement.

Four adjustments turn the table into a plan.

Ties inflate it. The table counts decisive pairs. If the grader calls 30 percent of your pairs a draw, you need 194 ÷ 0.70 = 278 total items to end up with 194 decisive ones. At a 40 percent tie rate, 324.

Extra candidates inflate it. The third column is the same test at α = 0.01, the Bonferroni correction for five simultaneous comparisons. Adding four more candidates moves the 0.60 requirement from 194 to 290. The fifth candidate is not free; it costs you 96 more comparisons on every arm.

Judge variance inflates it, and this is the big one. The table assumes sampling noise is the only noise. Apply the correction from the measurement-error study above and multiply by 2.8 to 6.3: the 194 becomes 539 to 1,213. If you take one number from this section, take that one. The floor is not the requirement.

Pairing deflates it, which is the only good news here. Because both systems answer the same input, the difference between them is far less variable than either score alone. If per-item scores correlate at ρ, the variance of the paired difference is proportional to (1 − ρ). At ρ = 0.5 the requirement halves; at ρ = 0.7 it falls to 30 percent. Pairing is the cheapest variance reduction available and it is free at the point of design. Never run the two arms on different inputs.

Now watch what those sizes do to the interval, because this is where the promotion gate gets decided. All four rows below observe the same 60 percent win rate, each interval computed from the Wilson formula above, and only the sample size changes:

Same point estimate in every row. Four completely different states of knowledge. A gate written on the point estimate passes all four identically, which is the precise sense in which a promotion gate on a point estimate is not a gate.

Stage 7 — Write the promotion gate before the run, in intervals

Build it. The gate is a document, agreed and dated before the first generation. Promote the candidate only if every one of these holds on the frozen set:

• The interval clears the line. The lower bound of the 95 percent Wilson interval on the candidate's win rate over the incumbent, computed on decisive pairs, exceeds 0.50. Not the point estimate. The lower bound.
• The grader is qualified. Kappa against human adjudication on the audit sample is at or above 0.6, construct sensitivity R is at or above 0.5 on the paired-edit probe, and the order-flip rate is at or below 0.10. If the grader fails, the run has no verdict, regardless of what the win rate says.
• The interval was computed honestly. If the win-rate interval was computed on sampling noise alone, without the judge-variance correction, the gate does not pass. Recompute or re-run.
• Nothing invariant broke. Any pass/fail suite you already run — safety, leakage, schema conformance, regression on public benchmarks as a guardrail rather than a claim — returns clean. A candidate can win on parity and still be blocked by an invariant, and it should be.
• The trade is stated. Cost per task and 95th-percentile latency are recorded for both arms, and the report says in one sentence what is being traded for what.

Two variants worth writing down now rather than arguing about later. To swap for a cheaper model, you are not claiming superiority, so the test is non-inferiority: require the lower bound to exceed 0.45 rather than 0.50, and say in the report that the bar was moved and why. To adopt something for a new workflow with no incumbent, there is no pairing available, so the gate reverts to an absolute threshold with its own interval, and it is a weaker claim. Label it as one.

The wrong conclusion this prevents: the candidate won, so we ship it. Without a written gate the rule is chosen after the numbers arrive, which means the rule is chosen to fit the numbers. Everyone does this in good faith and everyone gets the same answer, which is the one they were hoping for.

Stage 8 — Date the claim, and re-verify it

Build it. Every parity claim carries the date it was earned and the baseline it was earned against. Re-pin the baseline quarterly and re-run the gate on the frozen set. Refresh 20 percent of the frozen set each quarter from live traffic, re-freeze, and never train on it.

The wrong conclusion this prevents: we validated this. A parity result is a statement about two specific systems on one specific date. Incumbents get updated silently. Traffic drifts. A claim with no date on it is a claim that has quietly become an assertion, and it is the sentence that will be read back to you in an examination.

The artifact: the parity harness specification

Everything above, condensed into something you can hand to an engineering lead this week. Copy it, change the numbers where your context demands, keep the shape.

Inputs

• A workflow with real traffic you can sample.
• An incumbent system, running in the harness you ship.
• One to five candidate systems, named in advance.
• A judge model from a family unrelated to every arm.
• A human adjudicator for at least 10 percent of items.

Outputs

• One signed run report per candidate.
• One frozen blind set with a recorded hash.
• One promotion decision, with the interval that produced it.

The eight stages, with what each produces

1. Task capture → 400–1,000 stratified real items. Prevents: measuring someone else's distribution.
2. Freeze and decontaminate → a hashed blind set of 300+ (1,000+ for classifier metrics), 20 percent adversarial slices, split by source and time. Prevents: measuring memorization.
3. Pin the baseline → a model/version/date triple, plus the pre-declared candidate list. Prevents: beating a stale baseline, and hidden multiple comparisons.
4. Blind → paired outputs with identity stripped, history stripped, order randomized, 20 percent double-ordered. Prevents: grading the label instead of the answer.
5. Validate the judge → kappa, invariance, sensitivity, order-flip rate, all reported. Prevents: an assertion dressed as a measurement.
6. Size the run → a target n derived from the smallest difference that would change the decision, inflated for ties, candidates and judge variance. Prevents: an underpowered run reporting a verdict.
7. Gate → interval lower bound, grader qualification, invariants, cost and latency. Prevents: promoting on a point estimate.
8. Date and re-verify → quarterly re-pin, 20 percent set refresh. Prevents: an expired claim presented as current.

The blinding procedure, exactly

For each item i in the frozen set: generate answer A from the incumbent and answer B from the candidate. Apply the identity-strip transform to both. Draw a fair coin c(i); if heads, present (A, B), else present (B, A). Write c(i) to a mapping file the grading process cannot read. For the first 20 percent of items by index, additionally submit the reversed presentation as a separate judging call with an independent request identifier. Never include the item index, a run identifier, a timestamp, a revision number, or any prior score in the grader's context.

The judge rubric, exactly

You will see one task and two candidate responses, labeled Response 1 and Response 2. Decide which better completes the task. Score task success first. A response that completes the task poorly written beats a response that fails the task well written. Length is not a criterion. Do not reward a longer response for being longer. If both complete the task equivalently, answer TIE. Do not break a genuine tie. Answer in this form and nothing else: VERDICT: 1 | 2 | TIE, then REASON: one sentence naming the deciding difference.

The one-sentence reason is not decoration. It is what the human adjudicator reads on the audit sample, and it is how you discover that your grader has been deciding on formatting for a month.

Sample sizing, at 5 percent two-sided — computed, not cited

Then: divide by (1 − tie rate) for total items. Multiply by 2.8 to 6.3 for judge and prompt variance. Multiply by (1 − ρ) for the pairing gain. In practice, for a modest edge on a real workflow, budget 500 to 1,200 paired comparisons and be pleasantly surprised if you need fewer.

The promotion criterion, exactly

Promote if and only if all of the following hold on the frozen blind set: (a) the 95 percent Wilson lower bound on the candidate's win rate over the pinned incumbent, computed on decisive pairs, exceeds 0.50 — or 0.45 for a declared non-inferiority swap; (b) judge kappa ≥ 0.6 against human adjudication on a ≥ 10 percent audit sample, construct sensitivity R ≥ 0.5, order-flip rate ≤ 0.10; (c) the interval carries the judge-variance correction; (d) every invariant suite passes; (e) cost per task and p95 latency are recorded for both arms and the trade is stated. Otherwise: no promotion. A failed gate is a result, not a delay.

The run report, minimum fields

Candidate identifier and weights hash · pinned baseline as model, version, date · blind-set hash and item count · judge model, version, and the family-independence assertion · win rate with 95 percent interval · tie rate · order-flip rate · kappa with its own interval · invariance and sensitivity · per-slice breakdown including the adversarial slice · cost per task and p95 latency for both arms · verdict, and the gate clause that decided it.

No report, no claim. That is the whole governance layer, and it is one JSON file.

The seventy-two hours after this specification went to the gate

Three studies landed while this edition was in verification, and all three sit on the judge.

On 24 August, the creativity-evaluation study above found automated judges systematically preferring machine-written stories and automatic metrics showing near-zero correlation with human judgment. On 25 August, A Judge Should Know What Changed published the invariance-and-sensitivity profile and the finding that surface-only predictors reproduce 67.4 percent of the votes in the field's canonical judge-validation set. On 26 August, the anchoring study reported that metadata in the grader's context blocked 48 percent of error corrections across 185,271 evaluations.

A fourth, also on 25 August, moves in the other direction and deserves the same weight. RecurSE proposes a judge that improves itself against a synchronized checker auditing its reasoning, with a validity monitor deciding when to stop, and reports generalization gains across held-out medical, pairwise, summarization and professional benchmarks. Judges are not a fixed liability. They are an engineering problem with active work on it.

Read the artifact in front of you

There is a habit running through all five chapters this week, and it is the same habit in every one.

The AI Boardroom reads an examination list as it is actually written. Ship AI reads a permission model as it is actually written. Sovereign Stack reads a model card as it is actually written. Twin and Machine reads a certification scope as it is actually written. This chapter reads a benchmark as it is actually written — the reproduction instructions, the batch-size note, the maintainers' request that you stop comparing across papers.

In every case the document was public the whole time. In every case the team quoting it had read the headline number and not the page. Read the artifact in front of you is not a research method. It is the difference between a claim you can defend and a claim you inherited.

The limits, and what would falsify this

The sample-size arithmetic assumes independent items. If your captured tasks come in clusters — several requests from the same customer, several turns of one conversation — the effective sample is smaller than the item count, and the intervals above are too narrow. The correction is clustered standard errors, and the general treatment is in Adding Error Bars to Evals, which sets out how to analyze evaluation data as an experiment rather than a scoreboard. If your traffic is clustered and you use the raw table, you will overstate your confidence.

The 2.8-to-6.3 multiplier is one study's finding, generalized. It comes from a single measurement-error analysis and I have applied it as a planning rule across contexts it did not measure. It could be too aggressive for a narrow, deterministic task with a rule-based checker, where judge variance barely exists. It could be too gentle for a long-form open-ended task. The defensible version is the method, not the multiplier: measure your own judge variance by re-running the same pairs through the grader at temperature and across two prompt phrasings, and derive your own number.

Several load-bearing sources are preprints, two of them days old. The anchoring, construct-validity and creativity studies were posted within the last four days and none has completed peer review. The direction they point is mutually consistent and consistent with the older, larger work — but treat the specific decimals as provisional, which is the same courtesy this episode demands for any other number.

The clean falsifier. If a study with a frozen private set, an independent judge, and adequate power found that public leaderboard rank did predict workflow-level win rate well enough to make a purchase decision — that the ordering transferred — then most of this harness would be over-engineering, and the honest move would be to read the board and save the fortnight. Tuesday's chapter reported the opposite from a controlled harness swap, and the studies above explain the mechanism. But one clean transfer result would retire the argument, and it is better to name it than to pretend it could not exist.

And the one that cuts against how this gets used. A private harness measures your workflow, which means its result is not portable either. A parity claim earned on your traffic says nothing about anyone else's, and quoting it to a customer as a general model comparison would be the same category error this episode exists to correct, committed by someone who should know better. The number is decidable and local. Both halves matter.

The gap this closes, and the one it does not

Nobody is going to hand you the specification. The revised interagency model risk guidance — OCC Bulletin 2026-13, "Model Risk Management: Revised Guidance," 17 April 2026, with the parallel Federal Reserve designation SR 26-2 — expressly places generative and agentic AI outside its scope and says separate guidance will follow. Read that as a deferral rather than an exemption: the underlying obligations attached to the action did not move, and the framework that would have specified the controls did. Whatever gets written later will be written against whatever practice the market has already built.

Which means the parity report is not a compliance artifact you are behind on. It is the evidence you will be asked for, in a form nobody has yet standardized, and the institutions that have one when the question arrives will be the ones the standard is written from. The AI Boardroom chapter this week covers who is asking and what they have published. This chapter covers what you hand them.

Build the instrument. It is eight stages, a frozen file, a rubric, one table of sample sizes, and a rule you wrote down before you looked.

What to ask your team

1. When we compared models last quarter, what was the sample size, and what was the confidence interval on the difference? If nobody can answer the second half, the first half did not decide anything.
2. Do we have a frozen, private, decontaminated set of our own tasks — and has any vendor ever seen it?
3. Who or what graded our last model comparison, and what is that grader's measured agreement with a human on our work, chance-corrected?
4. Was the promotion rule written down before the results came in, or after?
5. How many candidates did we quietly try before the one we reported, and is that number written anywhere?

The series

This is Episode 2 of The Operator's Map — a weekly series in five chapters, advancing together: Ship AI teaches agent controls, Sovereign Stack the open-source stack, The AI Boardroom governance, Beyond the Benchmark evaluation, Twin and Machine physical AI. Next week, this chapter teaches how to read a failed parity run: what a wide interval, a split slice and a disqualified judge each mean, and the escalation ladder that is not simply a bigger model. Subscribe to follow the map as it fills in.

Cut in verification, and why

• A judge-protocol study on evidence locking (arXiv 2608.05353). It reports 24,000 judgments and clean effect sizes — agreement down 4 to 6 percentage points, answer-order inconsistency up 8 to 10 — and it would have made a tidy paragraph in stage 4. The paper is marked withdrawn by its author on account of a code error affecting the reported results. Cut. The withdrawal is the more useful lesson and it is not in the body: a number can be real, published, precise and wrong, and the only signal that it was wrong is a line on the abstract page that nobody reading a summary will ever see.
• The SWE-bench Verified annotation statistics. The human-validation process behind that 500-item subset is the canonical example of auditing a benchmark's own answer key, and the vendor write-up carries the fractions of underspecified problem statements and overly narrow tests. That write-up returned HTTP 403 to an automated client, and the dataset card states only that the subset is "500 samples … which have been human-validated for quality." Cut rather than cited to a secondary. The stage 2 paragraph rests on this chapter's Episode 1 audit finding instead.
• A measured relationship between public leaderboard rank and workflow win rate. No study was found that measures that correlation directly, with a frozen private set and adequate power. This episode therefore argues that the transfer is unestablished, and never that it has been measured and refuted. Cut the stronger phrasing wherever it appeared in draft.
• Naming the providers in the private-variant finding. Available in the cited study and deliberately not carried, under the standing rule that a firm is named only from its own disclosure. The mechanism is reproduced in full; the roster is not.
• A GCC or India instrument setting an evaluation-methodology requirement. Nothing was fetched in this pass that does so in either market, so nothing is claimed. The one regulatory sentence here is the US scope carve-out; the supervisory landscape is the AI Boardroom chapter's subject rather than this one's.
• Any claim that verbosity bias is large. It is the common folk belief, it appears in the 2023 source's own list of biases, and the largest measurement found in this pass puts it below 0.011. Cut, and the finding reported against the direction the story would have preferred.