The leaderboard measures the model to a decimal. It does not measure whether the agent is safe.

Start with the two numbers, because the argument lives in the gap between them and every attempt to wave the gap away depends on not looking at it.

OpenART is a red-team benchmark released this month by a group at Fudan University and the Shanghai AI Laboratory. It assembles more than ten thousand validated stateful attack scenarios across fifty domains, and it drives them at fifteen real agent harnesses paired with five frontier models. The headline figure is a pooled attack success rate of 85 percent. Read past the headline to the per-model table and the figure changes character. Against the same harness set, Qwen-3.7-Max is attacked successfully 94.6 percent of the time, DeepSeek-V4-Pro 94.7, GPT-5.5 88.5, GLM-5.2 87.9, and Claude Opus 4.8 59.2. The distance between the top of that column and the bottom is thirty-five and a half points.

Hold that next to how these same models are ranked everywhere else. On the public capability leaderboards, the five models above are separated by low single digits on most tasks, and their ordering shuffles from benchmark to benchmark. The capability numbers are reported to one decimal place and defended to that decimal in launch posts. And yet the property that actually determines whether one of these models is safe to put behind a payments endpoint — how often an adversary can turn the agent against its own task — varies across the panel by more than thirty-five points, on harnesses that did not change. The leaderboard measures one thing precisely. The thing that gets you breached, it does not measure at all.

That is the category error this piece is about, and it is worth naming at the root before anyone reaches for the objections. Safety is not a property of the model. It is a property of the deployment — the model, the harness it runs inside, and the environment state it acts against, taken together. The leaderboard attaches a scalar to the model and invites you to read it as a property of the model. For capability, that read is merely lossy. For safety, it is a different kind of number wearing the same clothes.

THE OPERATOR'S MAP · Chapter: Beyond the Benchmark · Episode 1. This article is the first episode of a weekly series for the people who have to run AI rather than admire it — five chapters, one per domain, advancing together each week: agent controls (Ship AI), the open-source stack (Sovereign Stack), governance (The AI Boardroom), evaluation (Beyond the Benchmark), physical AI (Twin & Machine). This chapter teaches evaluation. Every technical idea is restated in plain terms as it arrives.

Terms that matter this episode

• Benchmark / leaderboard — a standardized test for AI models, and the public ranking built from it.
• Harness / scaffolding — the software wrapped around a model that turns it into a working agent: tools, memory, retry logic. The same model in two harnesses is two different systems.
• Attack success rate — in security testing, the share of attempts that got the system to do the attacker's bidding.
• Stateful attack — an attack that manipulates the system's environment over time, rather than sending one malicious message.
• Point estimate vs distribution — one number, versus the range of outcomes across conditions. Safety only exists as the second.
• Judge — the automated grader scoring benchmark answers; its own error rate is part of any result it produces.

What the leaderboard actually measures

A capability benchmark reports, for each model, a single number: the fraction of a fixed task set the model completed under the benchmark's own harness. SWE-bench Verified, Terminal-Bench, GPQA Diamond, the agentic suites — all of them produce one decimal per model, and the industry has organized itself around comparing those decimals. Procurement decks quote them. Launch posts lead with them. A model that moves a benchmark two points is described as a generational step.

The unstated assumption underneath all of this is that the number transfers — that a model which scores 92 on a benchmark harness will behave like a 92 when you drop it into your own stack. A second August 2026 preprint, from an anonymous author group, measures exactly that assumption and finds it false in the direction that matters. The paper, StateM, wraps a frozen model in a YAML-runbook state machine and reports the score move from the harness alone. GPT-5.5 goes from 83.1 to 92.1 percent on Terminal-Bench 2.1 with the model held constant and only the scaffolding changed — a nine-point lift attributable to nothing the leaderboard would credit to the model. The authors state the consequence plainly: the model is not the main bottleneck. The same frozen runbook transferred to a newer model reaches 95.3 percent, and the authors are careful to flag that the 95.3 is a pre-adjudication figure, which is the kind of disclosure that separates a serious benchmark from a marketing number.

Put StateM and OpenART side by side and they are the same finding pointed at two different outputs. StateM says: freeze the model, change the harness, and capability moves nine points. OpenART says: freeze the harness set, change the model, and safety moves thirty-five. Neither number is a property of the model in isolation, and neither transfers across the boundary the leaderboard pretends is not there. The leaderboard's single decimal is a measurement of one particular pairing of a model with one particular harness, reported as though it were a measurement of the model.

There is a third preprint from this month that shows the same joint underneath capability. A study of agent skills across more than eight thousand trials finds that skills work as procedural anchors — they account for 65.7 percent of the success they produce by stabilizing execution, and only 4.5 percent by injecting knowledge. The same study finds that as the skill pool grows from five entries to a hundred, retrieval precision collapses from 29.6 percent to 3.3 percent. That collapse is the point. The scaffolding that determines whether the agent succeeds does not degrade gracefully as the environment gets larger; it falls off a cliff. A benchmark run against a five-skill pool tells you nothing about the hundred-skill deployment, because the mechanism that carries the score does not survive the scale-up.

The environment is the attack surface, and it has a curve

OpenART's real contribution is not the 85 percent. It is that the attacks are stateful. The benchmark does not throw single-turn prompt injections at a model and count refusals. It constructs environments with history and state, and it attacks the state. Its own construction reflects this: the attacks are generated by an evolutionary search over a Markov hypergraph of the environment, which is a formal way of saying the method looks for the sequence of environment manipulations that turns the agent, rather than the one sentence that jailbreaks it.

That distinction produces the number in the paper that matters more than the headline. As environment complexity rises, the attacker's advantage grows from roughly 2 percent to more than 17 percent. The safety of the agent is not a constant that you can measure once and quote. It is a function of how stateful the environment is, and it degrades as the environment gets closer to a real one. A benchmark run against a simple environment reports the flattering end of that curve. Production is the other end.

This is why the leaderboard's format is not merely incomplete for safety — it is structurally wrong for it. A scalar can represent capability tolerably because capability, however harness-dependent, is at least a single achievement you either reach or do not. Safety is a distribution over adversarial conditions, with a slope that steepens as conditions get realistic. You cannot compress that into a decimal without discarding the slope, and the slope is the part that predicts what happens to you. The agent-skills collapse from 29.6 to 3.3 percent and OpenART's climb from 2 to 17 percent are the same shape seen twice: the interesting behavior of an agent stack is what it does as the environment scales, and the leaderboard measures a point, not a scale.

Before trusting any ruler, audit the ruler

Here is the objection at full strength, and it is the strongest one. The capability leaderboard is a mature, adversarially-tested instrument. SWE-bench Verified was human-filtered precisely to remove bad instances. Terminal-Bench is run at hundreds of trials. These are not naive metrics; they are the result of years of the field sharpening its rulers. Perhaps safety simply needs the same maturation, and OpenART is an early, self-constructed benchmark that will be superseded by better ones. The gap is real, but it is a gap in the state of the art, not a category error.

That objection deserves a concession and then a correction. The concession: OpenART is a self-constructed benchmark. The same group built the attack method and the scoring, the model panel is self-selected, and the judging has a large language model in the loop. The authors mitigate the last of these with a dual judge — a deterministic rule check alongside the model judge — and a 10 percent human audit that reports 99.3 percent agreement, which is the right way to do it, but the number is still the benchmark's own construction and should be read as such. Take that concession at full weight.

Now the correction, and it is the reason the "let it mature" reading is too comfortable. The capability leaderboard the objection holds up as mature was itself, this month, shown to be measuring the wrong thing. A fourth August 2026 preprint, SWE-Bench ProMax, was motivated by an audit of SWE-bench Verified finding that roughly 60 percent of its unsolved instances rest on flawed tests — 35.5 percent whose tests are too narrow to catch a wrong answer, and 18.8 percent whose tests are too broad to accept a right one. The most trusted agentic capability benchmark in the field, the one that gets quoted to a decimal, had a majority of its hard instances resting on tests that could not distinguish a solution from a non-solution. The retirement of a benchmark is the admission that the ruler was miscalibrated. If the field's best-audited capability ruler was measuring the wrong thing until someone audited it this month, the confident position on the safety ruler is not "it will mature into something trustworthy." It is "audit it before you quote it, and notice that on safety there is not yet a ruler to audit."

That is the discipline this lane exists to enforce. Capability is measured to a decimal, and even the decimal turned out to need auditing. Assurance is not measured at all, and the market has noticed the vacancy even where the leaderboard has not. The observability and evaluation vendors are consolidating around exactly the measurement the leaderboard skips — the reported roughly 915-million-dollar move to absorb an agent-evaluation platform into a monitoring one is the visible edge of a wider consolidation, and what is being bought is the ability to measure the deployment, not the model. The vendors are building the receipt. The leaderboard is still selling the sticker price.

What an assurance benchmark would have to measure

Naming the gap is cheap. The useful thing is to state what an instrument would have to do to close it, precisely enough that you could tell whether a given benchmark qualifies. An assurance benchmark has to have at least four properties, and the fourth is the one that the current leaderboard format cannot be patched into having.

It measures the deployment, not the model. The unit under test is the triple of model, harness, and environment, re-run on the harness you actually ship. OpenART's whole design is an argument for this: it is the pairing of fifteen harnesses with five models, not five models in a vacuum, and the thirty-five-point spread only exists because the harness is held in the frame. A safety number produced against the benchmark's own harness and quoted against your deployment is a StateM violation — it is the nine points that do not transfer, in the safety direction, where the non-transfer gets you breached rather than merely disappoints you.

It reports a distribution and a degradation curve, not a point estimate. The number that predicts production is not the pooled attack success rate. It is the slope of attacker advantage against environment complexity — OpenART's 2 percent climbing to 17 — because that slope tells you where your real, stateful environment sits on the curve. An assurance benchmark that reports one number has thrown away the only part of the measurement that extrapolates. The capability leaderboard's scalar format is not a neutral container here; it actively cannot hold the thing worth knowing.

Its judge is named, and the judge's own error rate is part of the result. A safety number produced by an unaudited model judge is an assertion dressed as a measurement, which is exactly the failure this lane refuses everywhere else. OpenART's dual rule-and-model judge with a 10 percent human audit reporting 99.3 percent agreement is the minimum bar, not a nicety: the accuracy of the instrument that produced the number is itself a reported figure, so a reader can discount the result by the instrument's own uncertainty. A capability leaderboard almost never reports the reliability of its own grader. An assurance benchmark cannot get away with that, because the stakes of a mis-scored safety result are asymmetric — a false pass ships an exploitable agent.

It is adversarial and stateful by construction. It attacks the environment state, because that is where agents actually break, and not the single-turn prompt, because that is where they were already hardened. This is the property that no reweighting of the existing leaderboard produces. You cannot arrive at OpenART's finding by running the capability suite more carefully. The capability suite asks whether the agent can do the task. The assurance benchmark asks whether an adversary who controls part of the environment can make the agent do something else, and then asks how that answer degrades as the environment gets more real. Those are different questions, and only the second one is the one a regulated buyer is actually exposed to.

What I learned building the harness myself

I run a blind parity-eval harness for my own model work — the tooling I use to decide whether an open-weight specialist is good enough to stand in for a pinned frontier model on a specific workflow, scored blind against that baseline so I cannot flatter the candidate I want to win. The whole apparatus exists because the public leaderboard numbers were not decidable for my question. I could not read a model's benchmark score and know whether it would hold up in the one harness and the one workflow I was going to ship.

The lesson that harness taught me, before I had a benchmark to cite for it, is the lesson OpenART and StateM now make quotable. The first time I fixed a workflow, froze the harness, and varied only the model, the ranking I got back did not match the public leaderboard order — not by a little, and not consistently in either direction. A model that led the public board trailed in my harness on my workflow, and one that the board placed mid-pack led. I assumed, at first, that my harness was wrong. It was not wrong. The leaderboard number and my number were measurements of different things: the leaderboard had measured that model in its harness, and I had measured it in mine, and there was no law that made those two numbers agree. Once you have watched a ranking scramble under a controlled harness swap, you stop believing the decimal is a property of the model. You start asking which harness produced it, and you notice that the safety version of the question has no published answer for your deployment at all.

The limits, and what would falsify this

Four things could be wrong here, and they are worth stating at their strongest.

The entire argument rests on preprints, and one of the load-bearing ones is self-constructed. OpenART built its own attack method, its own scenarios, and its own scoring, and selected its own model panel. The 85 percent and the thirty-five-point spread are the benchmark's own construction, mitigated by a dual judge and a human audit but not independent of the people who wanted a striking result. StateM, the agent-skills study, and the SWE-Bench ProMax audit are all likewise preprints, none yet through peer review. The correct posture is that these are strong, mutually corroborating signals pointing the same way, not settled facts. The argument would be stronger with an independent replication, and it does not have one yet.

The judge could carry a systematic bias. A model-in-the-loop judge might lean toward scoring a model that refuses in a particular verbose style as safe, and a model that complies tersely as breached. The 59.2 at the bottom of the OpenART column could be, in part, an artifact of a judge that rewards a refusal style rather than a refusal. The 10 percent human audit at 99.3 percent agreement bounds this but does not eliminate it, because the human auditors and the model judge may share the same blind spot. I would want to see the spread reproduced under a judge built by a different group before treating the exact magnitude as load-bearing. The direction of the finding survives this caveat; the precise thirty-five and a half points may not.

The clean falsifier is worth being specific about, because a vague falsifier is not one. If an independent benchmark — not self-constructed, built by a group with no stake in the result — ran the same frontier models across a wide set of real production harnesses and found that attack success rate was nearly invariant to the harness, then safety would be a property of the model after all, a scalar would be a legitimate way to report it, and the leaderboard's format would be defensible for safety too. I have not seen such a result. OpenART's thirty-five-point same-harness spread and StateM's nine-point same-model lift are the current evidence against it. But the argument is falsifiable in one clean move, and I would rather name the move than pretend it does not exist.

The fourth cuts against how this argument gets used commercially. Nothing above shows that any of the named models is unsafe to deploy, or that a low ASR on one benchmark makes a model safe on your stack. The whole point is that the number does not transfer — which cuts both ways. A buyer who reads Claude Opus 4.8's 59.2 as a safety guarantee for their own deployment has made the identical category error as the buyer who reads a capability decimal as a competence guarantee. The finding is not "pick the model with the best ASR." The finding is that the deployment is the unit that has to be measured, and the model's benchmark number, high or low, is not a substitute for measuring it.

The receipt is being built somewhere other than the leaderboard

The leaderboard is not going away, and it should not. Capability has to be measured, and measuring it to a decimal is a real achievement that took the field years — even if the decimal, as ProMax showed this month, still needs its own audit. The mistake is not measuring capability. The mistake is letting the capability number stand in for a safety number it was never measuring, and mistaking the precision of the one for the presence of the other.

The rest of the industry is already routing around the gap. Red-team benchmark authors are building stateful attack suites because they concluded the environment is the attack surface. Observability vendors are consolidating around deployment measurement because they concluded the model number is not the receipt. The frontier labs themselves are pricing containment regimes and publishing the controls, because they concluded that what an agent did, under which authority, in which environment, is the thing that has to be provable after the fact. All of them are building the receipt — the evidence of what the deployment actually did — because the leaderboard was only ever selling the sticker price.

If you are standing agents up in a regulated shop, the operational verdict is narrow and it is not about which model to pick. It is that your assurance number does not exist yet, that no vendor's benchmark decimal is it, and that the only number that will survive an examiner's question is one you generated against your own harness, against a stateful adversary, with a judge whose error rate you can quote. That number is not on any leaderboard. It has to be built, on the deployment, by you — which is the conversation worth having before the agent is behind the endpoint rather than after.

What to ask your team

Questions to carry into your next AI review. None requires a technical background; all of them have answers your team either holds or does not.

1. Which model comparisons currently circulating in our decks are leaderboard scores — and who checked whether they were measured on anything resembling our stack?
2. Have we ever evaluated a model inside the actual harness we ship, against the incumbent, scored blind?
3. What is our safety number — attack success rate against our own deployment — and if none exists, what is standing in for it?
4. When a vendor quotes us an evaluation figure, do we ask who the judge was and what its error rate is?
5. As our agent's tool and skill count grows, has anyone re-measured at the new scale — or are we quoting the five-tool test at a hundred tools?

The series

This is Episode 1 of The Operator's Map. Next week, this chapter teaches building a parity harness — measuring a model against your own workflow, blind. The other four chapters advance the same day — the hub at /series holds the map.