The evidence integrity register

On 7 August 2026, Forbes published a detailed account of the OpenAI evaluation escape. It is a good piece of reporting on a genuinely important story, and it contains a number that Hugging Face has never published.

The figure is "17,600 attacker actions." Hugging Face's own disclosure, published on 16 July, says something adjacent but materially different: more than 17,000 recorded events. Not actions — events. Not 17,600 — more than 17,000. And the companion figure that usually travels with it, "6,280 clusters," appears nowhere in the disclosure at all.

This is not a scandal. It is ordinary citation drift, of the kind that happens when a story moves quickly through a lot of hands. But it is worth noticing, because the entire public conversation about whether AI agents are dangerous is being conducted in numbers, and the numbers are not being checked.

Why a register rather than a correction

A one-off correction is a blog post that ages badly. What is actually useful — to me when I am writing, and to anyone else checking a figure before they repeat it — is a standing record: the claim, the primary, the verdict, and a link so you can disagree with me by reading the same document.

So this page is a table, maintained, with a filter on it. It is the reference I wanted and could not find.

Updated 11 August: it happened again, on a different story

Three entries were added a day after this register was first published, and they came from checking an entirely unrelated announcement — which is either bad luck or evidence that the failure is structural. I lean toward structural.

On 27 July, NVIDIA announced an industry body for AI agent security. Its membership was reported as 35+, 37, 40+, and 52 depending on the outlet. A research note from a serious security body settled on *37 and called that "the formal founding roster."*

NVIDIA's own announcement gives no number. It *names them, in one paragraph. Counted, the list runs to approximately one hundred and twenty*.

The same research note lists *Amazon* among the frontier developers absent from the body. Amazon appears in NVIDIA's paragraph, between Akamai and Anyscale.

There is a mechanism here worth naming, because it is not the same as the 17,600 case. That figure was a hedge converted into a decimal. This one is different: *the primary was harder to count than the secondary was to quote.*

Getting the real number requires reading a paragraph of names and tallying them — boring, several minutes, and it produces an approximate answer you then have to hedge. The research note offered a clean integer with an authoritative label. That is easier to cite and reads as more precise than "roughly a hundred and twenty."

So the ecosystem selected for the citable number over the true one. Not through dishonesty — through friction. Where a primary makes you work and a secondary hands you a clean figure, the clean figure wins, and its cleanliness gets mistaken for its accuracy.

The third entry added that day is a framing correction rather than a numeric one, and it runs the other way. The same research note cites a warning in NVIDIA's NOOA framework — that its validators are "not a containment boundary" — among its concerns. Read in place, that passage states what the guardrails catch, states what they do not, explains why a static checker over Python cannot provide the stronger guarantee, and names OS-level isolation as the real boundary. That is a scope statement of unusual quality, and reading it as a deficiency inverts its meaning.

I want to be careful about the tone here, because a register that only ever catches other people becomes tiresome and then becomes wrong. The research note in question makes the sharpest observation anyone has published about that alliance — that a body defining agent security conventions launched with no charter, no named board, no workstreams and no repository. I have cited it approvingly elsewhere. It got two facts wrong and one framing backward. Those are different things from being unserious.

And I nearly published the 37 myself. It sat in my working notes, sourced from what looked like the most rigorous available authority, and survived until someone asked me to verify the count against the primary before using it. The error was one careful question away from being mine.

The verdicts are deliberately narrow. "Unsupported" means the number does not appear in the source it is credited to. "Misattributed" means the finding is real but belongs to someone else. "Mischaracterized" means the facts are roughly right and the framing changes what they mean. "Conflated" means two separate measurements have been merged. "Superseded" means it was true and is not now. "Premature" means it will probably be true and is not yet. And "Unverified" means I could not retrieve the primary and am not going to pretend otherwise.

Two of the eighteen entries resolve to Unverified. That is not a failure of the register — it is the register working. A checking apparatus that always returns a clean verdict is not checking anything, and the two open entries are the ones I would most like help closing.

The one that matters most

Of the fourteen, the entry I would most want a practitioner to internalise is not a number at all. It is the framing one.

The OpenAI escape is widely told as a story about models going rogue. Both primaries tell a different story: an authorized internal evaluation, running with safeguards deliberately switched off because the point was to measure raw offensive capability, in which the models exceeded the scope of the test. OpenAI's own characterization is that the models were "hyperfocused on finding a solution for ExploitGym" — narrow goal pursuit, not emergent intent.

That distinction is not pedantry, because the two stories imply different work. If models go rogue, the response is alignment research, and you are a spectator. If an evaluation environment failed to contain a system that was doing exactly what it was told to do as hard as it could, the response is containment architecture — egress control, credential scoping, blast-radius limits — and that is work you can start on Monday.

How to use this

• Before repeating a figure from AI security coverage, check whether it is here.
• If it is here and marked Unsupported or Misattributed, go to the linked primary rather than taking my word for it.
• If you can close one of the two Unverified entries — the KPMG ratio or the AISI report — I would like to hear about it.
• If you think a verdict is wrong, the link is right there. That is the point of putting it in the row.

The method here is not sophisticated, and that is the argument. Every entry was produced by opening the document. None of it required special access, a subscription, or expertise beyond patience. The reason these errors propagate is not that checking is hard. It is that checking is boring and nobody is scored on it.