Nothing outside this line was tested

THE OPERATOR'S MAP · Chapter: Twin & Machine · Episode 2 · 28 August 2026. The five chapters advance together each week — agent controls (Ship AI), the open-source stack (Sovereign Stack), governance (The AI Boardroom), evaluation (Beyond the Benchmark), physical AI (Twin & Machine). This chapter's Episode 1, and every other chapter's Episode 2, are linked at the foot of the piece.

The Operator's Map is a weekly series for the people who have to run AI rather than admire it — five chapters, one per domain, all advancing together each week. This chapter teaches physical AI. On Tuesday: why you cannot roll back a robot, and why a fleet push is not a software deploy. Today: what a digital twin actually certifies, and what the sim score cannot say. Every technical idea gets restated in plain terms as we go.

Why this reaches your desk. Somebody is going to hand you a number from a simulation and ask you to sign something. The number will be high and the claim attached to it will be broad, and the two will have almost nothing to do with each other. There is exactly one industry that has made simulated evidence carry legal weight for decades, and the way it did that was not by raising the score. It was by writing down, on the certificate, everything the simulation does not cover.

Terms that matter this episode

Start at the edge

In the middle of a United States federal regulation, in a list of definitions, there is a description of a digital twin that is knowingly wrong.

It is called a Class III airport model. It is a visual model of a real place, loaded into a certified flight simulator, used to train and test airline pilots. Here is what the regulation says about it, in Appendix F to 14 CFR Part 60, word for word:

"This is a special class of airport model… and includes models that may be incomplete or inaccurate when viewed without restriction, but when appropriate limits are applied (e.g., 'valid for use only in visibility conditions less than 1/2 statute mile or RVR2400 feet,' 'valid for use only for approaches to Runway 22L and 22R'), those features that may be incomplete or inaccurate may not be able to be recognized as such by the crewmember being trained, tested, or checked."

Read that again with an operator's eye. The regulator is not claiming the model is right. The regulator is saying the model is defective, naming the exact conditions under which the defect stops mattering, and requiring a person to accept that fence in writing before anyone may train against it. The same appendix defines a Generic Airport Model as a model that "combines correct navigation aids for a real world airport with a visual model that does not depict that same airport." Correct data, wrong picture, and the regulation has a name for it.

That is where this episode starts, because that is where a twin's evidentiary power actually lives. Not in the fidelity. In the fence.

The exclusions are printed on the certificate

Work inward from that edge and you find that the entire instrument is built out of boundaries.

14 CFR Part 60 governs the initial and continuing qualification of flight simulation training devices in the United States. It has been in force since 2006, was last amended in November 2024, and title 14 of the Code of Federal Regulations was current as of 25 August 2026 when this was written — a date anyone can re-check at the eCFR currency record. It is long, unglamorous, and it is the most complete public answer in existence to the question what does a simulation certify.

When a device passes its evaluation, the FAA issues a Statement of Qualification. Section 60.15(g) says what goes on it. Among the items is this, verbatim:

"A statement that (with the exception of the noted exclusions for which the FSTD has not been subjectively tested by the sponsor or the responsible Flight Standards office and for which qualification is not sought) the qualification of the FSTD includes the tasks set out in the applicable QPS appendix relevant to the qualification level of the FSTD."

The certificate carries its own exclusions. Not in an appendix, not in a footnote a lawyer wrote afterward — on the face of the document, as part of what qualification means. Section 60.17(b) names the two attachments that travel with it: a Configuration List and a List of Qualified Tasks. The certificate is a list of things, bounded by a list of things it is not.

Then the regulation does something that people building AI twins should sit with. In Appendix A, it says plainly that not everything on the list was exercised:

"It is not required that all of the tasks that appear on the List of Qualified Tasks (part of the SOQ) be accomplished during the initial or continuing qualification evaluation."

So even inside the covered region, the covered region and the tested region are different sets, and the regulation says so out loud rather than letting anyone assume otherwise. That single sentence is worth more than most vendor validation decks, because it tells you the honest shape of the claim: this is what we stand behind, and here is how much of it we actually ran.

And the scope cannot be widened by using the device more. Section 60.16(a): a currently qualified device "is required to undergo an additional qualification process if a user intends to use the FSTD for meeting training, evaluation, or flight experience requirements of this chapter beyond the qualification issued for that FSTD." Familiarity does not extend a certificate. Neither does a good track record. Only a new evaluation does.

Certifying the twin is not authorizing the claim

Here is the distinction that collapses most often in AI conversations, and the regulation keeps two separate words for it.

Appendix F defines Qualification Level as "the categorization of an FSTD established by the NSPM based on the FSTDs demonstrated technical and operational capabilities." It separately defines FSTD Approval as "the extent to which an FSTD may be used by a certificate holder as authorized by the FAA."

Two records, two decisions, two authorities. One says what the device is. The other says what it may be used for. A simulator can be fully, currently qualified at the highest level and still not be authorized for a particular credit, because authorization is granted somewhere else, by someone else, against a different rule.

You can watch that second decision being made, task by task, in 14 CFR § 61.156, which governs the training a pilot must complete before taking the airline transport pilot knowledge test. It requires at least ten hours in a qualified simulation training device, and then it splits those hours:

Nothing in that rule says "simulators are acceptable." It says upset recovery needs a Level C or higher device and flight-management-system work does not. Evidentiary power is assigned per task, against a named fidelity level, in a rule published before anyone bought the machine. And when the boundary needs to move, paragraph (c) says how: "The Administrator may issue deviation authority… upon a determination that the objectives of the training can be met in an alternative device." A named person, making a determination, on the record.

That is the shape the AI industry is missing. Not the tolerances, not the hardware — the practice of assigning a simulation's evidentiary weight one decision at a time, in advance, with a named route for changing it.

What the score is a score of

Now the question the chapter promised: what can the sim score actually say?

Appendix F again, and this definition deserves to be read twice. "Objective Test—a quantitative measurement and evaluation of FSTD performance." Of the device's performance. Not of the aircraft's. The number produced by the test is a statement about the simulator.

What converts that statement into evidence about the world is the thing on the other side of the comparison. Section 60.13(a) is explicit: the validation data package "must include the aircraft manufacturer's flight test data and all relevant data developed after the type certificate was issued." Appendix F defines flight test data as "aircraft data collected by the aircraft manufacturer or other acceptable data supplier during an aircraft flight test program." Somebody flew the actual airplane and wrote down what it did.

The test, then, is a distance. The simulator's output is compared against that measurement, parameter by parameter, inside published tolerance bands. From the objective test tables: airspeed within ±3 knots, altitude within ±20 feet, pitch attitude within ±1.5 degrees, heading within ±2 degrees, vertical velocity within ±100 feet per minute or ten percent. Those numbers only mean something because of what sits on the other side of them.

The regulation's own confession about simulation checking simulation

This is the passage that should be printed and pinned above every desk where twin claims get evaluated.

Aircraft manufacturers sometimes want to validate a training simulator against another simulation rather than against fresh flight test data, for reasons the regulation states without embarrassment: "Flight-test data are often not available due to technical reasons," "Alternative technical solutions are being advanced," and "High costs." Fair. So Appendix A, Attachment 2 permits it, and then it does two remarkable things.

First, it explains why the comparison is weak. Verbatim: "Engineering simulator data are acceptable because the same simulation models used to produce the reference data are also used to test the flight training simulator (i.e., the two sets of results should be 'essentially' similar)." And it lists exactly what the two sides can still differ by: hardware, iteration rates, execution order, integration methods, processor architecture, and digital drift including interpolation methods, data handling differences and auto-test trim tolerances.

Read that list. Every item on it is a property of the implementation. Not one of them is a property of the airplane. The regulator is saying, on the page, that when both sides of a comparison come from the same models, the test measures software engineering, not agreement with the world.

Second — and this is the part that inverts most people's intuition — the same attachment makes the tolerance tighter:

"If engineering simulator data or other non-flight-test data are used as an allowable form of reference validation data for the objective tests… the data provider must supply a well-documented mathematical model and testing procedure that enables a replication of the engineering simulation results within 40% of the corresponding flight test tolerances."

Forty percent of the band. A stricter pass mark precisely because the test proves less. In the AI world the instinct runs the other way: a result obtained purely in simulation is usually reported with wider error bars and softer language, as though acknowledging the weakness were the same as pricing it. Part 60 prices it.

Five words that are not synonyms

Episode 1 of this chapter shipped three claims that a later verification pass had to cut or correct. Two of the three failed for the same reason: five different kinds of claim were being described with one vocabulary. Part 60 gives us the vocabulary to separate them.

Those are five different claims. A demonstration video is none of them. A press release describing a simulated facility is the first one wearing the clothes of the fifth. The discipline is not skepticism, it is bookkeeping: whenever a twin claim arrives, write next to it which of the five it is, and if the answer is not obvious from the document itself, that ambiguity is the finding.

Part 60 even legislates the crossing between the first two. An "audited" engineering simulator may supplement flight test data — but only when the baseline simulation was "fully flight-test validated," only for "changes that are incremental in nature," only where the supplier can "demonstrate that the predicted changes in aircraft performance are based on acceptable aeronautical principles with proven success history and valid outcomes," and only with "comparisons of predicted and flight test validated data." Simulation may extend physical evidence. It may not originate it.

The grounds record already exists, and it has a column for what is missing

Episode 1 of this chapter ended on a field that almost everyone drops: what evidenced the fidelity of the world the policy was validated in? It was presented as something that had to be invented.

It did not. It has a name, a required format, and years of use. It is called a Validation Data Roadmap, and Appendix A describes it like this:

"The VDR should identify (in matrix format) sources of data for all required tests. It should also provide guidance regarding the validity of these data for a specific engine type, thrust rating configuration, and the revision levels of all avionics affecting airplane handling qualities and performance. The VDR should include rationale or explanation in cases where data or parameters are missing, engineering simulation data are to be used, flight test methods require explanation, or there is any deviation from data requirements."

A matrix, one row per test, saying where the evidence came from — and required to explain the empty cells. Not to hide them, not to average over them. To explain them.

The same regulation carries the idea into the test guide itself, which must record for every objective test: the tolerances, the source of the validation data down to document and page number, a copy of the validation data, and the simulator's own results. And it carries the idea into a second document, the Statement of Compliance and Capability, defined as "a declaration that a specific requirement has been met and explaining how the requirement was met… including references to sources of information for showing compliance, rationale to explain how the referenced material is used, mathematical equations and parameter values used, and conclusions reached."

Then it does the thing that turns paperwork into governance. Appendix F defines a Discrepancy — a defect, formally logged, repaired on a clock — to include "errors in the documentation used to support the FSTD (e.g., MQTG errors, information missing from the MQTG, or required statements from appropriately qualified personnel)."

Missing documentation is a defect. Not a nice-to-have, not a maturity level. A logged defect on the same list as a broken instrument.

What a twin structurally cannot say

Everything above is what a simulation can discharge. Here is the other side, and each item is a structural limit rather than a quality problem, which means no amount of additional fidelity fixes it.

It cannot validate itself. The output of a model is a fact about the model. Validation requires a second source that was not produced by the model, and the regulation's answer is a flight test program. Where no such source exists, no score can be computed that means anything about the world — only agreement between two computations, which Part 60 explicitly prices at 40 percent of the real band.

It cannot cover what was not modeled. This is what the Class III restriction is for. A model has boundaries by construction, and a system outside those boundaries does not stop; it keeps producing outputs that look exactly like the ones inside. NASA-STD-7009B, the agency's standard for models and simulations, requires "a record of the M&S limits (e.g., boundary conditions)" with the rationale that they "provide the broadest bounds for potential M&S Use that produce results, beyond which the M&S does (or will) not function (correctly)." Boundaries have to be written down because they are invisible from the inside.

It cannot extend its own scope. Section 60.16(a) again, and its companions: modifications require notification and possibly re-evaluation; and under § 60.27, qualification is automatically lost when the device "is physically moved from one location and installed in a different location, regardless of distance." Move the twin and the certificate lapses by operation of the rule. Nobody has to notice. Nobody has to decide. That is what a scope statement looks like when it has teeth.

It cannot report its own absence of evidence. NASA-STD-7009B requires that when results go to decision makers, explicit warnings accompany them for unachieved acceptance criteria, violated assumptions, violated model limits, execution warnings, unfavorable use assessments and outstanding defects. And then the sentence that no dashboard in the AI industry implements: "In the absence of substantiating evidence (e.g., data, information, records)… a warning is to be provided." Missing evidence generates a warning. A high score with nothing behind it is required to announce itself.

That same standard also declines to set a passing mark: "This Standard levies no requirements with respect to what levels to achieve (the sufficiency threshold levels), merely that the levels be determined and reported." The obligation is disclosure, not attainment. The reader decides what is enough.

The scope line that should worry anyone selling an AI twin

Which brings us to the boundary that matters most for the machines this chapter is actually about.

The Food and Drug Administration issued a final guidance on assessing the credibility of computational modeling and simulation in medical device submissions. The notice of availability was published at 88 FR 80314 on 17 November 2023, under docket FDA-2021-D-0980. It is the most developed public framework in United States regulation for deciding whether a simulation counts as evidence in a submission, and it points at the American Society of Mechanical Engineers V&V 40 standard for the underlying method.

Here is its scope sentence, verbatim from the notice as published:

"For the purposes of this guidance, CM&S refers to first principles-based (e.g., physics-based or mechanistic) computational models, and not statistical or data-driven (e.g., machine learning or artificial intelligence-based) models."

The framework for judging simulated evidence stops at the machine-learned model.

Hold that next to the other carve-out this series has already established. In OCC Bulletin 2026-13, "Model Risk Management: Revised Guidance," issued 17 April 2026 with the parallel Federal Reserve designation SR 26-2, generative and agentic AI are expressly outside the scope of the revised guidance.

Two United States regulators, two entirely different sectors, the same line drawn in the same place: the mature apparatus for judging model evidence does not reach the machine-learned model. Read both as deferrals rather than exemptions — the obligations attached to the underlying decision are untouched, and what has been withheld is the framework that would have specified the controls. Which means the specification, when it arrives, will be written against whatever practice the industry has already built.

So the useful move is not to wait. It is to take the four things the mature frameworks require — a scope statement, a named comparator, a stated applicability domain, and a record of the gaps — and apply them by hand to the claim in front of you. None of the four requires a regulator's permission. All four are free.

The worked artifact: a twin certification-scope matrix

This is the thing to take to work. One page, six columns, six rows, and it works on any twin or simulation claim regardless of domain. Column one of the working copy holds the claim as made to you, copied verbatim from the deck, the page or the email — not paraphrased, because paraphrase is where scope quietly grows.

How to read a completed matrix. Three rules, and they are the whole method.

1. An empty cell under Covered means the claim covers nothing on that row. Not less — nothing. Silence is not permission, as the driving license is silent about buses.
2. A matrix with an empty Excluded column is not finished. A claim that excludes nothing has not been scoped, it has been advertised. Every real certification in this episode carries its exclusions on its face, and their absence is the fastest tell there is.
3. The Silently assumed column is where the work is. Covered and Excluded can be filled by reading. Silently assumed requires someone who knows the physical site to sit with someone who knows the model. If you completed the matrix without that conversation happening, you did not complete it.

A worked example. The following is constructed and system-class — assembled from public patterns, describing no real operator, product or incident. The mechanism it illustrates is the point.

The claim, as made: "Our simulation validated the new pick policy across four million simulated hours at a 99.4 percent success rate, with zero collisions."

Notice what the matrix did. It did not dispute the 99.4 percent. It never needed to. The number is very probably correct, and correct about the simulator.

The machinery, dated this week

One last piece, because it shows how small the apparatus actually is that converts simulated hours into a real credential.

On 27 August 2026, the FAA published a notice under docket FAA-2025-2535, OMB control number 2120-0755, seeking to reinstate an information collection that expired in December 2025. The notice describes what the collection is for:

"FAA aviation safety inspectors (ASIs) review the Airline Transport Pilot (ATP) Certification Training Program (CTP) submissions to determine whether the program complies with the applicable requirements of 14 CFR 61.156."

That is the section three headings above — the one assigning upset recovery to a Level C or higher simulator. And the notice gives the size of the operation: 5,523 respondents, once per year, an estimated average burden of 14 minutes per response, 1,329 hours of total annual burden. Comments are due 28 September 2026.

Fourteen minutes. That is what stands between a simulator's qualification and a person being credentialed on the strength of hours spent inside it: an inspector, a form, and a rule that says which tasks that device's level may cover. The apparatus is not large. It is specific, it is written, and it existed before anyone needed it.

The strongest objection

The sharpest pushback is that this whole comparison is a retrofit, and the objection deserves full strength.

A flight simulator models a type-certificated airframe: a fixed, bounded, extensively instrumented physical object with an authoritative comparator produced by a flight test program that cost a fortune. A robotics or autonomy twin models an open world with no comparator dataset, no type certificate, no equivalent of a flight test program, and a system whose learned behavior can differ between two Tuesdays. Two provisions of Part 60 do not survive contact with that: the annual pilot statement and the recurring objective tests both assume a system that holds still between reviews. Anyone claiming aviation has solved twin certification for machine learning is overstating it, and this piece does not.

Here is why the objection does not dispose of the argument.

The transferable object was never the tolerance table. It is the scope statement, the named comparator, the stated applicability domain, and the record of the gaps. Those four cost nothing and require no flight test program. They are documents. An operator can demand all four tomorrow from any supplier, and the supplier's ability to produce them is itself the most informative test in this entire episode.

And the regulation has already been tested against exactly the situation the objection describes — a genuinely new class of machine with no simulator standard in existence. Section 194.105, part of the powered-lift special federal aviation regulation issued in November 2024, says what happens then:

"For flight simulation training devices (FSTDs) representing powered-lift for which qualification standards have not been issued under part 60 of this chapter, the applicable requirements will be the portions of the flight simulation training device qualification performance standards contained in appendices A through D to part 60 of this chapter that are found by the Administrator to be appropriate for the powered-lift and applicable to a specific type design, or such FSTD qualification criteria as the Administrator may find provide an equivalent level of safety…"

And the next paragraph: those proposed standards "will be published in the Federal Register for comment."

No standard existed. The regulator did not let the simulation's own score fill the vacuum, and did not wait for a standard either. It named an official to decide which portions of the existing framework apply, per type design, and published the decision. That is the move available to any operator right now, at company scale, without waiting for anybody: decide which parts of a mature framework apply to this specific system, write the decision down, and let people argue with it.

What would falsify this

It is a claim about the prevailing pattern, not about every supplier. A twin package that already ships a scope statement, a named comparator with its date and the unit it was measured on, an enumerated applicability domain, and a gap record with rationale for the empty cells, is doing the thing this episode asks for, and the argument is inapplicable to it. I have not surveyed every supplier, and absence of publication is not absence. One public, checkable counterexample would confine this to a description of what the rest of the field does.

I cannot quantify how much a scope statement reduces risk. No published measurement exists, and anyone offering a number should be asked for the source.

A regulator could extend a credibility framework explicitly to machine-learned models, which would make building the discipline early merely the cheapest way to have been right.

And no published incident is claimed here. This is an argument about what an operator can demonstrate about a simulated claim, not an assertion that harm has occurred.

Read the artifact in front of you

All five chapters of this series are running the same exercise this week, on five different documents. The AI Boardroom reads an examination list. Ship AI reads a permission model. Sovereign Stack reads a model card as a contract. Beyond the Benchmark reads a benchmark. This chapter reads a certification scope.

The instruction underneath all five is the same, and it is deliberately unglamorous: read the artifact in front of you, as it is actually written, including the parts that say what it does not do. The interesting sentence in a certification is almost never the one on the cover. It is the one that starts with the exception of.

The close

A digital twin certifies a boundary. Inside it, within a stated tolerance, against a named measurement of the real system, over a stated range of conditions, for a listed set of tasks, until something changes. That is a great deal, and it is the reason airline pilots practice engine failures on the ground rather than over a city.

The sim score cannot say where that boundary is. It is computed inside it. It cannot say what was left out of the model, because the model does not know. It cannot say whether the real thing has moved since the last time anyone checked. And it cannot warn you that a cell in the evidence table is empty — unless somebody built the table and required the empty cells to be explained.

Somebody did, decades ago, in a different industry, in a document you can open right now in a browser without paying anyone.

Nothing outside the line was tested. The line is the product.

What to ask your team

1. For any twin or simulation we rely on: where is the sentence that says which decisions it may be used for, and whose name is on it?
2. What measurement of the real system was the model compared against — who took it, when, and on the production article or a prototype?
3. Over what range of conditions was that comparison made, and where does our intended use sit relative to that range?
4. Which cells in our evidence table are empty, and where is that absence written somewhere a decision maker will actually see it?
5. What changes to the physical site or system automatically invalidate the model, and who is told when one of them happens?

The series

This is Episode 2 of The Operator's Map — a weekly series in five chapters, advancing together: Ship AI teaches agent controls, Sovereign Stack the open-source stack, The AI Boardroom governance, Beyond the Benchmark evaluation, Twin & Machine physical AI. Next week, this chapter teaches what an operator owes a machine it did not build — how an acceptance test differs from a demonstration, and what belongs in one. Subscribe to follow the map as it fills in.

Cut in verification, and why

This chapter's Episode 1 shipped three claims that a later pass had to cut or correct. The list below is long on purpose.

• Everything about the internal framework of the FDA guidance. Context of use, model risk, model influence, decision consequence and the credibility factors are all described widely in commentary. The guidance itself could not be retrieved from any FDA host — the landing page and the document URL both redirect an automated client to an abuse-detection page and return HTTP 404. Cut. This episode claims only what appears verbatim in the Federal Register notice of availability.
• Any quotation from ASME V&V 40. Paywalled, and its product page returned 404 to an automated client. Cut. The standard is named only because the Federal Register notice names it in full.
• ISO 21448, ISO 10218, UL 4600 and UN Regulation No. 157. All four were candidates for anchoring this episode in autonomy or industrial robotics. Every issuing-body URL refused an automated client or paywalled the text. Cut entirely. No claim here rests on a standard the reader cannot open.
• Any claim about India's or the Gulf's own simulator qualification regimes. Both civil aviation authority sites answer, but no stable, citable instrument URL was located and no instrument text was read. Cut. The cross-border point is made instead through § 60.37 of the anchor instrument, which recognizes another State's qualification only through a bilateral agreement and a simulator implementation procedure.
• ICAO Doc 9625. Not fetched. Cut as a source. It appears in this chapter's research only inside a verbatim quotation from 14 CFR Part 60, which is where the reference actually lives.
• A security angle drawn from recent advisories on engineering-analysis software. On-topic advisories exist but fall outside the 72-hour window this episode's sweep was bounded to, and each would need verification against the vendor's own published advisory. Cut from this episode.
• Anything from the national vulnerability database. Keyword-filtered queries inside the 25 to 27 August 2026 window returned zero results. An unfiltered query in the same window returned 1,302 records. That is a query behavior, not a finding, and reporting it as one would repeat the error this series warns about. Cut.
• The claim that no regulator anywhere has qualified a robotics or autonomy twin under a published scope statement. Plausible, not established, and the search was not exhaustive. Cut, and replaced by the falsifiability section above.
• Every vendor metric. Episode 1 of this chapter used vendor-published figures, labeled them correctly, and still had to correct a deployment description that turned out to describe a simulated plant rather than a physical line. This episode cites no vendor at all, which removes the defect class rather than managing it.
• A worked example drawn from a real deployment. No real twin claim could be read at a primary in enough detail to fill the matrix honestly, so the example above is constructed and system-class and is labeled as such where it appears.