I spend most of my working life on a question that sounds abstract until it isn't: when a software agent takes an action, what authority permitted it, and can you prove it afterwards? Over the past year that question has migrated into physical systems, and the migration exposes something worth stating plainly.
Embodiment does not introduce a new governance problem. It removes the escape hatches from the existing one.
In software, a wrong action is usually reversible. A transaction is unwound. A record is restored from a snapshot. A message is retracted with an apology. That reversibility is load-bearing for how enterprises actually govern agents — far more load-bearing than most governance documents admit — because it is the reason detect-and-correct is an acceptable posture for a great many workflows. You do not have to prevent everything if you can catch and undo most things.
Give the agent a body and the posture stops being available. Mass moves. Contact happens. The correction, if there is one, is medical or structural.
The uncomfortable implication is not that robotics needs a new governance discipline. It is that a lot of enterprise AI governance has been quietly relying on reversibility to do the work of controls that were never built — and embodiment is simply the case where that reliance becomes visible. The obligations were always there.
Entitlements become envelopes
In software, an agent's grant is a scope over data and actions: which systems, which operations, which value limits, granted by someone with the authority to grant them. For an embodied system it is also a physical envelope — speed, force, torque, workspace boundaries, proximity conditions, permitted operating states.
The good news, and it is genuinely good, is that the industrial world already knows how to express most of this. Safety-rated monitored stop, speed-and-separation monitoring, and power-and-force limiting are decades-old concepts with rated implementations, and ISO 10218 together with ISO/TS 15066 gives them shape for industrial and collaborative robots specifically. A physical envelope is not a novel artefact that someone has to invent; it is an existing engineering object with existing verification practice.
What is new is that the entity operating inside the envelope is now a learned policy whose behaviour was characterised statistically rather than specified deterministically. That single change has one non-negotiable consequence: the envelope must be enforced outside the policy, by mechanisms that do not share the policy's failure modes. A bound that the learned system can reason its way past is not a bound — and the way learned systems get past bounds is rarely dramatic. It is usually that the bound was computed from the same inputs the policy was reasoning about, so when the inputs were wrong, both were wrong together.
The independence property, which is where this actually goes wrong
This deserves its own section, because it is the failure I would expect to see most often as learned systems arrive inside industrial cells, and it is a failure with a well-known name in functional safety: common-cause failure.
Picture an architecture that looks responsible on a slide. A perception stack processes camera input. A learned policy decides actions from that perception. And a protective function monitors for unsafe states — using the same perception stack, because that is where the information is and because building a second sensing path costs money.
Now consider what happens when perception fails — an unmodelled lighting condition, an occlusion, a surface the segmentation model has never seen. The policy does the wrong thing, and the protective function does not notice, because it is looking through the same eyes and is blind in exactly the same way. The guard is decorative in precisely the circumstances it was installed for.
This is the oldest mistake in functional safety, freshly re-made in a new idiom, and it is easy to make because the shared-dependency version is cheaper, simpler and performs better in testing — since the correlated failures it cannot catch are, by construction, the ones least likely to appear in a test set drawn from the same distribution.
The architecture that works keeps a deterministic, rated protective layer between the learned system and the physical world, with its own sensing, and treats the learned system — however sophisticated — as untrusted with respect to that layer. This is where physical engineering is genuinely ahead of enterprise software: functional-safety practice has always insisted that protective functions be independent, rated and verifiable, and IEC 61508 and ISO 13849 exist precisely to formalise that insistence.
The mistake I watch teams make is assuming the AI stack inherits that rigour by proximity. It does not. A learned policy is not a safety function. An ML-based perception pipeline is not a safety-rated sensor. And “the model is very accurate” is not a performance level — it is not even the same kind of statement, since a performance level is a claim about failure behaviour under specified conditions and an accuracy figure is a summary of performance on a sample.
From the learning side, the field is developing its own contributions to this layer, and they are worth knowing rather than dismissing: control barrier functions, model-predictive safety shields, constrained reinforcement learning, and the broader separation between safe exploration and safe execution — surveyed comprehensively in the December 2025 review of safe learning for contact-rich robot tasks. These are real and improving. They are also, by that survey's own framing, mitigations within an evolving challenge rather than guarantees, which is exactly why they belong alongside a rated independent layer rather than instead of one.
Attestation becomes reconstruction under adversarial conditions
Six months after an incident, an investigator — a regulator, an insurer, opposing counsel — will ask what the machine perceived, what it inferred, what authority it acted under, and what the protective layer saw at the moment of action.
In software this is a logging problem, and a well-understood one. In robotics it is a synchronisation problem across sensors, policy inference, control loops and safety hardware, at timescales where ordering matters and clock skew is a real adversary rather than a pedantic concern. The question is not merely what happened but in what order, across subsystems running at different rates on different clocks, where the difference between the protective layer having seen a condition before or after the policy committed to an action is the difference between two entirely different findings.
It is harder to build than software attestation, and the consequence of not having it is proportionally worse — because physical incidents attract a class of scrutiny that software incidents mostly do not, and they attract it from parties with subpoena power and their own experts.
The design implication is that attestation cannot be retrofitted as logging. It has to be a synchronisation architecture decided early: a common time base across the safety and control domains, records that capture the protective layer's view alongside the policy's, and retention that outlives the investigation window rather than the debugging window. Every one of those is cheap at design time and effectively impossible to add afterwards.
Model risk becomes lifecycle risk
This is the layer I would point every embodied-AI team toward first, because it is the one neither parent culture handles and therefore the one most likely to be nobody's job.
These systems are not deployed once. They are updated — a new perception model, a new policy, new firmware, a new platform version you may not have chosen. And every update silently re-opens every safety claim that was made about the previous configuration. Not conceptually: literally. A validation performed against configuration A is evidence about configuration A, and configuration B is a different system that happens to share a name.
The industrial world has change control, which is thorough and far too slow for a stack whose components ship monthly. The AI world has continuous delivery, which is fast and constitutionally unwilling to ask whether a deployment invalidates a safety argument. Embodied AI sits precisely where those two cultures collide, and what is needed is neither of them unmodified.
The 2026 work on governed capability evolution makes this concrete, with embodied agents as its case study: lifecycle-time compatibility checking and rollback for AI-component-based systems. The mechanism is a gate that asks, before an update reaches hardware, which prior claims this invalidates and what must be re-established — plus a rollback path that is rehearsed rather than theoretical.
The reported results are worth carrying into a design discussion precisely because they are not a free lunch. Governed upgrade retained comparable task success at 67.4 percent while maintaining zero unsafe activations across all rounds, in contrast to ungoverned approaches. That is a real capability cost, honestly stated, in exchange for a safety property. Which is the correct shape for a control to have: something you can weigh, rather than something that claims to cost nothing.
One caution on reading that number. It is a single study on a specific setup, and the absolute figures will not transfer to your system. What transfers is the structure — a gate keyed to claims rather than to metrics, and a rollback that has been practised — and the demonstration that the structure is implementable rather than merely advisable.
The general-purpose problem: an envelope presumes a bounded task
Everything above works cleanly for a robot doing a defined job in a defined cell. It gets substantially harder for the systems the field is most excited about, and the difficulty is structural rather than a matter of maturity.
A physical envelope is derived from a task. You know the workspace because you know what the machine is reaching for; you know the force limits because you know what it is handling; you know the proximity conditions because you know where people need to be. The risk assessment that produces the envelope presumes a bounded set of things the machine will do.
A general-purpose embodied system inverts that. Its value proposition is precisely that its task set is not fixed — that it can be asked to do something tomorrow that nobody enumerated today. Which means the envelope cannot be derived from the task, because the task is open. You are left with two options and both have costs.
The first is to derive the envelope from the workspace instead of the task: bound speed, force and reach conservatively enough that the machine is safe for any task it might attempt within that space. This works and is what most current deployments effectively do. It also throws away much of the capability that motivated a general-purpose machine, because the conservative envelope that is safe for every possible task is dramatically tighter than the one that is safe for the actual task.
The second is to make the envelope task-conditional: the system declares what it is about to do, an envelope appropriate to that task is instantiated, and the protective layer enforces it for the duration. This preserves capability, and it introduces a new and interesting attack surface — the declaration. If the system's own statement of intent selects the bound it will be held to, then a system that is wrong about what it is doing selects the wrong bound, and the failure mode of the perception stack propagates directly into the envelope. Which returns us to independence: the declaration has to be validated by something other than the thing making it, or the envelope is once again derived from the policy's own view of the world.
I do not think this is solved, and I would treat any vendor claim that it is with scepticism. The honest current position for general-purpose embodied systems is the conservative envelope, accepting the capability cost, with task-conditional envelopes reserved for task classes that have been individually assessed. That is unsatisfying and it is where the evidence currently sits.
What “may” means when the actor has mass
There is a distinction I press on in every governance review, and it survives the move into physical systems intact: the difference between what a system can do and what it may do.
Can is a statement about capability — reachability, force, the action space the hardware and software make available. May is a statement about authority: a scope, granted by a named person with the standing to grant it, over a defined period, for defined purposes. Most programmes can answer the first question in detail and struggle visibly with the second, and the struggle is more consequential here than in software because the gap between can and may is measured in kilograms and metres per second rather than in database rows.
The physical version of the authority question has a feature software lacks, which is that the answer is partly architectural and therefore partly permanent. If a machine's actuators cannot produce a force above a rated limit, it may not exceed that limit in the strongest possible sense — the authority is enforced by physics rather than by configuration. Programmes routinely under-use this. There is a persistent preference for enforcing limits in software because software is flexible, and flexibility is precisely the property you do not want in a bound.
So the design question worth asking early, while it is still cheap: of the limits this system will operate under, which are enforced mechanically, which by rated hardware, which by software outside the policy, and which by the policy itself? Those four categories carry radically different assurance, and the ordering is not a matter of taste. A limit enforced by the policy is not a limit; it is an intention.
The strongest objection: the industrial world already has all this
The best challenge to this whole argument comes from experienced safety engineers, and it goes roughly: none of this is new. Machinery safety has handled dangerous automated equipment for decades. We have risk assessment, we have performance levels, we have independent protective functions, we have change control. Learned policies are a new kind of occupant for a very old kind of cell, and the existing framework already handles unpredictable occupants — that is what the envelope is for. The AI governance world is arriving late and re-describing solved problems in its own vocabulary.
A great deal of that is correct, and the sections above concede it explicitly: the envelope concepts exist and are rated, the independence principle is established, and physical engineering is ahead of enterprise software on exactly the layer that matters most. A team that rigorously applies existing machinery-safety practice to a learned system will get most of the way, and will do better than a team that applies novel AI-governance frameworks without it.
Two things are genuinely new, though, and they are the two the existing framework was not built to carry. The first is the pace and character of change: machinery change control assumes changes are discrete, infrequent, and initiated by the operator, and none of those hold for a system whose perception model updates on a vendor's cadence. The second is that the envelope's adequacy now depends on evidence produced in simulation, whose residual is unstated — so the existing framework's assumption that the hazard analysis is grounded in characterised behaviour is weakened at its base.
So the honest position is that this is not a new discipline, it is an old discipline with two specific gaps, and the useful contribution from the AI-governance side is narrow: lifecycle-time change gating, and evidence provenance for simulated validation. Everything else, defer to the safety engineers.
Three markets, three different starting positions
This is not a Silicon Valley story, and the first move differs by where the machine is being installed.
In North America, the machinery-safety and workplace-safety framework already governs the cell and the institutional capability exists. The open question is narrower and harder: who accepts responsibility when the thing inside the cell learns, and how an apparatus built around specified behaviour consumes a statistically characterised policy. The work is integration with a mature regime.
In India, an industrial build-out is deploying automation at speed alongside a modernising factory-safety regime. The opportunity is to build the assurance layer as the automation goes in, rather than retrofitting it a decade later — which is the more expensive path every industrialised economy has already taken. The constraint is the small population of people who can write a credible safety case for a learned system, which argues for documented, transferable practice over individual expertise.
In the Gulf, giga-projects, automated ports and logistics are being commissioned as new-build, which is the single best position anyone can be in: the evidence layer costs a fraction to design in than to add, and nothing legacy is in the way. The specific exposure is that new-build buys integrated stacks, and integrated stacks make the independence property easy to assume rather than to verify — the protective layer arrives in the same box as the thing it is protecting against, and somebody should still be asking whether it shares a sensing path.
Six questions for the design review
Compressed into the form I actually use, because the four layers are a framework and a framework is only useful when it turns into questions somebody has to answer out loud.
What may this system do, and who granted that — in business terms, with a named grantor and a scope, not a description of what the hardware is capable of. If the answer describes capability rather than authority, layer one does not exist yet.
Which limits are enforced mechanically, which by rated hardware, which by software outside the policy, and which by the policy itself? The last category is not enforcement and should be named as such in the room.
Does the protective layer share any sensing, compute or software dependency with the thing it is protecting against? If yes, name the common-cause failure and say why it is acceptable — which is a legitimate answer, but only when it has been said.
If this system did something unexpected on a Tuesday, what could you reconstruct about what it perceived, what it inferred, what authority it acted under, and what the protective layer saw — and can you order those events across subsystems? Ordering is the part people discover they cannot do.
What happens to your safety claims when the perception model, the policy, the firmware or the platform updates — and is there a gate before that reaches hardware? Follow up with: when was the rollback last rehearsed? An unrehearsed rollback is a plan, not a capability.
Which of your claims rest on simulated validation, and what is the stated residual? This is the question that connects this piece to the rest of the series, and it is the one most likely to produce a silence.
What I expect to break first
A prediction, offered as a prediction so it can be judged later rather than quietly revised.
I do not expect the first serious embodied-AI incidents to come from a policy doing something exotic. I expect them to come from the update path — a perception model or policy revision that improved every metric its team tracked, shipped without a compatibility gate, and invalidated a safety claim nobody re-checked because nobody had written down which claims depended on that component.
There are three reasons to expect this shape. The failure is mundane and therefore likely, since mundane failures are the ones that recur. It sits exactly in the seam between two organisational cultures, and seams are where accountability goes missing. And it is invisible in testing, because the update genuinely is better on the distribution it was evaluated against — the claim it broke was about a condition the evaluation did not include.
If that is right, the highest-value thing a programme can do this quarter is not to improve its models or buy a better simulator. It is to write down which safety claims depend on which components, so that an update has something to be checked against. That register is unglamorous, takes a week, and is the artefact that turns a lifecycle failure from an unbounded investigation into a lookup.
Where this argument is weakest
Two admissions, in keeping with the rest of this series.
First, I am arguing from software-agent governance into a domain where safety engineers have decades of practice I do not have. The four-layer mapping is a translation, and translations lose things. Where this piece and an experienced functional-safety engineer disagree about the physical layers, they are probably right and I am probably importing an assumption that does not hold outside software.
Second, the lifecycle-risk argument rests substantially on one study. It is a good study, its case is embodied agents, and its result is stated with an honest cost — but it is one result, and the specific numbers should not be carried into anyone's design review as though they were a benchmark. The structural claim, that updates re-open claims and therefore need a gate, is robust independent of that study. The quantification is not.
The uncomfortable summary
Enterprises are about to discover that the hardest part of physical AI is not making the machine capable. Capability is arriving on a steep curve from platforms and foundation models, and it will keep arriving whether or not the assurance practice is ready.
The hard part is the same as it has been in every regulated deployment of the past two years: bounding what the system may do, enforcing that bound with something that does not share its failure modes, and being able to prove afterwards what happened. What changes with a body is only this — you no longer get to fix it afterwards.
Design accordingly. The machine will not wait for the governance to catch up, and physics does not accept a remediation plan.