Vendor AI-claim teardown rubric
Eleven claims that appear on almost every AI vendor's page, what each would require to be true, and the question that settles it.
Why it exists
Vendor diligence usually fails politely. The security questionnaire comes back complete, every box is ticked, and nobody has established what the words mean. These are the claims that recur, the specific thing each one is doing rhetorical work to avoid saying, and a single question that cannot be answered with marketing language. None of this is an accusation — most of these claims are true in some narrow sense. The problem is that the narrow sense is rarely the one the buyer heard.
What it contains
Eleven recurring claim patterns, with the gap between the claim and its usual meaning
What would have to be true for the claim to mean what a buyer assumes
One question per claim that cannot be answered with marketing language
The answer that should end the conversation
How to record the result so it survives into the contract rather than the sales call
No email required, and nothing is recorded when you download. Use it, adapt it, argue with it — attribution is welcome, not a condition.
The claims
| Claim | What it usually means | What it would take to be true | The question that settles it |
|---|---|---|---|
| SOC 2 compliant | A Type I report, or a Type II whose scope excludes the AI service you are buying. | A current Type II whose system description names this service, with exceptions read rather than skipped. | Send the Type II report and show me where this product sits in the system description. |
| Explainable AI | Feature attributions such as SHAP on a model score. | A reconstruction of the decision the customer actually experienced, including retrieval and tool calls. | Show me the explanation a customer would receive for one declined case. |
| Human in the loop | A screen exists where a person can intervene. | Evidence of what the reviewer saw and how long they had before deciding. | What is your median reviewer dwell time, and what do they see? |
| Your data is never used for training | Not used for training the shared foundation model. | No retention, no human review, no fine-tuning, no evaluation reuse — with retention windows stated. | Where does my prompt and output go, for how long, and who can read it? |
| Enterprise-grade guardrails | Prompt-level instructions and a content classifier. | Enforcement between the agent's intent and the system of record, capable of refusing the action. | Show me a blocked action in your logs, not a blocked message. |
| Full audit trail | Chat transcripts and API logs. | Immutable action records with authority, policy version, inputs, and model and prompt versions. | Reconstruct why one action six months ago was permitted. |
| EU AI Act ready | A blog post and a compliance page. | A stated role — provider or deployer — and the specific obligations accepted, with dates. | Which role do you take under the Act, and which articles do you accept? |
| Zero hallucination | Retrieval-augmented generation with citations. | A measured error rate on your data, with the measurement method disclosed. | What is the residual rate, on what evaluation set, measured how? |
| GDPR compliant | A DPA template and EU hosting. | A named role, a lawful basis, sub-processor transparency and a working erasure path through the model estate. | Delete one subject's data end to end and show me the evidence. |
| Isolated tenant | Logical separation in a shared model deployment. | Named isolation boundary — dedicated inference, or shared with stated controls — and where it stops. | Draw the boundary and tell me what crosses it. |
| Model agnostic | A provider abstraction layer. | Evidence that behaviour, evaluations and controls survive a provider swap. | When did you last switch a customer's underlying model, and what changed? |
Recording the result
The point of the rubric is not to win the meeting. It is to convert an answer into something enforceable before signature.
| Outcome | What to do with it |
|---|---|
| Answered with evidence | Reference the artifact in the contract, with a right to re-request it. |
| Answered credibly, no evidence yet | Make it a condition precedent with a date, not a roadmap item. |
| Reframed rather than answered | Record the reframing verbatim. It is usually the real scope. |
| Refused as proprietary | Legitimate for some material. Convert it into an audit right and a notice obligation. |
| Answered with a certification | Certifications describe a scope. Read the scope, not the badge. |
How to use it
- Send the questions in writing before the demo. The demo is designed to answer different questions.
- Accept artifacts, not assurances. A named report beats a confident answer.
- Record verbatim answers, including reframings — the reframing is usually where the real scope is.
- Convert each answer into a contract term, a condition precedent, or an accepted risk with an owner.
- Re-run it at renewal. Most of these answers change when the underlying provider does.
Built from public standards and general practice. Regulatory instruments move — check anything cited here against its primary source on the day you rely on it. Related: the other artifacts · the diagnostic.
Want this applied to your architecture?
The artifacts are general by design. Applying one to a specific estate is what the governance review does.