Machine-Evidence Schema Audit — 6 September 2026
You can pay to be tested. You cannot pay to be trusted.
Question
After the first end-to-end external Assurance Runner commissioning run, Atinamos deliberately gave published evidence to independent AI consumers and asked a simple question:
Can a buyer agent read the evidence and reach only the conclusion the observation actually supports, without silently broadening the claim?
No production behaviour, payment policy, signing key, evidence record or historical receipt was changed by this audit.
What already works
The current evidence model separates important execution stages:
seller offer
→ payment contract
→ authorisation
→ authorised request
→ settlement
→ fulfilment
→ output validation
The public methodology already states that evidence is timestamped, later stages must not be inferred from earlier ones, missing evidence is not negative evidence, successful and unsuccessful observations are retained, and the buyer owns its procurement policy.
The newer signed Assurance Evidence Receipt adds canonical signed payloads, content commitments, signing-key identity and integrity metadata.
Finding 1 — capability scope can be broadened too easily
The IBANforge commissioning independently checked only the returned valid field using ISO 13616 MOD-97.
An AI consumer initially described the service more broadly in terms of IBAN validation, BIC, bank-code and SEPA data.
The evidence did not independently verify those enrichment fields.
Machine-readable evidence should preserve the distinction between:
tested capability
independently verified output fields
fields merely observed in the seller response
fields not independently evaluated
Finding 2 — observation counts are not reliability percentages
An AI consumer saw a small sample of successful seller-attributable paid observations and described the service as having “100% reliability”.
That is stronger than the evidence supports.
successful_fulfilments: 3
failed_fulfilments: 0
means exactly three successful observed fulfilments and zero observed failures in the published sample. It is not a statistical estimate of permanent or future reliability.
Finding 3 — failure attribution must remain explicit
The code402 LEI series retained one Circle smart-account pre-settlement interoperability observation alongside two successful EOA paid controls.
The narrow finding was path-specific:
- the tested Circle smart-account authorisation independently validated under EIP-1271;
- the observed code402 payment path rejected it before settlement under ordinary EOA signature recovery;
- no paid fulfilment was reached;
- the observation does not establish universal smart-account incompatibility.
Machine evidence should keep categories such as seller performance, verifier/system failure, payment interoperability, pre-payment observation and unknown separate.
Finding 4 — a fresh fixture is useful evidence, not proof of implementation
A fresh test input can reduce the chance that a provider is simply returning a memorised response to a known static fixture.
It still does not prove:
- the provider's internal implementation mechanism;
- arbitrary future-input behaviour;
- that verification traffic was indistinguishable from ordinary buyer traffic.
Finding 5 — one Boolean should not carry every meaning
The original public receipt v1 used fields including:
fulfilment.observed
fulfilment.output_valid
Historical receipts must remain valid and immutable.
Newer Assurance evidence needs clearer machine distinctions between:
useful output observed
independent correctness check reached
capability / field checked
expected value
observed value
validation result
The preferred direction is additive/versioned evolution rather than rewriting historical evidence.
Finding 6 — HTTP method is part of evidence identity
The exact route is not just a URL.
GET https://service.example/task
POST https://service.example/task
may represent different contracts with different payment requirements, request bodies, fulfilment behaviour and evidence histories.
Current exact-route Assurance evidence accepts both endpoint and method:
GET /v1/assurance/evidence?endpoint=<service-url>&method=<HTTP-method>
A buyer must not infer that evidence for one method automatically applies to another method at the same URL.
Current design direction
The audit found no reason to replace the core Atinamos evidence architecture.
The main improvement is to make these concepts harder for consuming systems to blur:
identity
endpoint
method
scope
tested capability
verified output fields
observed-only fields
correctness
independent method
expected
observed
result
attribution
category
seller-performance evidence
sample
observation counts
counts are not reliability estimates
limitations
future behaviour not established
implementation mechanism not established
traffic indistinguishability not established
Compatibility rules
Any future schema tightening should preserve these rules:
- historical receipts remain immutable;
- signed content is never silently rewritten;
- endpoint and method remain material service identity where applicable;
- unknown remains distinct from false;
- no evidence remains distinct from negative evidence;
- seller performance, verifier failure and payment interoperability remain separately attributable;
- a narrow correctness check must not be presented as verification of unrelated response fields;
- observation counts must not be converted into a reliability percentage by Atinamos;
- procurement policy remains buyer-owned.
Result
The audit supports keeping the core model and tightening machine semantics around:
identity · scope · correctness · attribution · limitations · observation counts
That is deliberately different from creating a central “trust score”. Atinamos should expose enough precise evidence for a buyer to make its own decision without encouraging the evidence to be inflated into a stronger claim.
Technical record
The fuller public technical note is maintained in the Atinamos research repository: