Add eval-result predicate (AI/ML evaluation results)

[b7n0de]

Closes the ask in #565 (proposal discussion since 2026-07-03), as offered there on 2026-07-18.

What this adds

eval-result.md per the ITE-9 template plus one line in the predicate directory: a predicate
for signed, offline-verifiable AI/ML evaluation results. A claim is a machine-checkable
metric <comparator> threshold verdict over N samples, with salted model/dataset commitments,
an optional per-sample RFC 6962 Merkle root, an optional pre-registration binding, and an
explicit assuranceLevel (self_attested | third_party | reproduced | enclave_attested).
Protobuf bindings follow as a separate PR, per the SVR precedent (#470 -> #519 -> #537).

Changes since the initial submission

Additive, all in the one file, all out of the discussion in #565 and in this thread: harness
gained an optional digest binding the exact harness artifact; a general rule that absence of
an optional field means no claim is made and consumers MUST NOT synthesize a default from it;
an explicit non-claim that this predicate does not establish that the harness or grader is fit
for purpose; passed stated as a signed threshold verdict; assuranceLevel marked as
issuer-declared; anchors[] removed (887708f); and an informative note on the asymmetric
consequences of the harness non-claim. The diff has been stable since 35c83da.

Why a new predicate

Discussed in #565: test-result carries PASSED|WARNED|FAILED plus test names but no
metric/threshold/dataset digests, commitments or per-sample structure; SVR's property list
would need a schema re-invented inside a free-form field. eval-result keeps the claim shape
fixed so a policy engine can evaluate "did model M clear threshold X on benchmark Y" without
parsing prose out of a system card.

Scope, pinned

Metric evidence only. Not an agent-decision attestation, not an action authorization, no
decision fields will be added (boundary suggested by @clementineCU in #565 and kept). A
separate decision-oriented predicate MAY reference signed eval-result statements by digest.
The attestation proves who signed which claim and that it verifies offline, not that the
metric is true, the benchmark well designed, or the model safe.

Maturity evidence

Item	State
Reference implementation	proofbundle (MIT), current release 4.0.0 on PyPI; emits DSSE-signed statements + RFC 6962 proofs
Example statements	three modes, versioned: public-model, private-model (receipt), release-gate
Real-world statement	1181-sample eval posted in #565, verifies offline (signature, canonical bytes, subject binder)
Second implementation	MarkovianProtocol/audit-anchor reproduces canonicalization and content roots byte-for-byte
Graduated cross-impl case	decision-crossimpl-schema-conformant in the conformance corpus: an independently implemented decision receipt binds an eval-result evidence statement by digest, both implementations agree byte-for-byte on canonical bytes and content roots (decision root 97e1d74b…), recorded schema-conformant against the published schema, vendored as pure data with permanent attribution
Composition in the wild	a third participant in #565 (2026-07-19) builds on exactly this composition pattern, referencing eval-result statements by digest from a broader evidence profile instead of widening the predicate

Two design points flagged for review

1. In the private-model mode the subject is the eval receipt (a metadata object, permitted by
   the DigestSet definition as an immutable reference). Its sha256 is a documented binder over
   commitments, Merkle root and timestamp, marked by an explicit subjectDigestNote. As
   offered in New predicate proposal: eval-result (AI/ML evaluation results) #565: the receipt-as-subject digest-set question stays open for review here,
   and we are happy to switch to a custom digest-set key before merge if you prefer.
2. The statements declare one top-level extension field, contentRootAlg, naming the
   canonicalization the content root is computed over; ignorable by generic consumers per the
   spec's parsing rules.

Note

anchors[] (external time anchors over a canonical root, drafted with @MarkovianProtocol in
#565) is deliberately NOT part of this PR. The thread consensus is that it is not
eval-specific and belongs as a shared optional field; we would bring it as its own
discussion if there is interest.

DCO signed. markdownlint last ran clean locally at f2c26cb (pinned 0.49.1, rc=0); the
repository lint workflow is still awaiting maintainer approval, so it has not run here. Happy
to adjust naming, casing or file placement to match directory conventions.

[MarkovianProtocol]

I've checked the example I posted in #565 against the PR's final field set. All required fields are present and typed as specified, threshold is a decimal string, subjectProfile is receipt, and the optional receipt and harness blocks match. One migration note: the example carries the provisional predicateType https://b7n0de.com/attestation/eval-result/v0.1 — on merge it should re-emit under the in-toto.io URI. I'll regenerate it against the merged version.

[b7n0de]

Thanks for checking your example against the final field set, and for confirming the fields line up on your side too.

Agreed on the predicateType. The current https://b7n0de.com/attestation/eval-result/v0.1 is provisional. On merge I will move the spec header (Type URI and Version) and every example statement to the in-toto.io namespace, so the type reads https://in-toto.io/attestation/eval-result/v0.1, and I will add the redirect line per the new-predicate guidelines. Final name, version and file placement per your directory convention. If you prefer something different, I will match it.

Once it lands, please regenerate your example against the merged predicateType. Both implementations re-emitting under the same type keeps the canonical bytes and content roots agreeing byte for byte, and that is the part worth preserving.

[b7n0de]

Three corrections pushed, all additive, one file, +12/-2:

• harness can now bind the exact harness artifact through an optional digest (in-toto DigestSet). {name, version} stays conforming, and the empty-object case is left undefined rather than blessed.
• A general rule for optional fields: absence means no claim is made, and consumers MUST NOT synthesize a default from it.
• An explicit non-claim: this predicate does not establish that the harness or grader is fit for purpose or has any particular detection performance.

All three came out of the discussion in #565 with @ppcvote and @Ganador1.

DCO and Kusari Inspector are green. The lint workflow is waiting on maintainer approval, so markdownlint has not run here; locally it is clean with the version pinned in package.json (0.49.1, npm run lint, rc=0).

@in-toto/attestation-maintainers could you approve the workflow run, and could this go on the agenda for the next maintainers meeting? Happy to adjust anything, including splitting the three corrections into separate PRs if you would rather review them apart.

[ppcvote]

Read the three corrections against the diff. They address the concern, and the third one is a better answer than the field I suggested.

This predicate does not establish that the evaluation harness or grader is fit for purpose, or that it has any particular detection performance.

Stating the non-claim costs producers nothing and cannot go stale, whereas a harnessCharacteristics field would have needed every producer to have numbers most do not have, and would have quietly invited 1.0 as a placeholder — which is the exact failure I cited garak for. Declining to carry the data while naming what is therefore unknown is the stronger version. digest binds identity only sitting next to the field it qualifies is the right place for it too.

The general absence rule reads correctly to me:

absence of an optional field means only that no claim is made for that field. Consumers MUST NOT infer or synthesize a default value from absence.

One optional refinement, non-blocking and easy to scope out. The non-claim is symmetric, but its consequences are not, and a consumer deciding how much to lean on passed: true may not work that out unaided:

• On a capability claim — the model achieves X — a harness that under-detects understates the model. The error is conservative.
• On a safety claim — the model refuses X — a harness that under-detects yields passed: true when the model did the thing. The error runs the other way, and the attestation makes it portable and durable.

Both of the safety-shaped examples in the doc (refusal_rate >= 0.98, release gating on a passing eval) sit on the second side. If a sentence to that effect fits under Non-claims, it turns the disclaimer into something a relying party can act on. If it reads as guidance rather than specification, it belongs elsewhere or nowhere, and I would not hold the PR for it.

Thanks for tracing the {name, version} versus name-plus-digest mismatch back through the thread — I had read it as a difference between the example and the spec text and would not have guessed it was unintended.

[b7n0de]

Agreed, and thank you for the careful read.

The second branch is not hypothetical in this document. The private-model use case and the inline example both use refusal_rate >= 0.98, and the referenced release-gate example carries the same safety-refusal claim.

What the document does not fix is which class the harness counts as a positive, and the direction of the error follows from that rather than from the wording of the claim. So the note stays conditional. My first attempt at it did not: it asserted that the examples above and below fall on the second side, which the predicate does not establish. I have replaced it with the form below.

It is under Non-claims as informative text, without a normative keyword:

The consequences can be asymmetric. When detected positives are evidence of capability, missed positives can understate performance. When passed: true depends on the absence of detected failures, missed failures can instead yield a passing verdict even though the failures occurred. This attestation authenticates either verdict without establishing the harness's detection capability.

That keeps the consequence tied to the semantics of the measured event rather than assuming one universal direction for every metric or positive-class convention.

After this commit I will hold the diff stable for maintainer review, apart from changes requested during review. Broader detector-characterization work remains separate.

And on the harness mismatch, credit is still yours. Treating the digest binding as a premise is what exposed the gap between the worked example and the spec text. I would not have inspected that boundary otherwise.

[MarkovianProtocol]

Worked-example update relevant to the anchors[] discussion: the transparency log referenced in the eval-result examples now takes public submissions and issues offline proof bundles. An eval bundle's sha256 can be notarized with one POST; the receipt carries an RFC 6962 inclusion proof and a c2sp.org/tlog-proof bundle under a 7-witness quorum, so a reviewer can re-verify an anchors[] target fully offline.

https://markovianprotocol.com/log.html

[b7n0de]

Thanks for the update, and it lands on the right side of the scope line for this PR.

I verified one of that log's proof bundles independently before replying, an existing leaf rather than one I submitted, so this exercises the read and verify path and not the POST door you describe. An RFC 6962 recomputation written from the spec, not from our own code, reproduced the checkpoint root from the leaf bytes; proofbundle agreed at --threshold 4, with five witness keys taken from parties other than the log rather than from its own /policy; and flipping a single payload bit failed inclusion while the log signature and the cosignatures stayed valid, which is the split a relying party needs. The write-up and the frozen fixture live where the anchor work lives, in b7n0de/proofbundle#7 and b7n0de/proofbundle#136, so they do not land here.

For this PR nothing changes. anchors[] came out in 887708f and stays out of eval-result/v0.1; if there is interest it comes back as its own discussion, as the description says. What your update does support is that a target of this shape is re-verifiable offline by a third party, which is an argument for that separate discussion rather than for widening this one.

The diff has been stable since 35c83da.

[b7n0de]

The diff has been stable since 35c83da, and this is a design question rather than another
change: before iterating further we would like maintainer input on three points that came out
of an external review of the current draft. We are deliberately not touching the schema until
there is direction.

1. Role separation: producer, evaluator, verifier.
verifier.id is currently documented as "the party that emitted/verified the result" — one
required field carrying two roles. As we read it, the name also overlaps with the verifier role
in the SVR predicate, and the same field name meaning different things across predicates is
worse for a policy engine than a missing field. The direction we would propose: an
evaluation-specific evaluator descriptor for the party that ran the evaluation; reserve
"verifier" for the SVR sense; and leave the statement signer to the signature envelope, where
it already lives. Does that split match how you want roles modeled across predicates?

2. Should the salted commitments be mandatory only for private profiles?
commitments is (object, required), with model and dataset each carrying salted. That
makes a privacy-preserving special case the global default. For a public model or a public
benchmark a real content digest is arguably the more useful reference — a plain
ResourceDescriptor OR a commitment. We would keep the commitment form fully specified, it is
the core of the private-model profile, and drop only the obligation for public artifacts.
Reasonable?

3. Generic evidence references instead of the receipt block.
receipt is {schema, merkleRootB64}, which presumes a receipt shaped like ours: a named
schema plus a Merkle root. A predicate in the in-toto directory probably should not presume any
single emitter's shape. A generic evidence[] of ResourceDescriptors — digest, media type,
optional URI — carries the same link and stays tool-neutral, with a Merkle-based receipt then
being one artifact among others. We would make that swap once you confirm the direction.

Appendix — one open alternative rather than a proposal: assuranceLevel.
6fdf5bf already marked it issuer-declared, which removes the worst reading. What stays open is
whether it belongs in v0.1 at all: without shared verification rules, third_party,
reproduced and enclave_attested are not interoperable levels, and corroboration arguably
belongs in referenced evidence with the assessment left to relying-party policy. The options we
see are to drop the field from v0.1, or to keep it optional and issuer-declared as it now is.
No strong position on our side; we would follow your preference.

Two process asks, repeating the ones from two weeks ago rather than adding new ones:
@in-toto/attestation-maintainers the lint workflow is still awaiting approval, so markdownlint
has not run here, and it would help to get this onto the agenda for the next maintainers
meeting.