A Forged Receipt That Verifies: Offline Evidence with proofbundle

A number travels further than the system that produced it. A CI job says a model scored 0.97 on a benchmark, and that 0.97 ends up in a Slack thread, a slide, a vendor questionnaire, an audit binder. By the third hop, nobody can say who produced it or whether it’s the same number the job printed. It’s a screenshot.

The standard fix is a digital signature: the producer signs the result with a private key, and anyone holding the matching public key can check that the bytes haven’t changed and that the key holder signed them. Signing software artifacts is well understood by now; projects like Sigstore do it at scale. But most of that machinery assumes the person checking can reach the internet: a transparency log, a certificate authority, an identity provider. An auditor in a locked-down review room, or an incident responder archiving evidence for a year, may not have any of that.

The Rabbit Hole Generator picked proofbundle this week. It packs a signed result into one JSON file that verifies with no network at all. The point for a security reader isn’t the cryptography, which is a few lines of standard code (I re-check it by hand in lab step 2). The point is the question every signature leaves open: whose key is this, and why should I trust it? proofbundle is unusually blunt about that, and my lab showed exactly where the bluntness matters. A receipt I forged with a key I’d generated ten seconds earlier printed CRYPTO: OK and exited 0. Only a trust policy, a small file I had to write myself, told it apart from the real one.

What is proofbundle used for in supply chain security?

proofbundle is an open-source Python tool and file format for portable, offline-verifiable evidence, aimed first at AI evaluation results. It takes a payload (an eval score, a test report, any JSON), signs those exact bytes with an Ed25519 key, places them in a Merkle tree, and writes everything a verifier needs into a single receipt file. proofbundle verify checks that file locally, with no server, log, or certificate lookup. Its own tagline sets the scope: “Integrity, not truth.” A receipt proves which key signed which bytes, and that they haven’t changed since; it doesn’t prove the number is right. It ships pytest and Inspect AI plugins (Inspect is the UK AI Security Institute’s model-evaluation framework), and it’s MIT-licensed and on PyPI.

The Architecture: signed bytes, a tree, and a file you supply

A few terms first.

• Ed25519 is a digital signature algorithm (RFC 8032). A private key (here, a 32-byte file) produces a signature over some bytes; anyone with the public key can check that signature. Change one byte and the check fails. Ed25519 signatures are also deterministic: the same key and the same bytes always produce the same signature, which comes up in lab step 7.
• A Merkle tree is a tree of hashes. Each leaf is the hash of one item, and each parent is the hash of its children, so the single hash at the top (the root) summarizes every item. An inclusion proof is the short list of sibling hashes that shows one item is in the tree with that root. proofbundle uses the layout from RFC 6962, the same structure Certificate Transparency uses to log every public TLS certificate.
• A relying party is whoever receives the receipt and has to decide whether to act on it: an auditor, a deploy gate, a customer.
• A trust policy is the relying party’s own file saying which signing keys, algorithms, and assurance levels it accepts. proofbundle never fetches one; you hand it over.

Here’s how the pieces connect, drawn from the spec and what the lab actually produced:

The receipt is the payload plus everything needed to check it. emit base64-encodes the payload bytes, signs them, and writes the signature and the public key into the receipt. That’s what makes it self-contained. It’s also the design’s central caveat: a key inside the receipt can only prove the receipt is internally consistent. The project’s own trust-anchor guide calls the bundle key “self-asserting” and says the relying party must pin the expected key out of band, meaning through some separate channel, not from the receipt itself.

The Merkle tree is for collections. For one result, the tree has one leaf and the root is simply the hash of that leaf. The tree earns its place when a producer commits to many items at once, such as every sample in an evaluation, so an auditor can later demand to see a random handful and check each one against the signed root (the project calls this a per-sample audit). The root itself is not covered by the signature, which is why every verify prints a ROOT-AUTHENTICITY: NOT_EVALUATED line unless you supply an expected root you got from somewhere else.

Eval receipts add structure on top. The generic emit command signs whatever bytes you give it. emit-eval takes a structured eval claim (suite, metric, threshold, pass/fail, sample count, issuer, an assurance level) and canonicalizes it first with JCS, the JSON Canonicalization Scheme (RFC 8785), which defines one exact byte layout for a JSON value so two programs serialize it identically. The model and dataset names go in as salted commitments, hashes that hide the name until the issuer chooses to reveal it.

What’s different from the obvious alternatives. Sigstore signs with short-lived certificates tied to an identity (a GitHub Actions workflow, an email) and records every signature in a public transparency log, Rekor, so a verifier usually goes online. in-toto and its DSSE signing envelope define standard attestation formats; the Tejolote post earlier in this series produced one. proofbundle can export to in-toto, but its core bet is narrower: no network, no identity provider, one file, and a verifier output that spells out what it did not check. In the oci-explorer post I found a supply-chain score that awarded points for a signature existing without checking who signed it. proofbundle is the opposite temperament. It prints the gap on every run.

A note on drift, since the series works through a backlog: the Rabbit Hole Generator picked proofbundle on July 10, when the current release was 2.1.0. Today it’s 6.3.1, with 52 releases on PyPI and roughly 1,950 commits since the first one on July 1. Its AGENTS.md describes coding agents proposing changes and the maintainer deciding. The verifier’s output grew new lines along the way, so if you follow an older write-up, expect your terminal to show more than it does. The receipt format itself didn’t move; lab step 7 checks that directly.

The Reality Check: where this breaks

That’s the pitch. Each week the Rabbit Hole Generator also writes a critique of its pick: where the tool fails at scale, which assumptions it quietly depends on, and what security hole it might introduce. For proofbundle there are six distinct points, and I tested all six against the lab below, since none of them needed more than one container and a handful of keys:

• Points 1 to 3 are about what the design can’t do at organizational scale: distribute trust, find things in a pile of receipts, or vouch for the result itself.
• Points 4 and 5 are assumptions it leans on: that verifiers get the right key, and that the bytes stay stable.
• Point 6 is the vulnerability it introduces: what a stolen signing key buys an attacker.

1. Offline verification needs a trust channel it doesn’t provide (tested)

Verifying offline means the verifier can’t ask anyone which key is legitimate. The receipt carries its own public key, so a receipt someone forged with their own key is just as internally consistent as the real one. In lab step 4 I forged one in two commands, and verify printed CRYPTO: OK. A pinned trust policy rejected it (lab step 5), but that policy file has to reach every verifier somehow, stay current, and change when keys rotate. proofbundle defines the file format. Delivering it is your problem.

2. A pile of receipts isn’t a searchable log (tested)

Certificate Transparency works because every certificate lands in a public, append-only log that monitors watch around the clock; a certificate issued for your domain by mistake shows up in hours. proofbundle receipts are separate files. In lab step 9 I emitted 300, and each one became its own one-leaf Merkle tree with its own root. Verification was fine (about 34 ms each, mostly Python start-up), but “which receipts has this key ever signed?” or “has anyone issued a receipt claiming our model scored 0.42?” are questions only a loop over every file you happen to hold can answer.

3. Signing an output says nothing about how it was computed (tested)

A receipt is created after the evaluation finishes, from whatever the evaluation printed. It can’t see whether the eval ran on tampered code, a swapped dataset, or a quietly retried best-of-ten. The project says this itself, repeatedly. The sharpest version showed up in lab step 8: an eval receipt where I simply declared the assurance level reproduced (“independently re-run and matched”) passed a policy that required reproduced, because the level is whatever the signer writes.

4. The creators assume verifiers can pin the right key (tested)

Everything in point 1 rests on the relying party already knowing the genuine public key. In my lab I got it the lazy way: I copied it out of the first receipt I received (lab step 5). That’s trust on first use, the same bet SSH makes the first time you connect to a server, and it’s only safe if that first receipt was genuine. The tooling around the pin is careful: a policy with a typo’d field is refused outright, and policy lint flags a policy that pins nothing. But no tool can tell you the key you pinned belonged to the right party.

5. The creators assume serialization stays stable across versions and wrappers (tested)

A signature covers exact bytes. Reformat the JSON, reorder the keys, change the line endings, and you have a different payload. Lab step 7 found the design mostly sidesteps this: the signed bytes travel inside the receipt as base64, so re-serializing the receipt itself is harmless, duplicate JSON keys are rejected, and a receipt from July’s 2.1.0 was byte-for-byte identical to one from 6.3.1. The real exposure is on the edges. The generic emit signs raw file bytes without canonicalizing them (only emit-eval applies JCS), so a wrapper that pretty-prints a result before signing, or a consumer comparing against its own re-serialized copy, will see a mismatch on data that never changed.

6. Introduced Vulnerability: a stolen key mints perfect receipts, and revocation means re-trusting everything (tested)

If the signing key leaks, the attacker’s receipts are indistinguishable from yours: same key, valid signature, POLICY: OK against your own pinned policy. Lab step 6 did exactly that. The verify command has no way to check whether a key has been revoked; a policy that asks for revocation checking fails closed (refuses, rather than quietly skipping the check) with a message saying so. The working fix is removing the key from every relying party’s policy, which also rejects every genuine receipt that key ever signed. Offline verification can’t tell “signed before the leak” from “signed after” without an independent timestamp.

The 30-Minute Lab: forging, pinning, and stealing receipts

The Rabbit Hole Generator’s challenge was to generate a receipt, tamper with it in two ways, and prove the verifier catches both. I did that, found that one of the two tampers proved less than it looked like, and then kept going to test the Reality Check points: a forgery, a trust policy, a stolen key, serialization changes, eval receipts, and volume. Everything ran in one Docker container with networking switched off, which is the cleanest way to show that verification really is offline.

All the lab files are in this post’s bundle so you can rebuild it: start with README.txt. The full output of the run quoted below is in lab-output.txt.

1. Building the lab

The lab is a single image with proofbundle installed, and a host folder mounted at /lab for inputs and outputs. The challenge said to clone the repo and pip install -e . from main. I installed a pinned release from PyPI instead, because main changed several times a day during this project’s busiest stretches, and a pinned version is the only way your run will match mine:

FROM python:3.12-slim
RUN pip install --no-cache-dir 'proofbundle==6.3.1'
WORKDIR /lab

run.sh builds that image (the only step that needs the network), copies the inputs into a fresh work/ folder, and runs lab.sh inside the container with --network none, which gives the container no network interface except loopback. If any step needed the internet, it would fail.

docker build -q -t pb-lab .
docker run --rm --network none -v "$PWD/work:/lab" pb-lab bash lab.sh

The payload is the challenge’s own result.json, a stand-in for what an evaluation job would write:

{
  "system": "ci-eval-runner-01",
  "model": "demo-model",
  "benchmark": "supply-chain-sanity",
  "score": 0.97,
  "samples": 128,
  "timestamp": "2026-07-10T12:00:00Z"
}

Three small things a reader will hit:

• sh in the python:slim image is dash, not bash. My first run used sh -c with a diff <(...) <(...) comparison and died with Syntax error: "(" unexpected. Process substitution is a bash feature. The lab runs under bash for that reason.
• The eval features come with the plain install. The README says to install proofbundle[eval] for proofbundle demo and eval receipts. On 6.3.1, proofbundle --version already listed features: eval after a plain install, and the demo and emit-eval both worked. Install the extra anyway if you’re following the README; it does no harm.
• The repo’s example policy fails its own strict lint. examples/trust_policy_strict.json has no policyPurpose field, and policy lint --strict now requires one. My policy in lab step 5 adds it.

2. The first receipt, and checking it by hand

emit with --new-key generates a signing key, saves the 32-byte private key to signer.key, and writes the receipt:

proofbundle emit --payload-file result.json --new-key signer.key --out receipt.json
proofbundle verify receipt.json; echo "exit=$?"

[PASS] ed25519-signature: payload signed by stated key
[PASS] merkle-inclusion: anchored at index 0 of 1 (Merkle-consistent under the STATED root)
CRYPTO: OK
ROOT-AUTHENTICITY: NOT_EVALUATED (payload-signature PASS, merkle-consistency PASS, tree-context NOT_EVALUATED, root-trust-level NONE, safe-for-automation false)
SAFE_FOR_AUTOMATION: NO
Reason:
  The Merkle root was not authenticated against a relying-party value (--expected-root or a policy trusted_roots entry)
  ...
  No trust policy was evaluated (supply --policy to authorise a signer)
POLICY: NOT_EVALUATED (no trust policy supplied)
ASSURANCE: n/a (not an eval receipt)
LIMITATIONS: NOT that the result is true, the eval well designed, the model safe/fair, or that the score generalizes (see NON_CLAIMS.md); and NOT when it happened, unless an external time anchor is present
exit=0

This output is worth reading line by line, because it’s the tool telling you what a green result does and doesn’t mean:

• CRYPTO: OK is the only thing actually proven: the signature is valid for the payload, under the key in the receipt, and the payload sits in the stated Merkle tree. Note the wording, “signed by stated key.”
• ROOT-AUTHENTICITY: NOT_EVALUATED says nobody vouched for the Merkle root. For a one-leaf tree that’s harmless. For a big tree, it means you only know the proof is consistent with a root, not the root the producer published.
• SAFE_FOR_AUTOMATION: NO is the line a CI gate should key on, not the exit code. It only turns yes with a passing trust policy that pins the signer, an authenticated root, and no blocking warnings.
• POLICY: NOT_EVALUATED means no trust decision was made. If you log this run as “receipt verified,” you’ve logged authorship as approval.
• LIMITATIONS includes “NOT when it happened.” The timestamp field inside result.json is signed, but signed means the signer wrote it, not that it’s true.

The exit code is 0 anyway. Exit 0 means only “the crypto checked out,” which is exactly the trap Reality Check 1 is about.

The receipt itself is small:

{
  "schema": "proofbundle/v0.1",
  "payload_b64": "ewogICJzeXN0ZW0iOiAiY2ktZXZhbC1ydW5uZXItMDEiLAogICJtb2RlbCI6...fQo=",
  "signature": {
    "alg": "ed25519",
    "public_key_b64": "86XhgteUxZ1Go1E76AeC187W5Q9ElQ/frwsFF9A5Vkc=",
    "sig_b64": "Wxdea8f4dgHRFsw0SVcSLVl/ftbAUPz6o8jHSPdTTZOq5AkBDBZe0BTSNoSANoxG42aeL1zZXRjhzHUdviRQDQ=="
  },
  "merkle": {
    "hash_alg": "sha256-rfc6962",
    "leaf_index": 0,
    "tree_size": 1,
    "inclusion_proof_b64": [],
    "root_b64": "R+0CMZ/J7jC35+uvGG8JWBbhyDiedG7faBLTkpWbdag="
  }
}

payload_b64 is the exact bytes of result.json, trailing newline included. public_key_b64 is the field everything else hinges on: it’s the key the signature is checked against, and it came from the same file. tree_size: 1 with an empty inclusion_proof_b64 confirms the one-leaf case.

I wanted to know how much of verify is cryptography and how much is policy, so I re-checked the receipt without any proofbundle code, using only Python’s hashlib and the cryptography library (check_by_hand.py):

payload = base64.b64decode(r["payload_b64"])
pub = base64.b64decode(r["signature"]["public_key_b64"])
sig = base64.b64decode(r["signature"]["sig_b64"])

Ed25519PublicKey.from_public_bytes(pub).verify(sig, payload)   # raises if invalid
leaf = base64.b64encode(hashlib.sha256(b"\x00" + payload).digest()).decode()
print("merkle root matches leaf hash:", leaf == r["merkle"]["root_b64"])

signature: valid for the key IN the receipt
merkle root matches leaf hash: True
signer key: 86XhgteUxZ1Go1E76AeC187W5Q9ElQ/frwsFF9A5Vkc=

The 0x00 prefix is RFC 6962’s marker for a leaf (interior nodes use 0x01), which stops anyone from passing off an interior node as a leaf. That’s the whole core check, in about ten lines. Doing it independently was worth it for two reasons. It confirms the receipt is a plain, standard construction that any language can verify, which matters for evidence you might need to check in five years. And it makes the real question obvious: the code has to read the public key from somewhere, and the only place available is the receipt.

3. The two tampers from the challenge

Tamper 1: change the result on disk. The challenge edits result.json to say score: 0.11 and re-verifies the unchanged receipt:

exit=0

Still verifies, as the challenge predicted. The receipt carries its own copy of the bytes and never looks at the file on disk. So if someone hands you a receipt and a results file, the receipt passing tells you nothing about the file. To tie them together, compare hashes yourself:

result.json          MATCH receipt 21917f5603e28904 file 21917f5603e28904
result_tampered.json MISMATCH receipt 21917f5603e28904 file 2c47c5d96dedc161

That’s the SHA-256 of the receipt’s decoded payload next to the SHA-256 of each file. A match means the file is byte-for-byte what was signed; a mismatch means someone changed it, or reformatted it (lab step 7 shows why those look the same).

Tamper 2: change the receipt. The challenge’s script (tamper_generic.py) walks the receipt and mutates the first plain value it finds:

<     "schema": "proofbundle/v0.1",
---
>     "schema": "proofbundle/v0.1-tampered",
ERROR: unsupported schema 'proofbundle/v0.1-tampered', expected 'proofbundle/v0.1'
exit=2

It failed, so the success criterion was technically met. But look at what it changed: the schema label, which is the first key in the file. The verifier rejected the file as malformed (exit 2) before checking any cryptography. That doesn’t prove the signature catches tampering; it proves the parser catches typos. A tamper test should change the thing that matters, so I edited the score inside the signed payload and left the signature alone (tamper_payload.py):

edited signed payload: score 0.97 -> 0.99
[FAIL] ed25519-signature: invalid signature
[FAIL] merkle-inclusion: inclusion proof failed
CRYPTO: FAILED
exit=1

That’s the real catch: exit 1, a signature failure, and the Merkle check fails too because the leaf hash changed. The exit codes are worth knowing, because a script that only checks “non-zero” will treat a typo and a forgery the same way:

4. Forging a receipt that verifies

This is the test the challenge didn’t ask for and the one the whole tool turns on. Instead of editing the genuine receipt, an attacker writes a fake result and signs it with their own brand-new key:

proofbundle emit --payload-file result_forged.json --new-key attacker.key --out receipt_forged.json
proofbundle verify receipt_forged.json

[PASS] ed25519-signature: payload signed by stated key
[PASS] merkle-inclusion: anchored at index 0 of 1 (Merkle-consistent under the STATED root)
CRYPTO: OK
exit=0

signature: valid for the key IN the receipt
merkle root matches leaf hash: True
signer key: OGTdE83qkiCqbmNhkYp/8AyjKgImRHHIiPHa+BALduw=

A fake score of 0.11, CRYPTO: OK, exit 0, and my independent check agrees. The only visible difference from the genuine receipt is the signer key: OGTd... instead of 86Xh.... This isn’t a bug, and proofbundle’s built-in demo doesn’t hide it either; its tamper matrix tests swapping in an attacker’s key without re-signing, which fails, and the docs are explicit that a full re-sign is the relying party’s job to catch. But it’s the failure mode a hurried integration will hit: a deploy gate that runs proofbundle verify and checks for exit 0 accepts this forgery. That’s Reality Check 1.

5. Pinning the signer with a trust policy

The fix is to tell the verifier which key you trust. make_policy.py writes a trust policy that pins the genuine key:

{
  "schema": "proofbundle/trust-policy/v0.1",
  "policy_id": "lab-pin-ci-eval-runner-01",
  "policyPurpose": "eval",
  "allowed_schema_versions": ["proofbundle/v0.1"],
  "allowed_issuers": [
    {"issuer": "ci-eval-runner-01", "public_key_b64": "86XhgteUxZ1Go1E76AeC187W5Q9ElQ/frwsFF9A5Vkc=", "kid": "lab-2026"}
  ],
  "signature": {"allowed_algs": ["ed25519"], "require_expected_signer": true},
  "merkle": {"required_hash_alg": "sha256-rfc6962"}
}

How the pieces fit. Each field pins one thing; anything you leave out simply isn’t checked:

The full field list is in the project’s trust-anchor guide. Before trusting a policy, the tool can tell you what it actually pins:

policy   lab-pin-ci-eval-runner-01  (proofbundle/trust-policy/v0.1)
  pins   policyPurpose == 'eval' (wrong verifier path fails)
  pins   schema version in ['proofbundle/v0.1']
  pins   issuer ci-eval-runner-01: public key pinned (86XhgteUxZ1G…)
  pins   signature alg in ['ed25519']
  pins   signer MUST match an allowed_issuers entry (require_expected_signer)
  pins   merkle.hash_alg == 'sha256-rfc6962'
[policy-lint] PASS · 6 pin(s) · 0 error(s) · 0 warning(s)

Then the two receipts, with the policy:

--- genuine
CRYPTO: OK
POLICY: OK
exit=0
--- forged
CRYPTO: OK
POLICY: FAIL (signer public key is NOT in allowed_issuers)
exit=3

Both receipts are cryptographically perfect. Only the policy separates them, and it does so with a distinct exit code, 3, so a script can tell “forged or unknown signer” from “broken file.” The fields to key on are the POLICY: line and the exit code, not CRYPTO:.

I also checked what a typo does, because a misspelled field that silently disabled the issuer check would be worse than no policy:

ERROR: unknown field(s) in trust policy: ["'alowed_issuers'"] (trust policy is fail-closed)
exit=2

Refused outright. Good.

Now the honest part. Where did I get 86Xh... to pin? I read it out of the genuine receipt. In the lab I know which receipt was genuine because I made it. A real relying party pinning the key from the first receipt it receives is doing trust on first use, which is Reality Check 4. In a real deployment the key should arrive by a different path than the receipts do: published in the producer’s repository or security page, exchanged during vendor onboarding, or distributed with your configuration management. A realistic policy also lists more than one issuer, say the CI runner’s current key plus next quarter’s key during a rotation overlap, each with its own kid, so a rotation doesn’t break every verifier on the same day.

6. Stealing the key

Reality Check 6 says a stolen key is the worst case. The lab version: the attacker has a copy of signer.key and uses it on a fake result.

proofbundle emit --payload-file result_stolen.json --key signer.key --out receipt_stolen.json
proofbundle verify receipt_stolen.json --policy trust_policy.json

CRYPTO: OK
POLICY: OK
exit=0

Indistinguishable from the real thing, by design: the policy trusts that key, and that key signed it. So what does revocation look like? The policy schema has a status section for revocation lists, and turning it on gives a clear answer:

POLICY: FAIL (policy enables a status requirement, but verify --policy (v0.1) has no status-snapshot input — evaluate revocation separately with verify_status_snapshot (fail-closed))
exit=3

The CLI can’t consume a revocation list yet, and rather than pretend to check, it fails closed and points you at a separate Python function. That leaves the blunt fix: generate a new key, and replace the old one in every relying party’s policy. I did that and re-checked all three receipts against the rotated policy:

receipt.json         POLICY: FAIL (signer public key is NOT in allowed_issuers)
receipt_stolen.json  POLICY: FAIL (signer public key is NOT in allowed_issuers)
receipt_v2.json      POLICY: OK

The forged receipt is gone, and so is the genuine one. Every receipt the old key ever signed now fails, because offline verification has no way to know which were signed before the leak. The project’s answer is external time anchors: an independent timestamp, such as an RFC 3161 timestamp authority’s signature, proving a receipt existed before a given moment. That path is marked experimental, and I didn’t test it. Practically: keep the key file somewhere a compromised CI job can’t read it (the plugins load it from a path in PROOFBUNDLE_KEY, so scope that secret tightly), keep the old genuine results so you can re-sign them under the new key, and decide in advance how a rotation reaches every verifier.

7. Serialization, duplicate keys, and July versus October

Reality Check 5 worried that byte layout would drift. Three checks.

Re-serialize the receipt. I minified it and reversed the key order:

--- receipt minified + keys reordered
CRYPTO: OK
exit=0

Fine, because the signed bytes are inside payload_b64 and base64 doesn’t care how the outer JSON is formatted.

Re-serialize the payload. I minified result.json and compared it to what was signed:

same bytes: False | same JSON value: True

Same data, different bytes, so a hash check like the one in lab step 3 reports a mismatch. This is the real risk: the generic emit signs raw bytes and doesn’t canonicalize them (the Rabbit Hole Generator’s summary said it does; that’s only true of emit-eval, which uses JCS). If your pipeline signs a pretty-printed file and a consumer compares against a compact copy, every check fails on data nobody changed. Sign the file you’ll actually ship, or sign through emit-eval.

Duplicate keys. A JSON object with the same key twice is a classic way to make two parsers disagree; some keep the first value, some keep the last. I added a second payload_b64 to the receipt:

ERROR: duplicate JSON key 'payload_b64' — rejected fail-closed (a duplicated key parses differently across JSON implementations; parser-differential guard, WP-C1)
exit=2

Rejected before anything else runs.

Across versions. Given how fast the project moves, I built a second image with 2.1.0, the release that was current when the Rabbit Hole Generator picked it (Dockerfile.v210, cross_version.sh), and signed the same payload with the same key in both:

same bytes from both versions: yes
--- 2.1.0 receipt, verified by 6.3.1
CRYPTO: OK
exit=0
--- 6.3.1 receipt, verified by 2.1.0
[PASS] ed25519-signature: payload signed by stated key
[PASS] merkle-inclusion: anchored at index 0 of 1
CRYPTO: OK
POLICY: NOT_EVALUATED (no trust policy supplied)
...
exit=0

Byte-identical receipts, and each version verifies the other’s. That’s Ed25519’s determinism plus a format that genuinely hasn’t changed (both versions report spec-revision: proofbundle/v0.1). The 2.1.0 output also shows the drift I mentioned earlier: no ROOT-AUTHENTICITY or SAFE_FOR_AUTOMATION lines yet. Over four months and dozens of releases, the verifier got more explicit; the format stayed put.

8. Eval receipts and the assurance level

The generic receipt signs any bytes. For AI evaluations, proofbundle has a structured claim. make_claim.py builds one with the library’s own helper, so the pass/fail is computed rather than typed in:

claim, salts = build_eval_claim(
    suite="supply-chain-sanity", suite_version="1.0", metric="accuracy",
    comparator=">=", threshold="0.90", score="0.97", n=128,
    model_id="demo-model", dataset_id="supply-chain-sanity-v1",
    issuer=issuer_fingerprint(signer), timestamp="2026-07-10T12:00:00Z",
    assurance_level=level,
)

{
  "schema": "proofbundle/eval-claim/v0.1",
  "suite": "supply-chain-sanity",
  "metric": "accuracy",
  "comparator": ">=",
  "threshold": "0.90",
  "passed": true,
  "n": 128,
  "model_id_commit": "sha256:d5be1fb9097c891c7542b1d492c8cf30d61ba13a516a319e94e26ed724b92e6c",
  "dataset_id_commit": "sha256:9113ff32abb66fe933a1389337736adcb4797cb5afdcc7072f1db8d5e3a69611",
  "issuer": "ed25519:86XhgteUxZ1Go1E76AeC187W5Q9ElQ/frwsFF9A5Vkc=",
  "assurance_level": "self_attested",
  ...
}

Two things to notice. The score 0.97 isn’t in the claim; only passed: true against the 0.90 threshold is. The exact score is used to compute the verdict and then dropped, and show-eval says so. And the model and dataset names are hashes (model_id_commit), salted so nobody can guess them by hashing a list of known model names; the salts stay with the issuer, who can reveal them later. The issuer field is inside the signed claim, so a receipt can’t be re-signed under a different key while keeping the original issuer name.

proofbundle emit-eval --claim claim_self_attested.json --key signer.key --out eval_receipt.json
proofbundle show-eval eval_receipt.json

suite      supply-chain-sanity (1.0)
metric     accuracy >= 0.90
passed     True   (n=128)
evidence   THRESHOLD_VERDICT_VERIFIED (proves `passed` against the signed threshold, not an exact score)
note       METHODOLOGY_NOT_EVALUATED (the receipt never judges whether the suite is well designed)
assurance  self_attested
...
WARNING    self_attested with no prereg_sha256 — the weakest assurance: trust rests entirely on the issuer, who could publish the best of many runs. Pre-register (prereg_sha256) or use a higher assurance_level (reproduced / enclave_attested) to strengthen it.
=> OK

The lines to read: evidence says what was proven (a threshold verdict), note says the benchmark design wasn’t judged, and the WARNING names a real attack: run the eval ten times, publish the best. The defence it suggests, pre-registration, means committing to a hash of the evaluation protocol before the run, so the issuer can’t quietly change the plan afterward.

The assurance levels run from self_attested (the issuer ran it) through third_party and reproduced to enclave_attested (run inside attested hardware). So I made a second claim, same key, same self-run eval, and simply declared it reproduced:

assurance  reproduced
=> OK

The warning vanished. Then I verified both against a policy requiring at least reproduced:

--- eval_receipt.json
CRYPTO: OK
POLICY: FAIL (assurance_level 'self_attested' below minimum 'reproduced')
ASSURANCE: self_attested (issuer-declared)
exit=3
--- eval_receipt_reproduced.json
CRYPTO: OK
POLICY: OK
ASSURANCE: reproduced (issuer-declared)
exit=0

The honest self-attested receipt fails; the one where I lied passes. That’s Reality Check 3 in two lines, and the tool isn’t hiding it: (issuer-declared) is printed right there, and the threat model spells out that “a dishonest issuer can sign reproduced on a self-run eval.” A minimum assurance level in a policy only means something if the key you pinned belongs to someone who would actually do the reproducing, such as a separate team or an outside lab with its own key. Pin their key for reproduced claims, not the producer’s.

9. Volume: 300 receipts

For Reality Check 2, I emitted 300 small receipts with one key and verified all of them against the policy, one CLI call each:

emit 300: 9.9s   verify 300: 10.3s   failures: 0
receipts: 300 | distinct Merkle roots: 300 | tree sizes: [1]

About 34 ms per verify, nearly all of it Python starting up; speed isn’t the problem. The second line is. Each emit call builds its own one-leaf tree, so 300 receipts means 300 unrelated roots. Nothing ties them into one history, and nothing watches for a receipt you didn’t issue. (The library’s emit_eval_receipt function can add a receipt to a tree that already holds earlier leaves, and eval claims can carry a separate signed tree of per-sample hashes; the CLI emit does neither.) If you need to answer “what has this key signed?”, you’ll be building an index yourself, or pairing proofbundle with a real transparency log. The project maps its format onto SCITT, the IETF working group building exactly that kind of log for supply-chain statements, but that’s a mapping, not something the CLI does.

Where this fits in a real environment

proofbundle is a good fit when evidence has to outlive the system that made it, or travel somewhere that system can’t reach. Attaching a signed receipt to an eval result, a test run, or a scan verdict, and archiving it alongside the release, gives an auditor or incident responder something better than a screenshot, verifiable years later on an offline laptop with ten lines of standard code (lab step 2). The pytest and Inspect AI plugins make that a few environment variables, and the verifier’s output is the most careful I’ve seen about separating “the bytes are intact” from “you should believe this.”

It stops being enough exactly where trust has to be shared. It doesn’t distribute keys (Reality Check 1, 4), so before adopting it, decide how relying parties get the public key and the policy, and how a rotation reaches them. It doesn’t revoke (6), so protect the signing key like any release-signing key. It doesn’t index or monitor receipts (2), so if you need to discover rogue receipts, you need a log as well, which is where Sigstore’s Rekor or a SCITT service comes in. And it never vouches for the result (3), so assurance levels only mean something when a different key, held by a different party, makes the stronger claim. If your verifiers are always online and your artifacts are container images, Sigstore already does most of this with identity built in; proofbundle’s niche is the offline, cross-organization, long-retention case.

Actionable Takeaways

• Never gate on verify exiting 0. A receipt I forged with a fresh key exited 0; gate on --policy, exit 3, and SAFE_FOR_AUTOMATION.
• Pin the signer’s key from a separate channel. Copying it from the first receipt you receive is trust on first use.
• Test tampering against the signed bytes. The challenge’s generic tamper changed the schema label and only proved the parser works.
• Sign the exact file you’ll ship. The generic emit doesn’t canonicalize, so reformatted JSON reads as tampered.
• Plan key rotation before you need it. A stolen key passes your policy, and removing it invalidates every genuine receipt it signed.
• Read ASSURANCE as a claim, not a fact. Require stronger levels only under keys held by whoever actually did the reproducing.

A signature tells you who said something and that it hasn’t changed since; deciding whether that “who” deserves your trust is a file you have to write yourself.