A signature on a piece of software proves someone with a key signed it. It doesn’t prove much about where that software actually came from: what source it was built from, what process built it, or whether anything happened to it between “code written” and “artifact shipped.” Provenance is the term for closing that gap: a verifiable record tying a specific commit, through a specific build process, to a specific output, in a way that’s checkable rather than just asserted.

This week’s Rabbit Hole Generator pick was Tejolote, a tool for generating exactly that kind of record without needing your build system’s cooperation. Instead of instrumenting the build itself, it watches the build happen from the outside, the way a security camera in a parking lot documents a delivery truck without needing access to the warehouse’s own inventory system. I’d started the sandbox exercise a few weeks back and lost the thread partway through; picking it back up, I skipped the paper exercise Notion had scripted and pointed Tejolote at a real GitHub Actions run instead. It surfaced two real bugs along the way.

What is Tejolote used for in supply-chain security?

Tejolote is an open-source tool from kubernetes-sigs (the Kubernetes project’s incubator for supporting tools) that watches a CI (continuous integration, the systems that automatically build and test code on every change) run happen and generates a signed record of it: what commit was built, what workflow ran, and exactly what came out, down to a cryptographic hash of each file, the kind of fingerprint that changes completely if even one byte of the file is different.

That record is shaped to satisfy SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”), an industry framework, built by Google, the Linux Foundation, and others, for describing how trustworthy a build’s origin story is. SLSA defines a ladder of levels, roughly: Level 0 is no guarantees, you’re trusting the vendor’s word. Level 1 means the build process at least produces a provenance record, even an unverified one. Level 2 means that record is signed and generated by a hosted build service you don’t fully control. Level 3 adds hardening against the build environment itself being tampered with mid-run. It’s a useful way to think about “how much can I actually verify about this build” as a spectrum instead of a yes/no.

Tejolote’s role is to help a CI system clear that first real hurdle, generating an actual SLSA provenance record, without requiring the CI system itself to natively support SLSA. It wraps what it observes into a standard document format called an in-toto Statement (in-toto is a separate, older supply-chain project that Tejolote builds on for its document format), and can optionally sign that statement with Sigstore/cosign, an open-source signing tool that works like the digital equivalent of a notarized signature, so anyone downstream can verify it themselves instead of taking your word for it.

The Architecture: provenance as an outside observer

Tejolote’s design splits cleanly into two pluggable halves. A build-run observer knows how to query a specific CI backend, currently Google Cloud Build and GitHub Actions, for run metadata: steps, timing, triggering event, commit SHA (the unique ID git assigns to a specific commit), workflow file. An artifact collector resolves and hashes the actual outputs from wherever they land: a container registry, a cloud storage bucket, a GitHub Release, or (as I used it) a GitHub Actions native artifact upload. Point Tejolote at a run identifier and it correlates the two: this commit, run by this workflow, produced these exact bytes.

The bet here: you don’t always need to rebuild the machine to get trustworthy signal out of it. An organization with a fragmented CI estate doesn’t have to migrate everything to a SLSA-native build system before it can start generating real provenance. It can bolt an outside observer onto what already exists and get the same evidentiary record, at the cost of trusting that the observer itself saw the whole picture.

The Reality Check: where “watch from the outside” breaks down

I’m calling this approach black-box observation throughout: Tejolote never looks inside the build process itself, it only watches the outside for a run to finish and an artifact to appear, like judging a bakery entirely by what comes out the door. That’s the whole strength of the design and also where its limits come from.

1. It trusts what you point it at

Tejolote explicitly trusts the inputs and artifact locations the operator tells it to watch. There’s no independent verification that the storage location you handed it is actually the output of the run in question; if you point it at the wrong bucket, or a bucket an attacker also has write access to, it will faithfully attest whatever it finds there.

2. Artifact correlation gets ambiguous at scale

In a high-volume pipeline where many builds publish into shared buckets, registries, or directories, output ownership becomes fuzzy. Black-box observation has no built-in concept of “this specific file belongs to this specific run” beyond timing and naming conventions, which erodes as concurrency goes up.

3. Ephemeral build steps can slip past it

Because Tejolote observes from outside rather than instrumenting the build itself, it can’t guarantee it captures every transient dependency fetch, secret mount, or side effect that happens and disappears inside the runner before the observation window catches it.

4. It assumes the build system is telling the truth

Tejolote reconstructs provenance from metadata the build platform (GitHub Actions, Cloud Build) hands back over an API: commit SHA, workflow path, trigger event, and so on. It has to assume that metadata is accurate and behaves consistently. If a build platform’s API returns stale, incomplete, or provider-specific metadata (and different CI systems really do model runs differently under the hood), the resulting provenance record can be quietly wrong in ways Tejolote itself has no way to detect.

5. It assumes artifact locations are the final answer

The tool treats wherever it finds the artifact (a bucket, a registry, an Actions upload) as the authoritative output of the build. Storage locations aren’t actually immutable by default; they can be overwritten after the fact, raced by a second concurrent write, or written to out-of-band by something other than the build. Tejolote has no independent way to confirm the bytes it hashed are the same bytes the build process actually produced, only that those bytes existed at that location when it looked.

6. Introduced Vulnerability

This is the one worth sitting with: a confused-deputy vulnerability, security jargon for “a trusted system does something harmful on an attacker’s behalf because it can’t tell a legitimate request from a malicious one.” If an attacker has write access to the artifact destination Tejolote is watching, they can get it to attest attacker-controlled outputs as if the observed build had legitimately produced them. The signature on the resulting attestation is real. What it’s vouching for isn’t.

The 30-Minute Lab: attesting a real GitHub Actions build

The original Sandbox Challenge doesn’t actually run Tejolote. It has you hand-craft a provenance-shaped JSON file and sign it with cosign directly, simulating what Tejolote’s output looks like without touching the binary. That’s a reasonable 30-minute exercise on its own, but I wanted to know what the real tool does against a real build, so I built it and pointed it at an actual GitHub Actions run instead.

1. Build and stand up a real target

Tejolote doesn’t ship a release binary I could just download, so I built it from source in a throwaway container (golang:1.26-bookworm; the repo’s go.mod currently requires Go ≥ 1.26.2, one version ahead of what a lot of local toolchains still have installed). The container matters for a second reason too: I wanted to run the actual tool against a real target, not just simulate it, and doing that safely means the repo I attest against, the build tooling, and any credentials involved all stay disposable and isolated from my own machine.

I pushed a throwaway public repo with a deliberately trivial workflow that writes a file and uploads it as a build artifact. That upload step matters specifically: Tejolote’s GitHub Actions collector looks for a run’s native artifacts by default (no --artifacts flag needed), and actions/upload-artifact is what makes a file a native GitHub Actions artifact instead of just a file that happened to exist during the job. Skip that step and there’s nothing for Tejolote to find.

name: build
on:
  push:
  workflow_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: build artifact
        run: |
          mkdir -p out
          echo "hello supply chain $(date -u +%s)" > out/app.txt
          sha256sum out/app.txt
      - uses: actions/upload-artifact@v4
        with:
          name: app
          path: out/app.txt

2. Gotcha #1: the obvious on: syntax crashes Tejolote’s YAML parser

My first version of that workflow used the compact list form, on: [push, workflow_dispatch], which is completely valid GitHub Actions YAML. Tejolote’s attest command choked on it immediately:

Error: generating run attestation: building predicate: fetching workflow: fetching workflow:
parsing workflow YAML: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal
array into Go struct field WorkflowData.true of type github.workflowTrigger

That WorkflowData.true is the tell. YAML 1.1 parses an unquoted on: key as the boolean true unless something downstream special-cases it back to the string "on", and Tejolote’s workflow decoder doesn’t handle the array-shorthand trigger form, only the block-map form. Switching to:

on:
  push:
  workflow_dispatch:

fixed it cleanly. Small thing, but it means any repo using the (extremely common) shorthand trigger list will hit this the first time someone tries to attest it.

3. Gotcha #2: a race between “job done” and “run done”

With the workflow fixed, I re-ran the attest command right after the run appeared in gh run list. Tejolote’s default --wait behavior is supposed to poll until the run concludes before collecting artifacts, and it returned successfully, but with an empty result:

level=info msg="Collecting artifacts from actions://algattsm/tejolote-lab/32567492685"
level=info msg="collected 0 subjects from 0 artifacts in run 32567492685"
level=info msg="Run produced 0 artifacts collected from 1 sources"

Checking the run directly explained why: GitHub’s own run-level status/conclusion fields lagged the job’s actual completion by about 15-20 seconds. The single job in the run had already finished successfully, but the run as a whole still reported "status":"in_progress","conclusion":"". Tejolote’s watch loop returned in that gap, before the run-level status caught up to reality, so it saw a run with no attestable state yet. Waiting another 20 seconds and calling tejolote attest again against the exact same run ID picked up the artifact correctly. If your build finishes fast, this race is real and reproducible, not a one-off fluke.

4. The real, verified attestation

Once the run had genuinely settled, Tejolote produced this SLSA v1 provenance statement, unsigned (no --sign flag needed for this exercise), straight from watching the live run:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    {
      "name": "app/app.txt",
      "uri": "https://api.github.com/repos/algattsm/tejolote-lab/actions/artifacts/9474465097/zip#app.txt",
      "digest": {
        "sha256": "e4139096c65ce31ba621dc876989ada2fb9cb7a2ff65746377f68b054cee10f3"
      }
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://actions.github.io/buildtypes/workflow/v1",
      "externalParameters": {
        "entryPoint": ".github/workflows/build.yml",
        "source": "git+https://github.com/algattsm/tejolote-lab@c94a92e77f8e033dcb792f66f2f773fe21e4ce7a",
        "workflow": {
          "path": ".github/workflows/build.yml",
          "repository": "https://github.com/algattsm/tejolote-lab"
        }
      },
      "internalParameters": {
        "github": {
          "event_name": "push",
          "repository_id": "1342696190",
          "repository_owner_id": "56924086",
          "runner_environment": "github-hosted"
        }
      },
      "resolvedDependencies": [
        {
          "digest": { "sha1": "c94a92e77f8e033dcb792f66f2f773fe21e4ce7a" },
          "uri": "git+ssh://github.com/algattsm/tejolote-lab@c94a92e77f8e033dcb792f66f2f773fe21e4ce7a"
        }
      ]
    },
    "runDetails": {
      "builder": {
        "id": "https://github.com/algattsm/tejolote-lab/.github/workflows/build.yml@c94a92e77f8e033dcb792f66f2f773fe21e4ce7a"
      },
      "metadata": {
        "finishedOn": "2026-08-22T10:23:58Z",
        "invocationId": "https://github.com/algattsm/tejolote-lab/actions/runs/32567492685/attempts/1",
        "startedOn": "2026-08-22T10:23:46Z"
      }
    }
  }
}

Here’s the step that actually matters most in this whole lab: I didn’t just read the digest Tejolote put in its own JSON and call it done. I downloaded the artifact myself, separately, and hashed it independently:

$ sha256sum app.txt
e4139096c65ce31ba621dc876989ada2fb9cb7a2ff65746377f68b054cee10f3  app.txt

Why bother, when Tejolote already printed a sha256 in its output? Because a tool telling you “here’s the hash of what I found” is exactly the claim provenance exists to make checkable, not just accept. If Tejolote had a bug in how it computes hashes, or read a stale cached copy of the artifact instead of the real one, or (worse) I’d pointed it at the wrong artifact entirely, its own output would look identical either way: confident, well-formatted JSON with a sha256 field. The only way to catch that class of problem is to compute the same value a second time through a completely separate path (my own sha256sum against the file I downloaded myself) and compare. This is the same logic as Reality Check #4 and #5 above, applied to my own workflow instead of Tejolote’s: don’t trust a claim about an artifact just because it’s formatted like an attestation. Verify it.

That comparison came back an exact match, which is the specific thing that turns this from “a tool that printed JSON” into “a verified record.” Here’s what I actually checked in the attestation to be sure of that, field by field, and why each one matters:

  • subject[0].digest.sha256, the value I just verified independently, is the artifact’s fingerprint. This is the one number the entire attestation is ultimately vouching for. If it doesn’t match the real file, nothing else in the document matters.
  • buildDefinition.externalParameters.source (git+https://github.com/algattsm/tejolote-lab@c94a92e...) ties the attestation to one specific commit by its full SHA, not a branch name or a tag that could later point somewhere else. I checked this against the commit I’d actually pushed; if it had named a different commit, that would mean the attestation is vouching for the wrong build entirely, whether from a bug or (per Reality Check #1) an attacker pointing Tejolote at the wrong thing.
  • buildDefinition.externalParameters.workflow.path identifies exactly which workflow file ran. In a repo with only one workflow this looks redundant, but in a real repo with several, it’s what tells you this attestation is about this release pipeline and not some unrelated CI job that happens to also write to the same repo.
  • runDetails.metadata.startedOn/finishedOn are timestamps I could cross-check against my own terminal history and the GitHub Actions run page. Given Gotcha #2 above, unusually tight or suspicious timing here is worth a second look, not just data to skim past.

The commit SHA, workflow path, repo, and trigger event in the attestation all line up with the real run. Combined with the independently verified digest, that’s the whole promise of the tool delivered: a statement tying one specific commit, through one specific workflow run, to one specific artifact hash, generated by watching from the outside with no changes to the CI system itself, and checkable rather than just asserted.

Where this fits in a real environment

If you’re already on GitHub Actions or Cloud Build and want a first provenance record without re-architecting your pipeline, pointing Tejolote at your existing release workflow is a low-effort way to start. Add --sign and it hands you a Sigstore-signed attestation you can attach to a release or a container image, and that alone gets a build most of the way to SLSA Level 1 or 2 without touching the build itself.

Where it stops being enough: anything with a shared artifact store and real concurrency (see Reality Check #2 and #5 above), anything where you need to prove the build environment itself wasn’t tampered with mid-run (that’s Level 3 territory, out of scope for an outside observer by design), and anything where “the artifact was where I expected it” isn’t a strong enough guarantee for your threat model. For a small project or an internal tool, it’s a genuinely useful stopgap. For a release pipeline feeding external customers, treat it as a first rung on the ladder, not the whole ladder.

Actionable Takeaways

  • Watch for the on: shorthand trap in any tool that parses workflow YAML. on: [a, b] is valid GitHub Actions syntax and will break naive decoders that only expect the block-map form.
  • A completed job isn’t the same signal as a completed run. If you’re polling GitHub Actions status programmatically, build in slack after the last job reports success; the run-level status can lag behind it by tens of seconds.

Black-box provenance is a real, practical stopgap for exactly the fragmented-CI problem it targets, as long as you remember it’s attesting to what it observed, not independently verifying that what it observed was the truth.