This week’s Rabbit Hole Generator pick was ocsfkit, and it’s aimed at a failure mode that doesn’t look like a failure at all: a security event that parses successfully, lands in your SIEM (Security Information and Event Management platform, the system that ingests and correlates security logs), and gets treated as trustworthy, while a real chunk of what it originally meant never made it through the translation.
That translation step is normalization: taking a vendor’s raw event format (AWS GuardDuty’s JSON, Okta’s login logs, whatever a tool emits) and reshaping it into a common schema so different security tools can actually talk about the same event the same way. OCSF (Open Cybersecurity Schema Framework) is the industry’s current attempt at that common schema. The part nobody’s tooling handles well is what happens to the fields that don’t map cleanly. Most pipelines either silently drop them or silently guess, and a downstream analyst has no way to tell the difference between a field that was confidently mapped and one that was quietly invented.
What is ocsfkit used for in OCSF normalization?
ocsfkit is a local-first Python CLI for mapping raw security events into OCSF format while making the mapping’s honesty visible. Instead of just producing valid-looking OCSF JSON, it tags every output field with where it came from: extracted directly from the source, transformed, defaulted to a fallback value, or guessed. It ships commands to map a single event, explain a mapping’s confidence and data loss in detail, lint the result for schema violations, diff two OCSF events by semantic meaning instead of raw text, and gate a batch of events against a minimum trust threshold for CI.
The Architecture: provenance as the product
Most normalization tools treat “did it parse” as the finish line. ocsfkit’s actual bet is that parsing successfully and mapping trustworthily are two different claims, and only the second one should let a pipeline call an event “done.” Its mapping engine runs against a YAML mapping file (or one of several bundled packs for common sources like AWS GuardDuty, Okta, or CrowdStrike) and produces, alongside the normalized output, a full accounting of every target field: which source path it came from, what transform ran, and a confidence score for the whole event based on how much of it was directly mapped versus defaulted or guessed.
That accounting is what the explain, gate, and diff commands are actually built around. explain shows the full breakdown for one event. gate takes a stream of events and a minimum confidence threshold, then fails loudly (non-zero exit, SARIF or JSON output for CI) if the average mapping quality drops below it. diff compares two OCSF events by what they mean, not by raw JSON text, so a genuine severity change shows up distinctly from formatting noise. The design bet is that if you can’t see what a mapping silently threw away, you can’t actually trust the pipeline built on top of it, no matter how clean the output JSON looks.
The Reality Check: where this breaks
Six distinct problems worth naming, drawn straight from what actually strains this kind of tool at real scale, plus one it introduces by design.
1. Mapping maintenance doesn’t scale linearly with source diversity
Every new vendor event type needs its own mapping, and vendor schemas change on their own schedule, not the security team’s. A shop normalizing a handful of sources can hand-maintain packs indefinitely. A shop normalizing thousands of heterogeneous event types is signing up for continuous mapping maintenance that grows with every new integration, not a one-time setup cost.
2. It’s built for review, not for the hot path
Everything about this tool’s design, explain reports, lint passes, gate checks, points at offline or CI-oriented workflows: run it against a sample, review the output, decide whether to trust the mapping. That’s a different job from streaming, low-latency normalization inside a live ingestion pipeline processing events in real time. A team that needs sub-millisecond normalization at wire speed is not the audience this tool is built for, whatever else it does well.
3. The review artifacts themselves become a scale problem
Provenance, explain output, SARIF, coverage reports, and diff output are exactly the value proposition; they’re also volume. Once you’re running this across a large telemetry corpus instead of a two-line sample, deciding what to actually store, review, and act on from that output becomes its own operational question, separate from whether the mapping itself is any good.
4. Confidence scores are a proxy, not a guarantee
The tool assumes its confidence score and unmapped-field count correlate strongly with whether the resulting event is actually trustworthy downstream. That’s a reasonable design bet, and my own lab run below backs it up in this specific case, but it’s still a proxy metric standing in for “would a human analyst trust this event,” not a direct measurement of that. A mapping can score well by this metric while still losing a field a specific detection rule actually needed.
5. Mapping packs assume source schemas hold still
Path-based transforms and mapping packs work as long as a vendor’s event structure stays stable enough to keep the paths valid. Vendors change their JSON shape without much warning. A mapping pack that scored perfectly last quarter can silently start guessing or dropping fields the moment the source vendor ships a schema change, with nothing forcing a re-validation unless someone remembers to run gate again.
6. Introduced Vulnerability
The whole workflow, explain, redact, batch review artifacts, encourages exporting real telemetry into local files, CI artifacts, and HTML/JSON reports so a human can review the mapping. That’s exactly the kind of file that tends to persist outside controlled logging boundaries: checked into a repo by accident, left in a CI artifact store past its retention window, or emailed around for review. The tool ships a redact command for exactly this reason, but redaction is opt-in and easy to skip under deadline pressure, and a review artifact full of account IDs, hostnames, and raw payloads is a real exposure if it ends up somewhere it shouldn’t.
The 30-Minute Lab: does a clean parse still fail a trust gate?
The generator’s challenge asked for a synthetic two-event GuardDuty-like sample, a comparison of what a “good” versus “bad” mapping outcome looks like, and one clear sentence on whether a syntactically valid mapping still fails a trust threshold. I ran the whole thing in a throwaway Docker container.
1. Installation
docker run --rm -it python:3.12-slim bash
pip install ocsfkit
Installed clean, current PyPI release 0.9.0. ocsfkit --help lists roughly thirty subcommands, from map and explain down to benchmark and schema-drift, which tells you this has grown well past a simple format converter.
2. Building the sample, and the fixture that isn’t accidental
cat > sample_guardduty.ndjson <<'EOF'
{"accountId":"111122223333","region":"us-east-1","severity":"High","title":"EC2 instance i-aaa communicating with suspicious host","eventTime":"2025-01-14T10:21:44Z","id":"finding-1","debug":"x","rawPayload":{"foo":"bar"}}
{"accountId":"444455556666","region":"eu-west-1","severity":"Low","title":"IAM user anomalous behavior detected","eventTime":"2025-01-14T11:00:00Z","id":"finding-2","resource":{"instanceDetails":{"instanceType":"t3.micro"}}}
EOF
Each line is NDJSON (newline-delimited JSON, one complete JSON object per line, the standard shape for streaming event data). Neither line is a full real GuardDuty finding; each one deliberately carries a couple of fields the bundled aws-guardduty mapping pack doesn’t know what to do with (debug, rawPayload, id, a nested resource.instanceDetails). That’s the point: a real vendor payload always has more in it than any static mapping pack accounts for, and this fixture is small enough to trace every dropped or unmapped field by hand.
3. Running it, and what the numbers actually mean
ocsfkit explain sample_guardduty.ndjson --pack aws-guardduty
Real output, trimmed to the parts that matter:
─────────────────────────────── Event 1 ────────────────────────────────
Confidence: 0.773
Dropped fields
$.debug
$.rawPayload
Unmapped source fields
$.id
─────────────────────────────── Event 2 ────────────────────────────────
Confidence: 0.622
Dropped fields
$.debug
$.rawPayload
Unmapped source fields
$.id
$.resource
$.resource.instanceDetails
$.resource.instanceDetails.instanceType
Both events mapped without error. Both produced valid OCSF JSON. Confidence is the fraction of the event ocsfkit could account for directly, and it already tells a different story per event: event 2’s nested resource.instanceDetails block has no path in the aws-guardduty pack at all, so its confidence drops to 0.622 versus event 1’s 0.773. A pipeline that only checks “did this parse” would treat both events identically. A pipeline that reads confidence per event would already know event 2 needs a closer look.
Then the actual trust gate, using the Sandbox Challenge’s own thresholds:
ocsfkit gate sample_guardduty.ndjson --pack aws-guardduty --min-confidence 0.70 --max-unmapped 2 --no-strict
Gate: failed
Grade: F
Average confidence: 0.698
Source field coverage: 0.857
Average confidence 0.698 is below 0.700
Unmapped source field count 5 exceeds 2
This is the actual insight the challenge was built to produce, verified for real: a two-event sample that parsed cleanly and produced individually reasonable-looking confidence scores still fails an averaged 0.70/2 trust gate, on both counts. Nothing crashed. Nothing threw an error. The mapping just wasn’t good enough to trust at the threshold this team decided mattered, and without running gate, that fact never surfaces. ocsfkit’s own default threshold is stricter still (0.85 confidence, zero tolerated unmapped fields), and the same sample fails that even harder. Whether 0.70 or 0.85 is the right bar for your own pipeline is a real decision every team has to make deliberately; the tool’s job is just making sure that decision has a number attached to it instead of an assumption.
Where the lab actually broke: the Sandbox Challenge’s next step called ocsfkit batch, and so does the tool’s own current README on GitHub. It doesn’t exist in the installed CLI:
$ ocsfkit batch sample_guardduty.ndjson --pack aws-guardduty --output out.ocsf.ndjson
Error: No such command 'batch'.
I checked the source directly rather than assume the README was just wrong: batch is a real, fully-implemented command on the project’s main branch (src/ocsfkit/batch.py, wired into the CLI at src/ocsfkit/cli.py), it just hasn’t shipped in a PyPI release yet. 0.9.0, the version anyone gets from pip install ocsfkit today, predates it. The lesson isn’t “this tool is broken,” it’s a specific, useful habit: a project’s README describes its main branch, not necessarily the version you just installed, and it’s worth checking which one you’re actually holding before you build a pipeline step around a documented command.
The workaround is two existing commands instead of one: map each version to NDJSON directly, then diff them.
ocsfkit map sample_guardduty.ndjson --pack aws-guardduty --format ndjson > out.ocsf.ndjson
map defaults to a pretty-printed JSON array, not NDJSON, which matters because diff expects one JSON object per line; leaving off --format ndjson here produces a file diff can’t parse. Small thing, easy to hit once, worth knowing up front.
4. Confirming diff catches a real semantic change
cp sample_guardduty.ndjson sample_guardduty_v2.ndjson
# edit event 1's severity from High to Critical
ocsfkit map sample_guardduty_v2.ndjson --pack aws-guardduty --format ndjson > out_v2.ocsf.ndjson
ocsfkit diff out.ocsf.ndjson out_v2.ocsf.ndjson
Event 1 semantic diff
changed severity "High" -> "Critical"
changed severity_id {"id": 4, "name": "High"} -> {"id": 5, "name": "Critical"}
Event 2: no semantic changes
This is the one-line change the challenge asked for, and diff caught exactly it, cleanly, with event 2 correctly reported as unchanged. Worth noting what this proves and what it doesn’t: diff confirms the mapping engine translated the severity change correctly into both the human-readable field and the numeric severity_id OCSF expects. It says nothing about whether the rest of either event is trustworthy; that’s still explain and gate’s job, not this one’s.
Where this fits in a real environment
This earns a real spot in a CI pipeline that gates OCSF mapping changes before they reach a data lake or SIEM: run gate against a fixed sample corpus on every pull request that touches a mapping file, and fail the merge if confidence or unmapped-field counts regress. That’s squarely the review-oriented, offline use case this tool is actually built for, per Reality Check point 2, and the lab above shows the exact kind of regression it would catch (a mapping that parses fine but quietly stops covering a nested field).
It stops being enough the moment you need it to run inside the actual ingestion path instead of a review step (point 2 again), or the moment your source count grows past what a small team can keep mapping packs current for by hand (point 1). And the confidence score it produces is a strong signal, not a certification, per point 4: treat a passing gate as “worth shipping,” not as proof nothing was lost, especially once a vendor changes their schema out from under a pack that used to score cleanly (point 5).
Actionable Takeaways
- A clean parse is not a trust signal. Both sample events mapped without error and still failed the trust gate; check confidence and unmapped-field counts separately from whether the tool exited zero.
- Verify a documented command against the version you actually installed. A project’s README describes
main, not necessarily whatpip installgave you five minutes ago. - Treat mapping review artifacts as sensitive by default.
explain,redact, and batch outputs exist to be shared for review; that’s exactly why they need the same handling as the raw telemetry they’re built from.
Syntactically valid isn’t the same claim as semantically trustworthy, and the gap between those two is exactly where silent data loss lives.
