Patching a production database is the most privileged routine change most teams make. Someone stops the database, rewrites the software it runs on, restarts it, and runs SQL against the data dictionary, usually in a window at 2 AM. The decision to do it lives somewhere else entirely: a change ticket, a meeting, a “yes, go ahead” in chat. Nothing technical connects the thing that was approved to the thing that actually runs. If the patch file changed since Tuesday, the target is a different server, or the backup the change board was promised failed last night, the 2 AM operator usually finds out afterward, if at all.

Oracle makes that gap feel smaller every quarter. Oracle’s Critical Patch Update ships on a fixed quarterly calendar (Oracle Security Alerts), and the database Release Update arrives on the same rhythm, so the time between “this fixes something serious” and “this is in production” keeps getting squeezed.

The Rabbit Hole Generator picked Oracle Patch Guard (OPG) this week. It is a set of shell and Python scripts that treats a database patch as a three-step transaction: PLAN records exactly what was checked, APPROVE has a person sign that record, and APPLY refuses to touch anything unless the signed record still matches the machine in front of it. The point for a security or infrastructure reader isn’t Oracle. It’s the pattern: change approval as a signed artifact bound to facts, re-checked immediately before the risky step, instead of a ticket number. I ran OPG’s own fake-Oracle test harness in containers, tried seven ways to tamper with an approved run, and every one was blocked before a database stopped. The design holds up. I did find one nuance worth a footnote: an environment variable that, on a look-alike server, lets an approval signed for one host apply on another. It isn’t a break-in (it needs the oracle account, which already controls the database), but it’s the kind of edge that’s easy to miss in a tool whose whole value is that the signature means what it says.


What is Oracle Patch Guard used for in database patch management?

Oracle Patch Guard is an open-source framework for patching single-instance Oracle Database 19c homes with an approval gate. It wraps Oracle’s own patch tools (OPatch for the binaries, datapatch for the SQL side) in a lifecycle of ASSESS, PLAN, APPROVE, and APPLY. PLAN writes a manifest that pins the target host, the Oracle software directory, the patch media’s hashes, and the database’s current state; APPROVE signs that manifest and a separate approval token with RSA; APPLY takes a lock, verifies both signatures and every binding, re-runs the time-sensitive safety checks, and only then starts downtime. It deliberately refuses clustered and replicated setups (RAC, ASM/Grid, Data Guard), and the README and most of the docs are in Dutch.


The Architecture: prepare early, decide separately, re-check late

Oracle databases come with a lot of vocabulary, so here is the minimum you need for the rest of the post:

  • An Oracle Home is the directory the database software is installed in (for example /u01/app/oracle/product/19c/dbhome_1). Patching rewrites files in it. Several databases can share one home.
  • A SID is the name of one running database instance, and oratab (/etc/oratab) is the plain-text file that maps each SID to its home and says whether it should start at boot.
  • A Release Update (RU) is Oracle’s quarterly bundle of fixes for a release. Oracle’s patch maintenance guide says RUs ship on the third Tuesday of January, April, July, and October. The OJVM patch is a separate quarterly patch for the Java engine inside the database.
  • OPatch is Oracle’s command-line tool that applies a patch to the files in a home. datapatch is the follow-up step that applies the SQL part of the patch inside each database once it’s running again, and utlrp.sql recompiles anything the patch invalidated.
  • A CDB (container database) can hold several PDBs (pluggable databases), each a self-contained database; OPG validates datapatch per container (Oracle’s multitenant overview). RAC (clustered databases), ASM/Grid (Oracle’s storage and cluster layer), and Data Guard (standby replicas, see Oracle’s overview) are the setups OPG refuses.
  • OEM, Oracle Enterprise Manager, is Oracle’s fleet management console. OPG ships small wrappers so OEM jobs can drive each phase.

Here is how the pieces connect. I drew it from the code at commit fb86948 and the artifacts my lab runs produced:

Diagram of Oracle Patch Guard’s approval chain in three columns. On the target host, ASSESS plus PLAN runs a fail-closed preflight and writes patch_manifest.json binding hostname, target_oracle_home, the DB and OJVM patch numbers and month, the patch tree SHA-256 hashes, the OPatch zip hash, the oratab hash, the database state and registry baselines, the recovery manifest hash, and the approval public key hash, plus findings.psv and a local patch_manifest.sha256 sidecar. On the signer host, APPROVE writes approval.json with approved true, manifest_sha256, hostname, target_oracle_home, expires_epoch, and an accept entry for every CONDITIONAL finding, then two RSA signatures; the single-run signer is site-provided and not in the repo. Back on the target host in the maintenance window, APPLY takes a flock on a hash of host and Oracle Home, verifies the approval, re-runs the pre-apply checks, and only then starts downtime. A red box says any mismatch ends the run as BLOCKED with exit 20 before a database or listener stops, and that hostname comes from hostname -f or from OPG_HOSTNAME_OVERRIDE. An amber box shows the privileged edge: root helpers reached through fixed sudoers entries for media staging and context publication, which is where the path-substitution critique aims.

Four design bets explain almost everything OPG does:

1. Preparation, permission, and mutation are separate steps, ideally done by different people. Security folks call this separation of duties (NIST’s definition): no single person can both prepare and authorize a sensitive action. The DBA’s run produces a plan; somebody holding the approval key signs it; the APPLY step only checks signatures. The key that signs never needs to be on the database host, and the database host only holds the public half.

2. Approval is bound to facts by hash, not by description. A change ticket says “apply the July RU to db01.” OPG’s manifest says “this host name, this exact home path, these patch trees whose files hash to 7962b3b9..., this oratab whose bytes hash to 6c4d6eff..., this list of databases, this public key.” A hash (here SHA-256) is a fixed-length fingerprint of some bytes; change one byte and the fingerprint changes completely. The approval token then carries the hash of the whole manifest, and both files get signed. Change any bound fact and something stops matching.

3. The expensive checks run early, and the volatile ones run again at the last moment. This targets a classic bug class called time-of-check to time-of-use, or TOCTOU (MITRE CWE-367): you verify something, time passes, the thing changes, and you act on the stale verification. A plan approved on Monday says nothing about whether a backup failed on Wednesday night. So APPLY re-runs the backup check, the maintenance-window check, the “is a Data Pump export running?” check (Oracle’s own guide warns that patching during a running Data Pump job risks logical corruption), free space, patch conflicts, and re-hashes the patch media, all before it stops anything.

4. Everything fails closed. Fail closed means that when a check can’t give a confident answer, the answer is “no.” Every finding is READY, CONDITIONAL (allowed only if the approval token explicitly names it), BLOCKED, or UNKNOWN, and UNKNOWN blocks just like BLOCKED. In the lab, a run that was blocked once at the approval step stayed blocked: rerunning APPLY on it returned BLOCKED again until I made a fresh PLAN.

How the pieces fit on a real host. Everything site-specific lives in one file, /etc/oracle-patch-guard/patchGD_guard.conf, copied from project/patchGD_guard.conf.example. OPG sources it as shell code, so in production it refuses to load unless root owns it and no group or other user can write it. The fields that decide the security story:

Setting What it controls
PATCH_ROOT, OPATCH_ROOT Where the downloaded patch media sits, usually a shared drive (/mnt/patch-share/... in the example)
LOCAL_MEDIA_MODE=required, LOCAL_STAGE_ROOT Copy media to a root-owned local stage under /u01/stage and V2-hash it there; production refuses any other mode
RUN_ROOT, LOCK_ROOT Per-run evidence (/var/log/oracle-patch-guard) and the APPLY lock directory
APPROVAL_PUBLIC_KEY The only key whose signatures APPLY accepts; its hash is pinned into each manifest
BACKUP_CHECK_COMMAND, MAINTENANCE_WINDOW_CHECK_COMMAND, DATAPUMP_CHECK_COMMAND, DATAGUARD_CHECK_COMMAND Site scripts OPG runs at ASSESS and again before APPLY; exit 0 is READY, 2 BLOCKED, 3 UNKNOWN
RECOVERY_BASE_IMAGE, RECOVERY_BASE_IMAGE_SHA256, HOME_RECOVERY_PROCEDURE The fallback if a patch breaks the home: a known-good install image (by hash) and a written rebuild procedure
ASSESSMENT_MAX_AGE_MINUTES How stale an assessment may be before APPLY refuses it (60 in the example)
MAX_PARALLEL_TARGETS, STOP_NEXT_WAVE_ON_FAILURE Wave settings for multi-target OEM runs
ALLOW_TEST_MODE=false Whether the fake-Oracle test mode is allowed at all

The check commands are the extension point. If your shop has its own “is the backup good?” or “is this inside the approved window?” logic, you point OPG at a script that prints its evidence and exits 0, 2, or 3, and OPG records the output in the run directory and blocks on anything but 0. The value has to be an absolute path to an executable with no arguments (OPG never evals it), and leaving a required check empty doesn’t skip it: OPG records NOT_CONFIGURED and treats it as UNKNOWN. A one-time bootstrap (oem-tasks/opg_bootstrap_host.sh) installs the root helpers, the sudoers fragment, and the stage directories with the expected owners; after that, the DBA account runs patchGD_guard.sh and never needs root directly.

One more piece needs explaining before the critique, because it’s where most of the security engineering lives: the tree hash. A patch arrives as a ZIP file, but OPatch works on the unpacked directory, so OPG needs a fingerprint of a whole directory tree, not one file. The newer format, OPG_TREE_HASH_V2 (specified in TREE_HASH_V2_SPEC.md), walks the tree, hashes each regular file, and combines the relative path, size, and content hash of every file in a fixed byte order into one SHA-256. It refuses symlinks, hard links, and special files outright, opens files with O_NOFOLLOW (an open(2) flag that refuses to follow a symlink at the last path component), and compares each file’s metadata before and after reading so a file swapped mid-hash is caught. In production, OPG copies the media from a shared drive into a root-owned local stage, hashes it there, and patches from that local copy. The older V1 format is a find | sha256sum pipeline; production mode now refuses it, and lab step 6 shows why.


The Reality Check: where this breaks

That’s the pitch, and it’s a good one. Each week the Rabbit Hole Generator also writes a critique of its pick: three ways the tool fails at scale, two assumptions its creators make that may not hold in production, and one security weakness it likely introduces. This pick was generated in late August, and the series works through a backlog, so OPG had gained 47 commits by the time I ran it in November. Where that drift changed a critique, I say so.

  • Points 1 to 3 are about scale. I tested what a laptop can honestly show: the lock’s scope (1), hashing cost on a realistically sized tree (2), and what’s actually in the repo for signing (3). A hundred-host fleet with a shared patch drive isn’t something I can fake.
  • Points 4 and 5 are assumptions. I tested a narrow but real slice of 4, and read and ran the backup check behind 5.
  • Point 6 is the introduced vulnerability. I tested the defenses against it, and they mostly hold.

1. Fleet concurrency has no global coordinator (partly tested)

The critique says run state lives in files, locks are per Oracle Home, and nothing provides distributed transactions or fleet-wide backpressure (slowing new work down when the shared pieces are saturated). That matches the code. The APPLY lock is a flock (a kernel-managed advisory lock on a file) on a file named after a hash of host name and Oracle Home, inside a local, root-owned lock directory. Lab step 6 shows it working exactly as scoped: a second run on the same home is refused with exit 60, and a run on a different home on the same host proceeds. Nothing in it can see another host. Fleet scheduling is left to OEM, and the README’s own evidence of parallelism is a two-host OEM run on September 10 that took about 22 minutes per host. Whether OEM plus a shared patch drive behaves at 200 hosts is exactly the part I can’t test on one Mac, so I won’t pretend to.

2. Big patch trees mean a lot of hashing (tested)

OPG hashes the patch trees at ASSESS and again in the pre-apply recheck (two passes per tree per run in my lab logs), and V2 adds a second directory walk and per-file metadata checks. The critique says that turns into real maintenance-window I/O. On a single host with local disk, it doesn’t: lab step 7 hashed about 2 GB in 20,000 files in roughly 1.6 seconds warm and 3.3 seconds with the page cache dropped. The real cost in a fleet is elsewhere: copying media from a shared drive to every host’s local stage, all at once. That’s a network and storage problem, and the same one any patching tool has.

3. The signer is a bottleneck you have to build (checked, still true)

The critique says the single-run signer that actually writes approvals is site-provided and not in the repo. That’s still true in November: signer/opg_approve_pending.sh lists runs that are ready and calls an external opg_approve_run.sh once per run, but that script isn’t published. What changed since August is everything around it. A November 4 commit added signer-side cycle preparation, and the batch wrapper asks for one “yes” and then signs each selected run separately. That’s the right shape for 50 runs on patch night, but it also means one approver confirming 50 runs at once, which is where approval fatigue lives. The batch and pending-run suites passed in my lab (lab step 2) against a stubbed signer, which is all I could test.

4. The creators assume the host stays trustworthy between PLAN and APPLY (narrow slice tested)

A signature proves the plan wasn’t changed. It can’t prove the machine reporting facts to the verifier is honest. If root or the kernel on the target is compromised, every check OPG runs is answered by the attacker. I can’t test a compromised kernel honestly in a lab, and no patching tool solves that one. But I found a smaller version of the same assumption that I could test. The host binding (the hostname field in both the manifest and the token) is compared against hostname -f, unless an environment variable called OPG_HOSTNAME_OVERRIDE is set, in which case OPG uses that value instead. That override isn’t limited to test mode. Lab step 8 copies a signed approval from container host-a to container host-b, sets the variable, and APPLY completes on host-b, writing evidence that says host-a. It also shows a one-line fix.

5. The creators assume “a backup ran” means “we can restore” (tested by reading and running the check)

OPG’s backup check (project/checks/check_rman_backup) asks RMAN (Oracle’s backup tool) and the database for the backup configuration and the age of the last full, database, and archive-log backups, and blocks if the backup target is wrong or a backup is too old. Its own header says it is read-only: no CROSSCHECK, which is the RMAN command that confirms backup pieces still exist on the backup storage. Nothing in it runs RESTORE ... VALIDATE, the RMAN command that actually reads the backup pieces and confirms they’re usable (Oracle’s docs on validating backups). In the lab, its open-checks suite passed all 42 cases, including “wrong backup host is BLOCKED” and “ORA error on stderr is UNKNOWN,” against a fake rman that just prints configuration lines. So the critique holds: OPG proves a recent backup is recorded, not that it restores. Whether that’s enough depends on whether something else in your shop restore-tests on a schedule.

6. Introduced Vulnerability: path substitution in the privileged helpers (tested)

Two helpers run as root through fixed sudoers entries (the sudo rules file that says which commands a user may run as root): one stages and hashes patch media, one publishes run context and approval files. The critique says if any path those helpers touch has a parent directory the oracle user can write, or resolves through a symlink, the oracle user can redirect a root write. That’s a real bug class (CWE-59, link following). OPG’s answer is to check owner and mode on every parent directory up to a trusted anchor, refuse symlinks everywhere, use O_NOFOLLOW and O_EXCL (create, and fail if it exists) on writes, and in the context helper open directories by file descriptor rather than by path. In lab step 6, a symlinked or world-writable lock directory was refused, V2 hashing refused a planted symlink, and the bootstrap suite’s 65 checks (run as root, because they need real ownership) all passed, including “writable OS lock symlink parent blocks.” The one gap I saw is the legacy V1 hash, which silently skips symlinks; production refuses V1 media, so it only bites if you’re running the legacy mode on purpose. Some of the bash checks in the context helper are still check-then-use (test that a path isn’t a symlink, then use it by name), which is only exploitable with write access to a parent, and that’s exactly what bootstrap refuses.


The 30-Minute Lab: approving a patch, then trying to break the approval

The Rabbit Hole Generator’s challenge was to run OPG’s open-checks test suite, then build a tiny signed “plan” with openssl and show that changing the target breaks the signature. I did both, then used OPG’s own fake Oracle Home to drive a full signed run and attack it from every side I could think of. Everything ran in Docker on a Mac. There is no Oracle Database anywhere in this lab: OPG’s test harness supplies stub sqlplus, opatch, and rman programs that answer from fixture files, which is also how upstream tests it.

All lab files are in this post’s bundle: Dockerfile, run_suites.sh, toy_binding.sh, lifecycle.sh, hardening.sh, v2hash.py, spec_v2.py, golden.sh, bench.sh, bench_cold.sh, replay.sh, prodmode_host.sh, and fix-hostname-override.sh.

1. Building the lab

Diagram of the lab. On a Mac with Docker Desktop, the post’s lab folder is mounted read-only at /lab into an opg-lab container built from oraclelinux:8, running as user oracle in group oinstall with bash 4.4, Python 3.6.8 plus 3.9, OpenSSL 1.1.1k, and oracle-patch-guard at commit fb86948. Inside, OPG’s test harness builds a fixture per case: stub sqlplus and opatch in a fake dbhome_1, fake patch directories for 39472050 and 39222882, oratab, opg.conf, an approval keypair, fixture.env with MOCK answers, and runs/RUN_ID holding the evidence. The same image runs as root for the bootstrap suite, the real flock lock, and dropping the page cache. Below, a container with hostname host-a plans and signs onto a shared volume, and a container with hostname host-b applies host-a’s approval from the same paths.

One image does all the work. I based it on oraclelinux:8 (Docker Hub) because that’s the operating system family real 19c hosts run, and OPG cares about details like Python 3.6 compatibility that only show up there. The image creates an oracle user in an oinstall group, the conventional owners of an Oracle install, and runs as that user by default, because OPG’s ownership checks behave differently for root:

FROM oraclelinux:8
# python39 only for two test files that need Python 3.7+ (see lab step 2); sudo + acl for the bootstrap suite.
RUN dnf -y install git python3 python39 openssl util-linux procps-ng findutils hostname which diffutils unzip zip sudo acl \
 && dnf clean all
RUN groupadd -g 54321 oinstall && useradd -u 54321 -g oinstall -m oracle
ARG OPG_REV=fb86948842c4794119dfa0c551b7661bb17659ce
RUN git clone https://github.com/stefanv2/oracle-patch-guard.git /opt/opg \
 && git -C /opt/opg checkout -q "$OPG_REV" && chown -R oracle:oinstall /opt/opg
USER oracle
WORKDIR /opt/opg/project
docker build -t opg-lab lab/

It built in about 40 seconds. The container reported Oracle Linux Server release 8.10, bash 4.4.20, Python 3.6.8, and OpenSSL 1.1.1k, on aarch64 because this is an Apple Silicon Mac. Real Oracle hosts are x86_64; nothing in this lab depends on that.

How the fake Oracle works. OPG’s main test file, tests/run_tests.sh, has a setup_case function that builds a throwaway directory per test: a fake Oracle Home whose sqlplus and opatch are three-line scripts that exit 0, two fake patch directories (JUL2026/39472050 for the RU and 39222882 for OJVM, the patch numbers OPG’s README reports validating against on a live 19c system), an oratab with one database, a config file, and a fixture.env of MOCK_* variables that tell OPG’s check code what to “find” (backup verified, Data Guard healthy, OPatch version 12.2.0.1.52, and so on). It also generates a fresh RSA key pair. My lifecycle.sh reuses those helper functions instead of reinventing them, so the fixture is exactly the one upstream tests against. Two small things bit me while wiring that up:

  • The harness finds itself through BASH_SOURCE. I loaded its helpers with source <(sed ...), and under that process substitution the script’s “location” is /proc/<pid>/..., so every fixture path broke with No such file or directory. Setting ROOT=/opt/opg/project after sourcing fixed it.
  • OPG’s path allowlist rejected my test name. I named one scenario manifest+sidecar, and assessment failed with exit 70 before doing anything. OPG only accepts runtime paths made of letters, digits, _, ., /, and -, and the case name ended up in a path. Annoying for about a minute, and exactly the kind of strictness you want from a tool that builds paths for root.

Language note. Most of OPG’s output and docs are in Dutch (geslaagd means passed, mislukt failed, Lokaal manifest is gewijzigd “local manifest was changed”). The machine-readable lines (OPG_RESULT|...|status=BLOCKED|exit_code=20) are English, and those are what I key on below.

2. Running upstream’s own tests first

Before attacking anything, I wanted to know the tool works as shipped. The Rabbit Hole Generator asked for the open-checks suite; I ran every suite in project/tests/ with run_suites.sh:

docker run --rm -v "$PWD/lab":/lab:ro opg-lab bash /lab/run_suites.sh
tests/run_bootstrap_tests.sh                  rc=70     0s  Bootstrap regressions vereisen root voor echte ownershipvalidatie.
tests/run_completion_publication_tests.sh     rc=0      0s  Completion publication results: 17 passed, 0 failed
tests/run_media_lock_fd_tests.sh              rc=0      0s  Media lock FD results: 6 passed, 0 failed
tests/run_oem14_approval_tests.sh             rc=0      4s  OEM14 results: 13 passed, 0 failed
tests/run_oem_wrapper_tests.sh                rc=0     29s  OEM wrapper results: 119 passed, 0 failed
tests/run_open_checks_tests.sh                rc=0      0s  Resultaat open checks: 42 geslaagd, 0 mislukt
tests/run_pilot05b_tests.sh                   rc=0     22s  Pilot05b-resultaat: 39 geslaagd, 0 mislukt
tests/run_pilot07_tests.sh                    rc=0      7s  Pilot07 media results: 64 passed, 0 failed
tests/run_release_layout_tests.sh             rc=0      0s  Release-layout results: 5 passed, 0 failed
tests/run_signer_batch_tests.sh               rc=0      0s  Signer batch results: 19 passed, 0 failed
tests/run_signer_pending_tests.sh             rc=0      1s  Signer pending results: 34 passed, 0 failed
tests/run_tests.sh                            rc=0     64s  Resultaat: 141 geslaagd, 0 mislukt
tests/check_python36_compat.py                rc=0      0s
tests/run_blackout_tests.py                   rc=0      0s  OK
tests/run_bootstrap_readiness_tests.py        rc=1      0s
tests/run_database_baseline_tests.py          rc=1      0s  TypeError: __init__() got an unexpected keyword argument 'text'
tests/run_pdb_validation_order_tests.py       rc=1      0s  TypeError: __init__() got an unexpected keyword argument 'text'
tests/run_prepare_cycle_tests.py              rc=0      1s  OK
tests/run_resume_approval_tests.py            rc=0      6s  Resume authorization: 342 cases passed
tests/run_sqlpatch_action_tests.py            rc=0     10s  SQLPATCH ACTION results: 166 passed, 0 failed
tests/run_state_write_tests.py                rc=0      0s  State write results: 15 passed, 0 failed

The open-checks suite the challenge asked for passed 42 of 42 in under a second. Four lines aren’t green, and each one has a different story:

  • run_bootstrap_tests.sh exits 70 with “bootstrap regressions require root for real ownership validation.” That’s honest: those tests chown files to root and check that OPG refuses anything else. Run as root (docker run -u root ...), the first attempt failed with vereiste OS-tool ontbreekt: sudo (required OS tool missing: sudo), because bootstrap checks that sudo exists before installing sudoers rules. With sudo and acl installed (now in the Dockerfile) it passed 65 of 65 in two seconds.
  • run_bootstrap_readiness_tests.py exits 1 because it expects two arguments; the bootstrap suite calls it with them, and it passed there.
  • Two Python test files crash on Python 3.6. They call subprocess.run(..., text=True), a keyword added in Python 3.7. Oracle Linux 8’s default python3 is 3.6.8, which is the version OPG explicitly targets on the database host. Under python3.9 they pass, 26 of 26 and 20 of 20.

That last one is a small irony worth showing, because OPG ships a checker for exactly this problem:

$ python3 tests/check_python36_compat.py                     ; echo "no-args rc=$?"
no-args rc=0
$ python3 tests/check_python36_compat.py ../oem-tasks/*.py ../signer/*.py ; echo "runtime rc=$?"
runtime rc=0
$ python3 tests/check_python36_compat.py tests/run_database_baseline_tests.py tests/run_pdb_validation_order_tests.py
tests/run_database_baseline_tests.py: line 52: keyword text is newer than Python 3.6
tests/run_pdb_validation_order_tests.py: line 30: keyword text is newer than Python 3.6

The checker only checks the files you pass it; with no arguments it checks nothing and exits 0. Pointed at the runtime code (what actually runs on a database host), it’s clean, which is what matters. Pointed at the two test files, it finds the problem immediately. So this is a test-hygiene gap, not a runtime bug: the production scripts are 3.6-safe, but two of the test files that guard them can’t run on the platform they’re meant to protect.

Adding up every suite, including the root and Python 3.9 reruns, I counted 1,196 passing test cases and zero real failures at commit fb86948. For a shell-heavy project, that’s unusually thorough, and it’s why the rest of this lab could focus on what tests don’t cover rather than on whether the basics work.

3. The Rabbit Hole Generator’s miniature binding

The second half of the challenge is a toy version of PLAN and APPROVE in plain openssl: write a four-line “manifest,” sign it with an RSA key using openssl dgst, verify it, append a different target line, verify again. I ran it as given (toy_binding.sh) and added one more step at the end:

Original manifest: Verified OK
Tampered manifest: Verification Failure
EXPECTED REJECTION: approval no longer matches the plan

Replay: copy the untouched plan + signature to a second host
Replayed manifest: Verified OK

The first two lines are the point of the exercise: a signature covers every byte, so changing target= breaks it. The last line is the point the exercise doesn’t make. The untouched manifest still says target=db01.example.com, and the signature still verifies perfectly on any machine you copy it to. A signature proves the document didn’t change. It doesn’t prove the document is about the machine reading it. Something on the target has to compare target= to its own identity, and that comparison is only as good as the way the target learns who it is. Hold that thought for lab step 8.

This is the same lesson as the proofbundle post from earlier in this series, from the other side. There, a forged receipt verified fine until a trust policy pinned whose key counts. Here the key question is answered (OPG pins the approval key, as step 5 shows), and the open question is where the signed thing is allowed to run.

4. One signed run, start to finish

Now the real thing. lifecycle.sh builds a case with the healthy_single fixture (one running database, DB1, set to start at boot), runs ASSESS and PLAN, writes an approval token, signs both files with the harness’s RSA key, and sets TEST_REQUIRE_SIGNATURES=true. That last setting matters: in test mode OPG skips signature verification unless you ask for it, so a lab that forgets the flag “passes” the signature checks without running them. Production mode always verifies.

docker run --rm -v "$PWD/lab":/lab:ro -v "$PWD/out":/out opg-lab bash /lab/lifecycle.sh
== happy path, artifacts kept for inspection
signed-happy                 rc=0   downtime_started=YES
                             status=COMPLETE|phase=VALIDATION|exit_code=0

downtime_started=YES means OPG wrote a shutdown_DB1.log, i.e. it got far enough to stop the (fake) database, which is what we want on the happy path and exactly what we don’t want in step 5. OPG writes one directory per run with 106 files of evidence. Three of them are worth reading.

The state history is the run’s audit trail, one line per transition:

NONE|01_ASSESS_STARTED|ASSESS
01_ASSESS_STARTED|02_ASSESS_OK|ASSESS
02_ASSESS_OK|03_PLAN_GENERATED|PLAN
03_PLAN_GENERATED|04_APPROVED|APPROVAL
04_APPROVED|OPATCH_READY|OPATCH_UPGRADE
OPATCH_READY|05_DATABASES_STOPPED|STOP_DATABASES
05_DATABASES_STOPPED|06_DB_BINARY_APPLIED|DB_BINARY
06_DB_BINARY_APPLIED|07_OJVM_BINARY_APPLIED|OJVM_BINARY
07_OJVM_BINARY_APPLIED|08_DATABASES_STARTED|START_DATABASES
08_DATABASES_STARTED|09_DATAPATCH_COMPLETE|DATAPATCH
09_DATAPATCH_COMPLETE|10_UTLRP_COMPLETE|UTLRP
10_UTLRP_COMPLETE|11_VALIDATION_COMPLETE|VALIDATION
11_VALIDATION_COMPLETE|12_COMPLETE|COMPLETE

What to key on: 04_APPROVED must come before 05_DATABASES_STOPPED. Everything up to 04 is reversible paperwork; everything after it is downtime. If a run ever shows a stop without an approval in front of it, the tool is broken. It also tells you exactly where an interrupted run stopped, which is what OPG’s resume command uses (342 resume-authorization cases passed in step 2).

The manifest is what the approver signs. Trimmed to the fields that bind something:

{
  "run_id": "H1",
  "hostname": "ff215eb60235",
  "target_oracle_home": "/tmp/opg-tests.d9BHbi/signed-happy/dbhome_1",
  "db_patch": "39472050",
  "ojvm_patch": "39222882",
  "opatch_zip_sha256": "cbdc2d35c82da63237bad8cd4e9769347b4c57c168f561ae6b1a4a6c80c894bb",
  "db_patch_tree_sha256": "7962b3b9f7aca31e4c7ea4b638916faf476db0259fb929fcd1e1af5f9880ef81",
  "ojvm_patch_tree_sha256": "6d77b5802e26ccada6d508908dc5c06cdc66b0447df8552145428f6fd0f9f581",
  "media_mode": "LEGACY_V1",
  "tree_hash_format": "OPG_TREE_HASH_V1",
  "oratab_sha256": "6c4d6eff62d346af6bc39627a93b096f4519d2114ee38952e4a086e8fa76b87d",
  "recovery_manifest_sha256": "UNAVAILABLE",
  "approval_public_key_sha256": "e05d41942fd7b950ad2e7d833a4d3b10b3eb98f697aed345ded47d96a04ea4d5",
  "database_state_before_sha256": "8ad25e23f2996793481b9e93129c9144fb7bd3462a387e74e37d47f043d915cc",
  "database_count": 1
}

Field by field, what to look at:

  • hostname and target_oracle_home are the “where.” ff215eb60235 is just the container’s random hostname. On a real host this is hostname -f, and step 8 is about how much to trust that.
  • The three media hashes are the “what.” If the patch directory changes by one byte between PLAN and APPLY, the pre-apply recheck recomputes these and blocks (step 5 shows it).
  • media_mode: LEGACY_V1 is a lab artifact worth noticing. The harness tests from a plain directory, so it uses the old V1 hash. A production manifest says LOCAL_IMMUTABLE_V2 and OPG_TREE_HASH_V2, and production mode refuses to start with anything else. If you ever see LEGACY_V1 in a real manifest, something is misconfigured.
  • approval_public_key_sha256 pins which key may approve this run, at PLAN time. It’s why the key-swap attack in step 5 fails.
  • database_state_before_sha256 pins the list of databases, their roles, and open modes. A database that appears or changes role between PLAN and APPLY changes it.
  • UNAVAILABLE values are not automatically fatal; the harness skips the recovery manifest here. The findings file is where “missing” turns into a decision.

The approval token is short:

{
  "approved": true,
  "manifest_sha256": "fc13a47f4630322c64b918c061c23d67454a35f9d7a0ea27b9b2e2166cc5cd75",
  "hostname": "ff215eb60235",
  "target_oracle_home": "/tmp/opg-tests.d9BHbi/signed-happy/dbhome_1",
  "expires_epoch": 1794780940,
  "accept_HOME_RECOVERY_REBUILD_VERIFIED": "HOME_RECOVERY_REBUILD_VERIFIED",
  "manifest_signature_file": ".../runs/H1/patch_manifest.sig",
  "approval_signature_file": ".../runs/H1/approval.sig"
}

I re-hashed the manifest on the Mac, outside the container, rather than trusting OPG’s own sidecar file:

$ sha256sum out/runs/H1/patch_manifest.json
fc13a47f4630322c64b918c061c23d67454a35f9d7a0ea27b9b2e2166cc5cd75  out/runs/H1/patch_manifest.json

It matches manifest_sha256. That independent check is worth the five seconds because it’s the link the whole design rests on: the token doesn’t describe the plan, it names its exact bytes. expires_epoch is one hour after signing here, so an approval can’t be held for a month and used later. And each accept_<FINDING> line is the approver acknowledging one specific CONDITIONAL finding from findings.psv; this run had one, CONDITIONAL|HOME_RECOVERY_REBUILD_VERIFIED (the fallback plan is “rebuild the Oracle Home from a known image,” which OPG wants a human to sign off on explicitly). A token that omits an accept_ line for a conditional finding is refused, so a reviewer can’t approve a run without at least being shown each caveat.

5. Seven ways to tamper with an approved run

Each scenario below starts from a fresh, validly signed run and changes exactly one thing before APPLY:

== tamper replays (each starts from a fresh, validly signed run)
manifest-edited              rc=20  downtime_started=no   APPROVAL CHECK BLOCKED: local manifest hash verification failed
                             status=BLOCKED|phase=APPROVAL|exit_code=20
manifest-sidecar             rc=20  downtime_started=no   APPROVAL CHECK BLOCKED: manifest hash mismatch
                             status=BLOCKED|phase=APPROVAL|exit_code=20
token-edited                 rc=20  downtime_started=no   APPROVAL CHECK BLOCKED: approval signature verification failed
                             status=BLOCKED|phase=APPROVAL|exit_code=20
token-expired                rc=20  downtime_started=no   APPROVAL CHECK BLOCKED: approval token expired
                             status=BLOCKED|phase=APPROVAL|exit_code=20
rogue-signer                 rc=20  downtime_started=no   APPROVAL CHECK BLOCKED: manifest signature verification failed
                             status=BLOCKED|phase=APPROVAL|exit_code=20
pubkey-swapped               rc=20  downtime_started=no   APPROVAL CHECK BLOCKED: approval public key differs from PLAN trust binding
                             status=BLOCKED|phase=APPROVAL|exit_code=20
media-drift                  rc=20  downtime_started=no   STATIC_CONTEXT_MISMATCH|check=db_patch_tree_sha256
                             status=BLOCKED|phase=PREAPPLY|exit_code=20

Seven for seven, all exit 20, and the column that matters most is downtime_started=no: no shutdown_*.log exists for any of them. Here’s what each one proves, because they hit different layers:

  • manifest-edited: I changed db_patch from 39472050 to 39472051 (say, to slip in a different patch). The first thing to notice is the cheap local sidecar, patch_manifest.sha256, which PLAN wrote and which no longer matches.
  • manifest-sidecar: same edit, but I also rewrote the sidecar to match, the way someone who read the code would. Now the sidecar agrees, and the next layer catches it: the signed token’s manifest_sha256 doesn’t match. The sidecar is a tripwire for accidents; the signature is the actual control.
  • token-edited: I pushed expires_epoch out to the year 2100 without re-signing. The approval signature no longer verifies.
  • token-expired: I set the expiry to 1970 and re-signed with the real key, so the signature is valid. Expiry is checked anyway, as data, not as a signature property.
  • rogue-signer: I generated a fresh RSA key and signed both files with it. Signature verification against the configured public key fails.
  • pubkey-swapped: the smarter version: sign with a rogue key and replace the host’s approval public key with the rogue public key, so the signatures verify. OPG still refuses, because the manifest pinned the original key’s hash at PLAN time (approval_public_key_sha256), and the key on disk now hashes to something else. This is the attack the proofbundle post’s trust policy existed to stop, and here it’s built in.
  • media-drift: I appended one line to a file in the DB RU directory after approval. The approval checks all pass (the manifest is untouched), and the pre-apply recheck re-hashes the tree, sees db_patch_tree_sha256 changed, and stops with “geen database of listener is gestopt” (no database or listener was stopped).

A note on what this does and doesn’t prove. Every one of these tampers is a file the oracle user can write, which is the realistic threat: a DBA account, or something running as one, changing things after review. None of them prove OPG survives root on the target, which is Reality Check 4 and genuinely out of scope.

6. Locks, the test-mode gate, and symlinks

hardening.sh runs as root and pokes at three local guardrails directly, by loading OPG’s core library and calling its functions:

== 1. OPG_TEST_MODE=1 without ALLOW_TEST_MODE in the config
Testmodus is niet toegestaan door de configuratie.
rc=70

== 2. Real flock home lock (OPG_TEST_MODE unset)
  RUN-A on /u01/app/oracle/product/19c/dbhome_1 -> rc=0
  log WARN: Lock bezet door host=db01.example.com pid=1; lock wordt niet verwijderd.
  RUN-B on /u01/app/oracle/product/19c/dbhome_1 -> rc=60
  RUN-C on /u01/app/oracle/product/19c/dbhome_2 -> rc=0
  lock files:
    5d310d954f6f90c8dac3eca0cecce5c692d2398ed098d0512dfeea6bbd05a49e.lock
    90b781d22e5aed0189aae6e08ba04198a050824ab6d96e8e668172fe159df2aa.lock
  -- world-writable lock root
  log ERROR: LOCK_SETUP|lock-root moet root-owned en niet world-writable zijn: /tmp/tmp.kFBibu5DOU/locks owner=root mode=777
  RUN-D on /u01/app/oracle/product/19c/dbhome_1 -> rc=20
  -- symlinked lock root
  log ERROR: LOCK_SETUP|lock-root ontbreekt, is een symlink of is niet toegankelijk/schrijfbaar: /tmp/tmp.kFBibu5DOU/locklink
  RUN-E on /u01/app/oracle/product/19c/dbhome_1 -> rc=20

== 3. V1 (legacy) vs V2 tree hash with a planted symlink
  clean tree:      V1=fadefe9b52454308f4dd0e462f70e311a4c50d07ed01bdd072ae12be277a344c
                   V2=bccc28cdec3de765af7d8066a77a1dc243265d54e12d09ad06287ad32a83c0f0
  + symlink added: V1=fadefe9b52454308f4dd0e462f70e311a4c50d07ed01bdd072ae12be277a344c
                   V2=OPG_MEDIA_ERROR|exit_code=20|message=symlink/hardlink/speciaal object in tree: tree/39472050/files/extra.bin

Section 1 matters because test mode relaxes a lot: mock locks, fixture answers, optional signatures. If anyone could turn it on with an environment variable, every control above would be a suggestion. They can’t: OPG_TEST_MODE=1 is refused unless the config file says ALLOW_TEST_MODE=true, and in production that config must be owned by root and not writable by group or others. That’s the right design, and it’s the design I’ll hold the hostname override to in step 8.

Section 2 shows the APPLY lock’s real scope (Reality Check 1). RUN-A holds the lock on dbhome_1 for three seconds; RUN-B on the same home gets exit 60 (“already running”) and a warning naming the holder, and OPG deliberately does not delete someone else’s lock. RUN-C on dbhome_2 proceeds, because lock files are named by a hash of host plus home (two homes, two files). A world-writable or symlinked lock directory is refused with exit 20 rather than trusted. (The pid=1 is a quirk of my harness: I call the lock function from a subshell, where $$ is the parent’s PID, which is the container’s PID 1.)

Section 3 is the one I’d circle (Reality Check 6). I planted files/extra.bin as a symlink to /etc/shadow in a patch tree. The legacy V1 hash is identical before and after, because find -type f skips symlinks: V1 proves “these regular files are unchanged,” not “this tree is unchanged.” V2 refuses the whole tree. If OPatch followed that link during apply, a V1-approved tree could carry content nobody hashed. Production refuses V1 media, which is exactly why it should, and why LEGACY_V1 in a real manifest is a red flag.

7. Checking the hash spec independently, and what hashing costs

A hash format is only useful if two independent implementations agree, since the signer side and the host side both have to compute it. The V2 spec publishes three golden vectors (known inputs with their expected outputs). I wrote spec_v2.py from the spec’s prose alone, without importing or reading OPG’s implementation, and compared both against the published values (golden.sh):

empty  spec=fc7148770e90462c53b00b50bf17c592601347ca8413c60df09caf6722e9d597
       opg =fc7148770e90462c53b00b50bf17c592601347ca8413c60df09caf6722e9d597 MATCH
       mine=fc7148770e90462c53b00b50bf17c592601347ca8413c60df09caf6722e9d597 MATCH
abc    spec=5d6926712e919b727b504ee586bd9280834355e7d804d115102c3f09641be616
       opg =5d6926712e919b727b504ee586bd9280834355e7d804d115102c3f09641be616 MATCH
       mine=5d6926712e919b727b504ee586bd9280834355e7d804d115102c3f09641be616 MATCH
odd    spec=0e3d6c0065671517043627886bff3b8f7b79962e85211c76e2e5d41c2b266c5b
       opg =0e3d6c0065671517043627886bff3b8f7b79962e85211c76e2e5d41c2b266c5b MATCH
       mine=0e3d6c0065671517043627886bff3b8f7b79962e85211c76e2e5d41c2b266c5b MATCH

The odd vector is the interesting one: file names a|b and one containing a newline. Those are the names that break naive “print path, print hash, join with newlines” schemes, because a crafted file name can make two different trees produce the same text. V2’s length-prefixed records make that impossible, and an independent implementation reproducing all three is good evidence the spec is complete enough to build against. (The spec mentions a tests/test_tree_hash_v2.sh that guards these values; that file isn’t in the repo at this commit. The Pilot07 suite’s “V2 refuses symlinks fail-closed” case covers part of it.)

Then the cost (Reality Check 2). I don’t have a real RU to unzip in a lab, and I couldn’t find a trustworthy published size for one, so bench.sh builds a synthetic tree: 200 directories, 20,000 files of 100 KiB random data, about 2 GB. Then it times OPG’s V1 hash, OPG’s own V2 function, and OPG’s bundled tools/opg_hash_benchmark.sh, wrapped in caffeinate -i so the Mac couldn’t sleep mid-run:

tree: 20000 files, 2.0G
run 1  V1   1151 ms  V2   1963 ms
run 2  V1    789 ms  V2   1623 ms
run 3  V1    731 ms  V2   1627 ms
-- OPG tools/opg_hash_benchmark.sh
OPG_HASH_BENCHMARK|label=synthetic-2g|metric=metadata_walk_sort|duration_ms=5
OPG_HASH_BENCHMARK|label=synthetic-2g|files=20000|bytes=2048000000|filesystem=overlayfs
OPG_HASH_BENCHMARK|label=synthetic-2g|metric=sequential_read|duration_ms=4344
OPG_HASH_BENCHMARK|label=synthetic-2g|metric=per_file_sha256|duration_ms=8269

Those runs were with the files still in memory (just written), so I reran V2 in a privileged container that drops the Linux page cache before each pass (bench_cold.sh):

cold run 1  V2   3277 ms
cold run 2  V2   3338 ms
cold run 3  V2   3281 ms

So V2’s extra safety (two directory walks, a metadata check per file) costs about twice V1’s time, and at this size both are seconds on local SSD with 10 cores. OPG’s own benchmark tool is the slowest at 8.3 seconds, for an unrelated reason: it starts one sha256sum process per file, so it mostly measures process startup. Two passes per run at a few seconds each won’t threaten a maintenance window on a single host. The fleet version of the critique is still open, because the expensive step is staging media from a shared drive to every host at once, and that’s the part a laptop can’t model.

8. Replaying an approval on another host

Back to the question from step 3: how does the target know who it is? In patchGD_guard.sh, line 184:

HOST_NAME=${OPG_HOSTNAME_OVERRIDE:-$(hostname -f 2>/dev/null || hostname)}

If OPG_HOSTNAME_OVERRIDE is set, it wins. Unlike OPG_TEST_MODE, nothing gates it on the config file, and none of OPG’s test suites use it. replay.sh tests what that means with two containers sharing a Docker volume, so both “hosts” see the same paths, the way a cloned server would:

docker volume create opg-shared
docker run --rm -u root -v opg-shared:/shared opg-lab chown oracle:oinstall /shared
docker run --rm --hostname host-a -v opg-shared:/shared -v "$PWD/lab":/lab:ro opg-lab bash /lab/replay.sh plan
docker run --rm --hostname host-b -v opg-shared:/shared -v "$PWD/lab":/lab:ro opg-lab bash /lab/replay.sh apply override
signed on host-a: manifest hostname="hostname": "host-a"
applying on host-b with OPG_HOSTNAME_OVERRIDE=host-a
rc=0
host=host-a status=COMPLETE|phase=VALIDATION|exit_code=0

Without the variable, the same replay is refused with manifest hostname mismatch, exactly as designed. With it, host-b ran the whole lifecycle to 12_COMPLETE under an approval signed for host-a, and its evidence files (host_info.txt, the OPG_RESULT line) record hostname=host-a. The audit trail says the right host was patched.

I hit one detour on the way that’s worth knowing about. My first attempt ran without the override, got BLOCKED, and then I ran the override against the same run. It was blocked again, instantly: the run’s state was now BLOCKED, and OPG won’t resume a run from that state. A failed approval poisons the run, and you need a new PLAN. That’s good fail-closed behavior, and it’s why the result above comes from a fresh plan.

Is this realistic? Partly, and it’s worth being precise. The lab ran in OPG’s test mode, because there’s no real Oracle here, and the fixture makes host-b look identical to host-a. In production, host-b’s pre-apply recheck reads its own oratab, database list, and patch media, and all of those must match host-a’s signed hashes. On genuinely different servers they won’t. On servers built from the same golden image, with the same SID, home path, and staged media (common for a test-to-prod pair or a standby rebuilt as primary), they can. And the person setting the variable already has the oracle account on host-b, so this isn’t privilege escalation. It’s approval scope confusion: an approval that should mean “db01, this window” being usable for “a server that looks like db01,” with evidence that says db01.

To rule out a test-mode-only quirk, prodmode_host.sh runs as root with a root-owned production config and no OPG_TEST_MODE, and just asks OPG for a run status:

hostname -f says: host-b
OPG_HOSTNAME_OVERRIDE=<unset>  -> host=host-b
OPG_HOSTNAME_OVERRIDE=host-a   -> host=host-a

Production mode honors the override too.

The fix is one line: honor the override only when the already-gated test mode is on. fix-hostname-override.sh applies it with sed:

if [[ ${OPG_TEST_MODE:-0} == 1 && -n ${OPG_HOSTNAME_OVERRIDE:-} ]]; then HOST_NAME=$OPG_HOSTNAME_OVERRIDE; else HOST_NAME=$(hostname -f 2>/dev/null || hostname); fi
hostname -f says: host-b
OPG_HOSTNAME_OVERRIDE=<unset>  -> host=host-b
OPG_HOSTNAME_OVERRIDE=host-a   -> host=host-b

With the patch applied, the main suites still pass (run_tests.sh 141 of 141, OEM wrappers 119 of 119, resume authorization 342 of 342), so nothing upstream relies on the override in production paths. I passed this along to the project as a friendly heads-up (issue #3), framed the same way it is here: a nuance in a tool I came away impressed by, not an alarm. A stronger version would bind the manifest to something the caller can’t set at all, such as /etc/machine-id (a per-install random ID) or a per-host key, but that changes the manifest format and is a design discussion, not a one-liner. Until then, the trust-but-verify control is outside OPG: have whatever launches APPLY (the OEM job, the sudo rule) start it with a clean environment, and compare the host= in OPG_RESULT against the inventory name of the machine the job actually ran on.


Where this fits in a real environment

If you patch single-instance Oracle 19c homes and your change process is a ticket plus a trusted DBA, OPG is worth reading even if you never run it. The pattern is the valuable part: a plan that pins facts by hash, an approval that signs those exact bytes, a pinned approver key, an expiry, explicit acknowledgment of each caveat, and a re-check of everything volatile immediately before downtime. My seven tampers in lab step 5 each hit a different layer, and none got as far as stopping a database. The 1,196 passing tests suggest that’s not luck. If you do run it, start where upstream did: one non-production CDB, the command-line flow, then OEM.

Know what it doesn’t cover. It refuses RAC, ASM/Grid, and Data Guard, which rules out a lot of the databases people most fear patching. It patches in place, while Oracle’s own guide now recommends out-of-place patching (installing into a new Oracle Home and switching over) for all patch operations (Oracle patch maintenance guide); OPG’s fallback for a broken home is a verified rebuild route rather than a switch back. Fleet coordination belongs to OEM, and the lock only sees one host (Reality Check 1). The signer you’d actually depend on isn’t public (3). Its backup check proves recency, not restorability, so pair it with scheduled RESTORE ... VALIDATE runs or real restore tests (5). And until the hostname override is gated, run APPLY with a scrubbed environment and check the reported host yourself (4, lab step 8).

It also fits next to the supply-chain tools earlier in this series. Macaron and Tejolote ask whether an artifact came from where it claims to. OPG starts one step later: given media you’ve already decided to trust, it proves the bytes you approved are the bytes you apply. It doesn’t check that the patch ZIP came from Oracle; that stays your download process’s job.


Actionable Takeaways

  • Make approvals name bytes, not descriptions. A token that carries the plan’s SHA-256 and is signed turns “approved the July patch for db01” into something a script can check.
  • Pin the approver’s key in the plan itself. The swapped-public-key attack only failed because the manifest recorded the key’s hash at PLAN time.
  • Re-check volatile facts at the last moment, before downtime. The media-drift tamper passed every approval check and was caught only by the pre-apply re-hash.
  • Treat host identity as an input to verify, not a given. hostname -f and an override variable are both caller-influenced; gate overrides behind the same switch as test mode.
  • Run the compatibility checker on your tests too. Two of OPG’s guard tests can’t run on the Python 3.6 hosts they’re meant to protect.
  • Distrust any LEGACY_V1 tree hash. It skips symlinks, so a planted link doesn’t change the hash.

A signature tells you the plan didn’t change; only the checks around it tell you the plan is about the machine you’re standing on.