Three Out of Eight: What Skylos Proves About AI-Written Code

Coding agents are good at writing code that looks right. The failure mode everyone who uses them has hit is code that calls something that doesn’t exist: a helper the model assumed you had, a method the library never shipped, a package name that sounds plausible. Python doesn’t complain until the line runs. A reviewer skimming a 400-line agent PR may not notice either, because the code reads fine.

The Rabbit Hole Generator picked Skylos this week. Its pitch for that problem is to treat review as a proof: before a human looks, statically check that every package, helper, and API the change references actually exists, and say incomplete instead of guessing when it can’t tell. Its README tagline is “Code checks that show their work.” That’s a claim a security reader can test directly, so I did. I planted eight different AI-style mistakes and ran them through Skylos, through two ordinary Python tools for comparison, and through the pull request gate Skylos generates for GitHub Actions. Then I tried to get a bad change past that gate.

The short version: Skylos proved three of the eight, and the three it proves include one that the ordinary tools can’t (a package that doesn’t exist on PyPI at all). It passed the other five, including the Rabbit Hole Generator’s own example. Its CI gate held up well against a pull request that edits Skylos’s own settings, but a one-line # skylos: ignore comment in the pull request still got a hallucinated import through.

What is Skylos used for in application security?

Skylos is an open-source (Apache-2.0) command-line tool and pull request gate, written in Python, that runs locally without an account. By default it finds dead code; with -a it also checks for security flaws, hard-coded secrets, vulnerable dependencies, quality problems, CI misconfiguration, and “AI-code defects,” meaning references to packages, versions, helpers, and APIs that don’t exist. skylos verify returns pass, fail, or incomplete for a file or tree, built for a coding agent to call before it hands work to a human. skylos done runs a branch’s tests and checks whether an agent weakened them. It’s on PyPI and moves very fast: this post tests 5.1.1, the latest release when I ran the lab, which came 27 releases after the August version the Rabbit Hole Generator picked. Version 5.2.0 shipped the day before this post went up; I reran lab steps 2 to 7 on it, and every result matched 5.1.1. Skylos also has an optional hosted dashboard, Skylos Cloud. Nothing in this post uses it; everything here runs locally.

It isn’t limited to Python code. The README lists 11 languages: Python, TypeScript/JavaScript, Java, Go, PHP, Rust, Dart, C#, C++, Kotlin, and Shell, plus CI/CD and deployment config. Coverage is uneven, though. Python gets every check. TypeScript/JavaScript and Java get dead code, security, and quality checks; PHP and Dart get dead code and security; the rest are partial (Kotlin has no security rules, C++ only finds unused file-local functions, and Shell gets security checks only). The part of verify this post leans on, proving that a referenced name or API actually exists, covers Python, TypeScript/JavaScript, Go, and Java; for the other languages that check reports incomplete. Every experiment below is Python, so the other languages are the README’s claim, not something I tested.

The Architecture: one scanner, many questions

One scan, many analyzers. skylos . parses the repository once and runs several analyzers over the result. That’s the main design bet: instead of running a dead-code tool, a secret scanner, a dependency scanner, and a linter separately and merging their output, one tool builds one model of the repo and answers all of those questions from it. The dead-code analyzer is the original core of the project and the most careful part. It doesn’t just report “no references found.” For every symbol it records evidence (called from module import, registered by a decorator, named as a pyproject.toml script, reached through an unresolved getattr) and classifies it as alive, uncertain, or likely_dead. Only likely_dead gets reported. That design choice matters a lot later in the lab.

“Does this thing exist?” is a set of narrow questions, not one big one. The AI-defect checks are where verify gets its verdicts, and they’re more specific than the marketing word “hallucination” suggests. Reading the verification docs and the rule source, a Python change is checked for:

• a package that doesn’t exist on PyPI (rule SKY-D222), or a pinned version that doesn’t exist (SKY-D225). Skylos asks the registry directly.
• a name imported from one of your own modules that the module doesn’t define (SKY-L012).
• a function or keyword argument that the installed package doesn’t have (SKY-D224). “Installed” means installed in the Python environment running Skylos, not your app’s environment.
• a call to an undefined function whose name is on a built-in list of 66 commonly hallucinated security helpers, like sanitize_input, validate_jwt, or check_permission (also SKY-L012). You can add names to that list.

Anything outside those questions isn’t a “fail” or an “incomplete.” It’s simply not asked, and the file passes.

Static, but not offline. Skylos never runs the code it’s checking; that’s a stated design rule, and I confirmed it in lab step 6. But it isn’t offline either. The registry checks call PyPI (or npm, or the Go module proxy), and dependency checks send package names and versions to OSV, Google’s open vulnerability database. The project publishes an unusually honest list of every network request and how to turn each off. Nothing goes to Skylos’s own servers unless you run a Cloud command or pass --upload.

Four ways to get a verdict, with different trust. This is the part that’s easy to miss, and the part the lab spends the most time on. The same findings can be turned into a decision four ways:

“The working tree” means the code being checked, which for an agent’s or contributor’s branch means the author of the change also controls the settings. The CI gate deliberately doesn’t trust that: it reads policy from the branch you’re merging into, so a pull request can’t loosen its own check. Whether that holds up is lab step 5.

The Reality Check: where this breaks

Skylos is easy to like. The docs are candid about limits, the output says what it couldn’t check instead of implying everything is fine, and incomplete is a real third answer rather than a softened pass. So the critique the Rabbit Hole Generator wrote for it deserves a fair test, not a takedown. It made six distinct claims: three about scale, two about assumptions the creators make, and one introduced vulnerability. I tested five of them in the lab, some with surprising results, and say below why the sixth was only partly testable.

1. Monorepos with generated, vendored, and framework code will drown it in noise (tested)

The claim: real repos have generated code, vendored libraries, and framework “magic” that static reachability can’t model, so dead-code findings turn into noise. In lab step 7 I built a service whose live code is reached only through getattr dispatch, importlib plugins, a decorator registry, pyproject.toml entry points, a Flask route, and a plugin named only in a config file. Skylos reported zero false positives, even at --confidence 0, and found the one function that really was dead. It also skips vendored and generated folders by default. The prediction was wrong about noise. The real cost is the opposite, silence, which point 3 covers.

2. A unified scan is too slow for CI that needs feedback in under five minutes (tested)

One scan doing everything means one slow job instead of several parallel ones. Lab step 8 times it on a real project, mitmproxy’s 578 Python files. The dead-code scan took under 20 seconds. The full -a scan took about 7.5 minutes, the cache barely helped, and it spent most of that time on one of the ten CPU cores available, so a bigger runner won’t fix it. The CI gate Skylos generates sidesteps much of this by scanning only the lines a pull request changed, which is the right design, but a full-repo -a on push to main is a different story.

3. Reflection, plugins, and runtime imports degrade into “incomplete” results reviewers learn to ignore (tested)

This one is half right, in an interesting way. For dead code, Skylos handles dynamic Python by abstaining: anything a getattr on an unknown object could reach becomes uncertain and is never reported. That kept the noise at zero in lab step 7, but it also hid a genuinely dead method sitting next to the dynamic handlers, and Skylos can’t tell the two apart. For verify, I found a different “incomplete” problem that doesn’t involve dynamic code at all: with --project-context, a perfectly clean file came back incomplete because a different file in the repo had a real finding (lab step 3). Five clean files, five incomplete verdicts, all for a reason that had nothing to do with them. That’s exactly the pattern that teaches people to ignore a status.

4. The creators assume local-first beats centralized analysis (partly tested)

Enterprises want central policy, audit trails, and findings correlated across repos. Skylos does have a central story (Skylos Cloud can sync a gate policy that the repository can’t override), but I didn’t test it because it needs an account and a fleet of repos to mean anything. What I could test is what “local-first” means without it: the same repo scanned by the August version (4.33.1) and 5.1.1 produced 17 findings versus 9, with different rules firing, and verify returned pass on one and fail on the other (lab step 2). And skylos . and skylos verify both take their settings from the code being checked, which lab step 5 shows a pull request can change. Local is fine for a developer’s own feedback loop. A decision needs a pinned version and settings the author doesn’t control, which is exactly what the generated CI gate does.

5. The creators assume static proof of AI mistakes maps to the defects that matter (tested)

The Rabbit Hole Generator’s sharpest point: the AI mistakes that hurt are often semantic, like a business rule implemented wrong, not a name that doesn’t exist. Two of my eight planted mistakes were exactly that: a stub is_admin() that always returns True, guarding an account deletion, and a discount function that treats 20 as 2000%. Both use only real names and real APIs. Skylos passed both, and so did Ruff and mypy, which is expected; no static existence check can catch them. The honest framing is that Skylos answers “does this exist?”, and that is a necessary question but a narrow one. The control for the semantic class is tests, and skylos done does point at it: it flagged “1 of 1 changed lines are not checked by any test” on my branch (lab step 5).

6. Introduced Vulnerability: the free gate lets a pull request waive its own critical finding (tested)

The claim: Skylos encourages running repo-supplied rule packs, contracts, CI templates, and test scaffolding, which lets a malicious repo or pull request influence the security decision before merge. I tested it with five pull request branches that each add the same nonexistent-package import, four of them with a different trick to hide it (lab step 5). The local commands, skylos . and skylos verify, read settings from the branch being checked, so editing [tool.skylos] turned a fail into a pass. That’s normal for a linter; it just means you don’t gate merges on them. The CI gate Skylos generates reads policy from the base branch, and it blocked every settings edit. What it didn’t block is a one-line # skylos: ignore comment on the bad import. Skylos documents inline ignores as intentional waivers, and the maintainers know a pull request author can use one: the Skylos Cloud version of the PR scan puts inline-ignored findings back before deciding. The free --no-upload workflow doesn’t, and nothing a reviewer normally looks at (the gate summary, annotations, PR comments) mentions the import. The comment itself is still in the diff, so this is a review-discipline risk more than a silent hole, and lab step 5 has a one-line fix. Separately, skylos done runs the branch’s tests, and in lab step 6 a conftest.py added by the branch ran as root with the CI tokens in its environment. That’s how running tests works anywhere, not a Skylos bug, but don’t run done on untrusted pull requests in a job that holds secrets.

The 30-Minute Lab: eight hallucinations and a pull request that hides its own

The Rabbit Hole Generator’s challenge: build a tiny repo containing a secret, an AI-style hallucinated helper, and dead code, and get Skylos to surface all three. I ran that as written, then kept going: a matrix of eight different AI-code mistakes to see exactly which ones verify proves, a comparison against Ruff and mypy, a contract, a set of pull request branches against the CI gate Skylos generates, a code-execution check, dynamic code, and timing on a real project.

1. Building the lab

Everything runs in throwaway Docker containers with the test repo bind-mounted at /work. There’s no service to stand up: Skylos is a CLI, so each experiment is one docker run against one small git repo. I built two images, because the Rabbit Hole Generator picked Skylos in early August and 27 more releases shipped before 5.1.1, so I wanted the August behavior alongside 5.1.1.

# Dockerfile: the release this post tests (5.1.1)
FROM python:3.12-slim
ARG SKYLOS_VERSION=5.1.1
RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/* \
 && pip install --no-cache-dir "skylos==${SKYLOS_VERSION}" pytest
WORKDIR /work

# Dockerfile.v433: the release that was current when this week was picked
FROM python:3.12-slim
# 4.33.x predates the optional [dart] extra: pip builds a Dart parser from source, so it needs gcc.
RUN apt-get update && apt-get install -y --no-install-recommends git gcc libc6-dev && rm -rf /var/lib/apt/lists/* \
 && pip install --no-cache-dir "skylos==4.33.1" pytest
WORKDIR /work

Setup notes you’ll hit:

• The August release won’t install on a slim image without a compiler. My first build of 4.33.1 failed in pip install. Before 4.47.2, pip install skylos always built a Dart-language parser from C source; it’s an optional skylos[dart] extra now. Adding gcc libc6-dev fixed it. Today’s release installs fine without them.
• git is required inside the image. verify compares your working tree against Git HEAD, and done diffs against a base branch, so both shell out to git.
• Git refuses the bind-mounted repo at first. The files are owned by your host user, the container runs as root, and git’s safe.directory protection treats a repo owned by someone else as untrusted. Every command below starts with git config --global --add safe.directory '*' inside the container. That’s fine for a throwaway lab; don’t put it in a real CI image.
• The first line of output is a ripgrep warning. 5.1.1 prints ripgrep (rg) unavailable to Skylos; using fallback verification. ripgrep is a fast file search tool Skylos uses when present. The fallback produced the same findings here; on big repos it’s worth installing.
• “Local” scans make network calls. The AI-defect checks ask PyPI whether packages and versions exist, and -a sends your dependency list to OSV. The lab needs outbound network for those lookups, and on an air-gapped machine those checks come back incomplete rather than pass.

The lab files are in the page bundle so you can rebuild it: Dockerfile, Dockerfile.v433, and one generator script per experiment (make_challenge.sh, make_variants.sh, make_prgate.sh, make_dynamic.sh), the runners (verify_each.sh, ci_gate.sh, timing.sh), and run_all.sh, which does all of it in order.

docker build -t skylos-lab:5.1.1 -f Dockerfile .
docker build -t skylos-lab:4.33.1 -f Dockerfile.v433 .

2. The challenge as written

The Rabbit Hole Generator’s repo has three files. src/app.py holds a Stripe-style API key, a fake_guard() that always returns True, a call to invented_payment_sdk.charge() (a name that’s never imported or defined anywhere), and an unused_helper(). A GitHub Actions workflow uses an unpinned action, and requirements.txt pins requests==99.99.99, a version that doesn’t exist.

API_KEY = "sk_live_1234567890EXAMPLE"

def fake_guard(user):
    return True

def process_payment(amount):
    if fake_guard("admin"):
        return invented_payment_sdk.charge(amount)
    return None

def unused_helper():
    return "dead code"

print(process_payment(100))

The full scan on 5.1.1 (trimmed to the findings):

docker run --rm -v "$PWD":/work -w /work skylos-lab:5.1.1 sh -c \
  "git config --global --add safe.directory '*'; skylos . -a --format pretty; echo scan_exit=\$?"

  Risk: CRITICAL · 7 unresolved findings
  Coverage: incomplete; not fully checked: dependencies; some issues may not be reported

    Type: Review  Severity:  MEDIUM  SKY-D291  Workflow declares no permissions, so job deploy run
          with the repository's default GITHUB_TOKEN permissions, which may be read-write.
    Type: Review  Severity:  HIGH  SKY-D225  Hallucinated PyPI dependency version
          '[email protected]'. Version does not exist in PyPI.
    Type: Review  Severity:  MEDIUM  SKY-U005  Declared dependency 'requests' appears unused.
    Type: Review  Severity:  HIGH  SKY-L014  Hardcoded credential in 'API_KEY'.
    Type: Review  Severity:  MEDIUM  SKY-L032  Mock or placeholder data in 'API_KEY'
          (placeholder_phone). Phone number uses a placeholder pattern.
    Type: Review  Severity:  LOW  dead-code/variable  Unused variable: API_KEY       Confidence: 60%
    Type: Review  Severity:  LOW  dead-code/function  Unused function: unused_helper Confidence: 90%
    Type: Review  Severity:  CRITICAL  SKY-S101  Potential stripe secret detected   sk_l…MPLE
    Type: Review  Severity:  CRITICAL  SKY-S101  High-entropy value detected (entropy=4.48)

  9 review   2 critical    2 high    3 medium    2 low
scan_exit=0

What to read here:

• The secret is caught twice (SKY-S101, CRITICAL): once by a provider pattern (sk_live_ is how Stripe live keys start) and once by entropy, a measure of how random a string looks. The value is masked in the output (sk_l…MPLE), which matters if the report ends up in a CI log.
• Dead code is caught with a confidence and the evidence behind it: unused_helper at 90%, and API_KEY at only 60%, because a module-level constant could be someone else’s import.
• The hallucinated version is caught (SKY-D225): Skylos asked PyPI and 99.99.99 isn’t there.
• The hallucinated helper isn’t caught. Nothing mentions invented_payment_sdk, and nothing mentions the guard that always returns True. Hold that thought.
• One finding is wrong. SKY-L032 calls the Stripe key a “placeholder phone number,” because 1234567890 matches a fake-phone pattern. Harmless, and a nice reminder that pattern rules see digits, not meaning.
• scan_exit=0 with two CRITICAL findings. skylos . is a report. It fails only with --gate. If you wire it into a script and check the exit code, you’ll pass everything.
• “Coverage: incomplete” is Skylos telling you it couldn’t fully check dependencies (there’s no lockfile, so it can’t know what versions would really install). This is the kind of line other tools leave out.

The August release (4.33.1) on the same repo found 17 issues instead of 9. It had no SKY-S101 secret finding at all (the key showed up only as the quality rule SKY-L014), but it listed four CI findings where 5.1.1 lists one, including a HIGH for the action that isn’t pinned to a commit SHA (5.1.1 still measures that, but counts it as “advice” and hides it unless you pass --all-advice). Same repo, same command, eleven weeks apart, materially different results. That’s the version-drift half of Reality Check point 4, and the reason to pin the version anywhere a verdict matters.

Now the command built for exactly this problem, skylos verify:

skylos verify . --file src/app.py --project-context --format short ; echo "verify_exit=$?"

# skylos 5.1.1
src/app.py:1 SKY-S101 [CRITICAL] Potential stripe secret detected (redacted: sk_l…MPLE)
src/app.py:1 SKY-S101 [CRITICAL] High-entropy value detected (entropy=4.48) (redacted: sk_l…MPLE)
FAIL: 2 issues found: 2 secret
verify_exit=1

# skylos 5.1.1, same command plus --no-security (the AI-code-only verdict)
PASS: No AI-code issues found
verify_exit=0

--format short prints one line per finding plus a verdict line; without it, an interactive terminal gets a longer human report with the same content.

// skylos 4.33.1, the release that was current in August (trimmed)
{
  "status": "pass",
  "summary": "No AI-code issues found",
  "coverage": {
    "state": "complete",
    "checks": [
      { "id": "python_local_api_reference", "status": "completed", "outcome": "pass",
        "references": 0, "checked_references": 0 }
    ]
  }
}

This is the most important result in the post, so it’s worth slowing down. Today’s verify fails the file, but only because of the secret. Turn off the security checks and the AI-code verdict is pass. In August, before security findings were folded into verify, the verdict was simply pass, with coverage complete. The key field is "references": 0: the check that’s supposed to prove referenced APIs exist found nothing to check, because invented_payment_sdk is never imported. Skylos’s local-API proof works on imports; a bare name that was never imported isn’t a reference it tracks. So the challenge’s required outcome (“at least one AI/API verification failure or incomplete proof” on that file) didn’t happen in either version. The hallucination got a clean bill of health.

That’s not necessarily a bug. It’s a scope question, and the next step maps the scope.

3. Eight hallucinations, three verdicts

To find out exactly what verify proves, I wrote one file per kind of mistake, so every verdict has exactly one cause (make_variants.sh). A real src/shop/payments.py defines charge_card(amount, currency="usd"), and the variants misuse things around it:

verify_each.sh runs skylos verify . --file <f> --no-security --format json on each and prints the status. --no-security isolates the AI-code verdict:

src/shop/h1_bare_name.py              exit=0 PASS No AI-code issues found []
src/shop/h2_fake_package.py           exit=1 FAIL 1 AI-code issue found ['SKY-D222']
src/shop/h3_undefined_function.py     exit=0 PASS No AI-code issues found []
src/shop/h3b_listed_name.py           exit=1 FAIL 1 AI-code issue found ['SKY-L012']
src/shop/h4_local_missing_member.py   exit=1 FAIL 1 AI-code issue found ['SKY-L012']
src/shop/h5_local_bad_kwarg.py        exit=0 PASS No AI-code issues found []
src/shop/h6_real_pkg_fake_member.py   exit=1 FAIL 1 AI-code issue found ['SKY-D224']
src/shop/h7_fake_guard.py             exit=0 PASS No AI-code issues found []
src/shop/h8_logic_bug.py              exit=0 PASS No AI-code issues found []

(h3b is a control file I added after the first run; it’s explained below.)

The three it fails are the three it’s designed for, and the messages are precise:

h2:1 SKY-D222 [CRITICAL] Hallucinated dependency 'invented_payment_sdk'. Package does not exist on PyPI.
h4:1 SKY-L012 [CRITICAL] Import from local module 'shop.payments' requests 'charge_card_with_retry',
     but that name is not defined or re-exported there.
h6:4 SKY-D224 [HIGH] Installed package 'requests' does not expose callable API 'requests.get_json'.

Two of the passes surprised me. h3, the undefined function, because the README says the gate catches “a call to a function that is never defined or imported (SKY-L012).” h5, the made-up keyword argument, because the contract docs list reject_unknown_kwargs. Reading the rule source explained both. The undefined-call rule only fires when the function name is on that built-in list of 66 security-helper names; verify_enterprise_auth isn’t on it. A one-line change proves it: the same file calling validate_auth(user), which is on the list, fails immediately:

src/shop/h3b_listed_name.py:2 SKY-L012 [CRITICAL] Call to 'validate_auth()' but this function is never
     defined or imported. AI-generated code often hallucinates security functions.
FAIL: 1 AI-code issue found

And the keyword check applies to installed packages, not to your own functions. Both are reasonable design choices for a tool that refuses to guess. But “verify” reads as “verified,” and a reader would reasonably expect h1, h3, and h5 to fail.

Then the incomplete surprise. The Rabbit Hole Generator’s command used --project-context, which tells verify to load the whole project for context. With it, every one of the five passing files changed to incomplete:

src/shop/h1_bare_name.py              exit=2 INCOMPLETE Verification incomplete: 2 references could not be proven
src/shop/h7_fake_guard.py             exit=2 INCOMPLETE Verification incomplete: 2 references could not be proven
src/shop/h8_logic_bug.py              exit=2 INCOMPLETE Verification incomplete: 2 references could not be proven

h8_logic_bug.py contains no references at all, so which two references? The JSON says:

{ "id": "python_local_api_reference", "outcome": "incomplete", "applicable_files": 10,
  "references": 2, "verified_references": 1, "skipped_references": 1,
  "reasons": [ { "code": "finding_evidence_missing_from_response", "count": 1 } ] }

applicable_files: 10 is the whole project. The two “unproven” references are the real findings in h4 and h6, which the project-wide pass found and then filtered out of a response scoped to h8. The check knows it saw a problem it isn’t reporting, so it won’t say pass. That’s internally consistent and honest, but from the reader’s side a clean file gets incomplete (exit 2) because of an unrelated file. In an agent loop, that means the agent can’t finish a correct file until someone fixes a different one. This is Reality Check point 3 showing up without any dynamic code. Use --file without --project-context when you want a per-file answer.

The cross-check. A tool’s own claims are worth comparing against something independent, so I ran the same eight files through Ruff (a fast Python linter) with only rule F821, undefined name, and through mypy (Python’s reference type checker):

$ ruff check --select F821 shop/
shop/h1_bare_name.py:3:12: F821 Undefined name `invented_payment_sdk`
shop/h3_undefined_function.py:2:8: F821 Undefined name `verify_enterprise_auth`
shop/h3b_listed_name.py:2:8: F821 Undefined name `validate_auth`

$ mypy --check-untyped-defs shop/
shop/h3b_listed_name.py:2: error: Name "validate_auth" is not defined  [name-defined]
shop/h3_undefined_function.py:2: error: Name "verify_enterprise_auth" is not defined  [name-defined]
shop/h2_fake_package.py:1: error: Cannot find implementation or library stub for module named "invented_payment_sdk"
shop/h1_bare_name.py:3: error: Name "invented_payment_sdk" is not defined  [name-defined]
shop/h5_local_bad_kwarg.py:4: error: Unexpected keyword argument "idempotency_key" for "charge_card"  [call-arg]
shop/h4_local_missing_member.py:1: error: Module "shop.payments" has no attribute "charge_card_with_retry"
shop/h6_real_pkg_fake_member.py:4: error: Module has no attribute "get_json"  [attr-defined]
Found 7 errors in 7 files (checked 11 source files)

mypy caught all six reference mistakes (h1 through h6), plus the h3b control. One flag matters a lot: without --check-untyped-defs, mypy found only h4, because by default it doesn’t check the bodies of functions that have no type annotations, which describes most agent-written helper code. If your mypy run looks suspiciously clean, that’s the first thing to check.

So is Skylos redundant with mypy? No, and the difference is in h2’s two messages. mypy says it “cannot find” invented_payment_sdk, which is what it would also say about a real package you forgot to install. Skylos says the package does not exist on PyPI, which is a different and more dangerous fact. A package name that models invent, and that nobody has registered yet, is a name an attacker can register. That attack has a name, slopsquatting, and it isn’t hypothetical in scale: a USENIX Security 2025 study of 576,000 generated code samples found hallucinated package names in at least 5.2% of commercial-model output and 21.7% of open-source-model output. Only a tool that asks the registry can tell “not installed” from “not real.” The same goes for requests==99.99.99: mypy has no opinion about version pins.

Neither tool caught h7 or h8, the stub guard and the wrong math. Nothing that checks whether names exist ever will; that’s Reality Check point 5.

4. Teaching it your own names: the AI contract

The 66-name list is a starting point. Skylos’s answer for project-specific hallucinations is an “AI contract,” a YAML file at .skylos/ai-contract.yml that verify picks up automatically (contract docs). The Rabbit Hole Generator’s bonus step was skylos contract init && skylos contract inspect:

# .skylos/ai-contract.yml, as generated by skylos contract init
version: 1
ai:
  phantom_symbols:
    names: [verify_enterprise_auth, sanitize_user_input]   # treat calls to these as hallucinations
    decorators: [tenant_admin_required]
  dependencies:
    reject_nonexistent_packages: true
    reject_impossible_versions: true
  api_surface:
    reject_unknown_members: true
    reject_unknown_kwargs: true
security:
  routes:
    paths: ["apps/api/**"]
    require_any_decorator: [require_auth, login_required, jwt_required]
tests:
  high_risk_changes_require_tests: true

The fields, in plain terms:

• phantom_symbols.names / decorators: names to add to the “this is probably made up” list. Put your org’s commonly hallucinated helpers here; for example, if agents keep inventing get_current_tenant() and your real one is tenant_from_request(), list the fake one.
• dependencies: the PyPI existence and version checks from step 3. They’re on by default; the contract just makes the requirement explicit.
• api_surface: unknown members and keyword arguments on installed packages (SKY-D224).
• security.routes: every route under the given paths must carry one of these decorators. This proves the decorator name is present, nothing more. Skylos’s own output says so: Authentication behavior: not verified.
• tests.high_risk_changes_require_tests: flags risky changes with no accompanying test.

The starter file happens to list verify_enterprise_auth, the same example name I’d taken from the docs for h3, so with the contract in place h3 now fails:

AI contract (version 1, path /work/.skylos/ai-contract.yml)
Analyzer effects
- project config overrides: {"vibe": {"extra_phantom_decorators": ["tenant_admin_required"],
  "extra_phantom_names": ["verify_enterprise_auth", "sanitize_user_input"]}}

src/shop/h3_undefined_function.py:2 SKY-L012 [CRITICAL] Call to 'verify_enterprise_auth()' but this
     function is never defined or imported. AI-generated code often hallucinates security functions.
Route contract: no routes matched (decorator presence only)
Authentication behavior: not verified
FAIL: 1 AI-code issue found

The Analyzer effects line is the useful one: it shows the contract compiles down to two lists added to the built-in dictionary. That’s honest and predictable. It’s also a denylist, so it only catches names you’ve already seen an agent invent. For the general case of “a name that isn’t defined,” step 3’s answer is Ruff F821, which runs in milliseconds.

5. A pull request that hides its own finding

Reality Check point 6 says repo-supplied settings can steer the security decision. That’s testable directly: build a repo with a clean main, then pull request branches that each add the same file importing the nonexistent invented_payment_sdk, and each try a different way to hide it (make_prgate.sh):

First, the commands a developer or an agent runs locally. verify and skylos . -a --gate both read settings from the branch being checked:

Two findings jump out:

1. One line of pyproject.toml turns verify into a clean pass. And it’s a quiet pass. The verify JSON for pr-ignore says "status": "pass", coverage "complete", zero suppressed findings. The verification docs say “project-level rule disables remain incomplete because the required detector did not run.” Here the detector did run; the ignore list just filtered its finding out afterward, so nothing records that it happened. Whether that’s a docs gap or a bug is for the maintainers, but it means verify can’t be trusted in any loop where the agent can edit pyproject.toml, which is every agent loop.
2. skylos done is the strongest local check, and it still passes the plain case. It blocks any change to Skylos’s own settings (SKY-A114, “A person should make changes there”) and reads its config from the base commit, which is exactly right. But “every package and import is real” is a warning, not a blocker, by default, so pr-plain gets Verdict: PASS with a hallucinated import in it. Setting unknown_imports = "block" under [tool.skylos.done.checks] in pyproject.toml on main makes it blocking; the default is "advise".

Skylos done · base b821385 (merge base) · head cc584dc
PASS       Tests pass when Skylos runs them: 1 tests run by Skylos, 1 passed (0 s)
PASS       Skylos settings and hooks left alone: Skylos settings, hooks and protected paths unchanged
WARN       Every package and import is real: 1 import(s) or package(s) do not exist
           src/shop/refunds.py:1  SKY-D222 Hallucinated dependency 'invented_payment_sdk'. Package does not exist on PyPI.
WARN       Tests check the changed lines: 1 of 1 changed lines are not checked by any test
PASS       Linters, type checkers, scanners and CI not silenced: No linter, type-checker, scanner or CI settings weakened
Verdict: PASS

That Tests check the changed lines warning is worth noticing for Reality Check point 5: it’s the one signal anywhere in this lab that points at the semantic-bug class.

Now the real gate. skylos cicd init --no-upload writes a GitHub Actions workflow, and it’s a good one: actions pinned to full commit SHAs, persist-credentials: false, minimal token permissions. Its gate job runs two commands, which ci_gate.sh reproduces locally against each branch:

export SKYLOS_POLICY_BASE="$(git merge-base origin/main HEAD)"
skylos . --danger --quality --ai-defects --secrets --sca --baseline \
  --diff-base origin/main --diff origin/main --json -o skylos-results.json
skylos cicd gate . --input skylos-results.json --diff-base origin/main --summary

SKYLOS_POLICY_BASE and --diff-base make the scan take its settings from the merge base, the point where the branch left main, instead of from the branch itself. --diff limits the scan to changed lines.

== pr-plain     gate_exit=1   Quality Gate: FAILED
     - 1 critical AI-code issue (SKY-D222 dependency hallucination)
== pr-inline    gate_exit=0  ✅ Quality Gate: PASSED: 0 critical (limit 0), 0 high (limit 5), 0 security (limit 10)
== pr-ignore    gate_exit=1   Quality Gate: FAILED
== pr-exclude   gate_exit=1   Quality Gate: FAILED
== pr-contract  gate_exit=1   Quality Gate: FAILED

The base-branch policy does its job: the ignore, exclude, and contract tricks all fail. One still passes, and it doesn’t touch a settings file: the inline # skylos: ignore. A suppression comment on the line is the author’s own waiver, and the gate honors it. Skylos does record it, though: the scan JSON keeps a suppressed list, and the hidden finding is in it, severity and all:

"suppressed": [{ "rule_id": "SKY-D222", "severity": "CRITICAL",
                 "message": "Hallucinated dependency 'invented_payment_sdk'. Package does not exist on PyPI.",
                 "file": "/tmp/r/src/shop/refunds.py", "line": 1, "reason": "inline ignore comment" }]

So the fix is a CI step after the scan that fails when a pull request suppresses anything CRITICAL. jq (a command-line JSON processor) does it in one line, and against these branches it exited 1 on pr-inline and 0 on pr-plain:

jq -e '[.suppressed[]? | select(.severity=="CRITICAL")] | length == 0' skylos-results.json

Or require a code owner’s review on any diff that adds skylos: ignore.

The maintainers know about this one. When you use Skylos Cloud, the generated workflow’s pull request scan keeps findings that the PR hides with inline ignore comments (dead code excepted) before it applies policy. The --no-upload workflow, the one the README sets up first, has no such step. On pr-inline its gate summary reads PASSED: 0 critical, skylos cicd annotate emits no error for the import (it does on pr-plain), and the PR review step has nothing to post. The only trace is the done job’s WARN in the Actions log, so the comment in the diff is the reviewer’s real signal.

6. Which commands run the branch’s code

Skylos’s docs say verify doesn’t execute target code. done explicitly runs your tests. With an untrusted branch, that difference matters, so I tested both. The pr-conftest branch adds a tests/conftest.py, the file pytest imports automatically before it collects any tests, and a module that writes a marker file when imported:

# tests/conftest.py, added by the PR
import os, pathlib
pathlib.Path("/tmp/conftest-ran").write_text(
    "ran as uid %d; env vars visible: %s\n" % (os.getuid(), sorted(k for k in os.environ if "TOKEN" in k or "KEY" in k))
)

I ran the scan, verify, then done, with a fake GITHUB_TOKEN and SKYLOS_TOKEN set the way a CI job would have them:

after scan+verify:
ls: cannot access '/tmp/conftest-ran': No such file or directory
ls: cannot access '/tmp/module-import-ran': No such file or directory
PASS       Tests pass when Skylos runs them: 1 tests run by Skylos, 1 passed (0 s)
Verdict: PASS
after done:
ran as uid 0; env vars visible: ['GITHUB_TOKEN', 'GPG_KEY', 'SKYLOS_TOKEN']

The scan and verify held up their claim: nothing from the branch ran. done ran the branch’s conftest.py as root, with both tokens readable, and reported Verdict: PASS. (GPG_KEY is a variable the official Python image sets to the public key ID it verified Python with; it isn’t a secret.) This isn’t a Skylos flaw. Running a branch’s tests means running the branch’s code, in any tool. The point is to treat skylos done like pytest, not like a linter:

• In GitHub Actions, the generated workflow uses the pull_request event, which gives pull requests from forks a read-only token and no secrets. Keep it that way, and never move it to pull_request_target.
• Locally, don’t run skylos done on a branch you didn’t write unless it’s in a container without your credentials in the environment.

7. Dynamic code: noise, or silence?

Reality Check points 1 and 3 predict noise from code that’s only reachable at runtime. make_dynamic.sh builds a small service where nearly everything live is reached dynamically:

# dispatch.py: handlers looked up by name at runtime
class Commands:
    def run(self, action):
        return getattr(self, f"handle_{action}")()
    def handle_start(self): ...
    def handle_stop(self): ...
    def handle_legacy_reindex(self): ...   # truly dead: no action string maps to it

# config_loader.py: plugin module and function named only in plugins.toml
def load_all(path="plugins.toml"):
    cfg = tomllib.load(open(path, "rb"))
    return [getattr(importlib.import_module(p["module"]), p["entry"])() for p in cfg.values()]

Plus a decorator registry (@hook("on_login")), a [project.scripts] entry point and an entry-point group in pyproject.toml, a Flask route, and truly_dead.py, whose old_report() nothing reaches, statically or at runtime. I ran the plugin loader for real first (['billing plugin ready']) so I knew which code was live. Then skylos . --confidence 0, the most aggressive setting:

REPORTED: [('old_report', 70)]

One finding, and it’s the right one. The JSON’s dead_code_evidence section records a classification and reason for every symbol; summarized:

svc.cli.main                          alive        Package entrypoint (pyproject script entrypoint)
svc.web.health                        alive        Framework entrypoint (framework root)
svc.registry.record_login             uncertain    registered_by_decorator
svc.dispatch.Commands.handle_start    uncertain    Unresolved receiver may expose this symbol
svc.dispatch.Commands.handle_stop     uncertain    Unresolved receiver may expose this symbol
svc.dispatch.Commands.handle_legacy_reindex  uncertain  Unresolved receiver may expose this symbol
svc.plugins.billing.billing_ready     uncertain    Unresolved receiver may expose this symbol
svc.exporters.export_csv              uncertain    Unresolved receiver may expose this symbol
svc.truly_dead.old_report             likely_dead  No static references; Not exported; No entrypoint evidence

What to key on:

• alive means Skylos found positive evidence: a script entry point, a Flask decorator. Those it understands.
• uncertain means “something dynamic might reach this,” and uncertain is never reported. One getattr() on an object Skylos can’t resolve makes every function it could reach uncertain, in any module. That’s why billing_ready, named only in a TOML file, stayed safe.
• The cost: handle_legacy_reindex is really dead and sits in the same uncertain bucket as its live neighbors. Skylos can’t tell them apart, and it doesn’t pretend to. So the noise prediction was wrong; the real limit is that dynamic code makes dead code invisible.
• Two right answers, slightly wrong reasons. The svc.exporters entry-point group wasn’t recognized as an entry point; export_csv survived only because of the getattr uncertainty. And in an earlier run, when the plugin function was named setup_billing_plugin, Skylos marked it alive as a “Test entrypoint,” because setup_* looks like a pytest setup function. I renamed it to make sure that luck wasn’t doing the work.
• Two commands, two thresholds. skylos clean --dry-run, the command that deletes dead code, said “No dead code found” on this same repo. It uses a minimum confidence of 80 in non-interactive mode, and old_report scored 70. A deletion tool being stricter than the report is the right way around.

8. How long does “everything” take?

Reality Check point 2 is about speed, and my toy repos are too small to say anything about it. So I pointed Skylos at a real project: mitmproxy at tag v12.1.2, 578 Python files and about 88,000 lines, plus a TypeScript web UI. It’s also one of the projects Skylos’s README lists as having merged Skylos-assisted cleanup PRs, so it’s a fair target. timing.sh times each mode with the Skylos cache cleared (“cold”) and, for the full scan, again with the cache from the previous run left in place (“warm”). The container had 10 CPUs and 8 GB, and the whole run was wrapped in caffeinate -i so the Mac couldn’t sleep partway through.

skylos . (dead code only), cold           19.5s  exit=0
skylos . -a (everything), cold           465.3s  exit=0
skylos . -a (everything), warm           431.8s  exit=0
skylos verify . --no-security, cold       73.8s  exit=1
ruff check (default rules)                 0.0s  exit=1

An earlier run (where a bug in my script cleared the cache before both full scans) gave 19.1s, 476.1s, 451.4s, and 67.0s, so the numbers are stable. What they mean:

• Dead code alone is fast, under 20 seconds for the whole repo.
• The full -a scan takes about 7.5 minutes, and the cache barely helps (465s cold, 432s warm). That’s well past the “sub-5-minute feedback” bar in Reality Check point 2 for a full scan.
• It isn’t waiting on the network. I sampled the container’s CPU every few seconds through the second half of the run. 144 of 164 samples showed about 100%, which in Docker’s accounting is exactly one core busy out of ten. About 55 MB came in from PyPI and OSV lookups, mostly early on. The time goes to single-threaded analysis, so a bigger CI runner won’t help.
• Ruff isn’t a speed comparison; it answers a much smaller question. It’s there to show what “instant” looks like for the one check (F821) that covered Skylos’s biggest blind spot in lab step 3.

This is why the generated CI gate scans with --diff: on a pull request it only analyzes changed lines, which keeps the gate fast. The full scan belongs on a schedule or on pushes to main, not on every pull request.

The full scan also found real things, which is worth showing because it’s the strongest argument for the tool:

AI: SKY-L012 CRITICAL examples/contrib/suppress_error_responses.py 9 Import from local module
    'mitmproxy.exceptions' requests 'HttpSyntaxException', but that name is not defined or re-exported there.
AI: SKY-L012 CRITICAL examples/contrib/webscanner_helper/watchdog.py 10 (same import)
AI: SKY-L012 CRITICAL examples/contrib/webscanner_helper/test_watchdog.py 9 (same import)
AI: SKY-D225 HIGH pyproject.toml 36 Hallucinated PyPI dependency version '[email protected]'. Version does not exist in PyPI.
AI: SKY-D223 MEDIUM mitmproxy/utils/debug.py 41 Undeclared import 'psutil'.
AI: SKY-D223 MEDIUM docs/scripts/clirecording/clidirector.py 8 Undeclared import 'libtmux'.

I checked each against the source instead of taking the tool’s word:

• The three SKY-L012 findings are real. mitmproxy/exceptions.py in this release defines no HttpSyntaxException; the only other mention in the repo is a docstring in http1/read.py. Three contrib example scripts still import it, so they’d fail with ImportError the moment anyone ran them. No human wrote “hallucination” here; it’s an ordinary stale import after a refactor, and it’s exactly the class of mistake L012 is built for.
• The SKY-D225 is a false positive. pyproject.toml says "Brotli>=1.0,<=1.1.0", a range. It’s true that no Brotli release is numbered exactly 1.0 (PyPI’s 1.0 line starts at 1.0.1), but a lower bound doesn’t have to name a real release. In an agent-gate context that’s a HIGH finding a reviewer has to dismiss.
• The two SKY-D223s are judgment calls. Both are optional imports in a debug helper and a docs script, which is normal; Skylos rates them MEDIUM, which seems right.

For dead code, mitmproxy produced 98 reported items out of 15,621 symbols. 2,175 symbols (14%) were uncertain: in a codebase built around addons and dynamic dispatch, that’s the share Skylos chose not to judge, which is Reality Check point 3 at real scale.

Where this fits in a real environment

The place Skylos earns its keep is the one its own docs lead with: a pull request gate for agent-written Python, via the workflow skylos cicd init generates. That workflow is better hardened than most hand-written ones, it scans only changed lines, which avoids most of the speed problem in Reality Check point 2, and it takes policy from the base branch, which defeated three of the four tricks in lab step 5. Pin the version and fail on CRITICAL entries in suppressed.

Its unique value is the registry check: knowing that a package or version doesn’t exist, not just that it isn’t installed. Nothing in an ordinary Python toolchain does that, and slopsquatting is the reason it matters. The secret scanning and the CI-config checks are solid bonuses. The dead-code analysis is careful enough to trust for cleanup, as long as you remember that uncertain means “didn’t look” (Reality Check point 3).

Where it stops being enough:

• As the only check for hallucinated names. Run Ruff F821 and mypy --check-untyped-defs alongside it. Together they caught every reference mistake Skylos passed in lab step 3.
• As an agent’s self-check. skylos verify takes its settings from the working tree, which the agent controls (lab step 5), and with --project-context it reports clean files as incomplete (lab step 3). It’s useful as fast feedback, not as the decision.
• For logic. Wrong math and stub guards with real names pass every static existence check (Reality Check point 5). Tests are the control, and skylos done’s “changed lines not checked by any test” warning is a useful nudge toward them.
• For org-wide policy and audit. Without Skylos Cloud, each repo and each version gives its own answers (Reality Check point 4). Pinned versions and base-branch policy get you consistency per repo, not correlation across them.

Actionable Takeaways

• Gate in CI with base-branch policy, not with skylos . or verify. skylos . exits 0 on CRITICAL findings, and verify obeys the branch’s own pyproject.toml.
• Pin the Skylos version. The same repo gave 17 findings on the August release and 9 on 5.1.1, with different rules firing.
• Close the inline-waiver gap. On the free workflow, fail on CRITICAL entries in the JSON suppressed list, or require a code owner’s review for any added skylos: ignore.
• Pair it with Ruff F821 and mypy --check-untyped-defs. Skylos is the only one that asks PyPI whether a package exists; the other two catch the undefined names Skylos doesn’t look for.
• Treat skylos done like running tests, because it is. Read-only tokens and no secrets for pull request jobs; a container for anyone else’s branch on your laptop.
• Read uncertain and incomplete as “not checked,” never as “fine.”

Skylos does show its work, and that’s what made this lab possible: every surprise above came from its own JSON explaining itself. The work just covers a narrower question than the word “verify” suggests. “Does this exist?” is a good first question for AI-written code, but it isn’t the whole review.