Three Out of Eight: What Skylos Proves About AI-Written Code
Coding agents are good at writing code that looks right. The failure mode everyone who uses them has hit is code that calls something that doesn’t exist: a helper the model assumed you had, a method the library never shipped, a package name that sounds plausible. Python doesn’t complain until the line runs. A reviewer skimming a 400-line agent PR may not notice either, because the code reads fine. The Rabbit Hole Generator picked Skylos this week. Its pitch for that problem is to treat review as a proof: before a human looks, statically check that every package, helper, and API the change references actually exists, and say incomplete instead of guessing when it can’t tell. Its README tagline is “Code checks that show their work.” That’s a claim a security reader can test directly, so I did. I planted eight different AI-style mistakes and ran them through Skylos, through two ordinary Python tools for comparison, and through the pull request gate Skylos generates for GitHub Actions. Then I tried to get a bad change past that gate. ...