There's a prompt going around right now — from Paweł Huryn's excellent Fable 5 guide — that points a strong model at your own instruction files and asks it to find the rot: the stale rules, the contradictions, the advice written for a dumber model. It's a genuinely good idea, and we wrote about it last week. So this week we did the obvious thing: we ran it on ourselves.
What we learned wasn't what we expected. The most valuable finding was that half the findings were wrong — and that the only thing standing between "wrong" and "committed bug" was a verification step a hurried agent would have skipped.
Here's exactly what happened, false starts included.
We pointed it at Elyra's own AGENTS.md — the 240-line rulebook our agents follow. It came back with a clean, confident list. Two items looked like real bugs:
A stale label list. The rules told agents to tag issues with one of five pkg:* labels — written back when the monorepo had a handful of packages. It now has thirty-six. Genuinely out of date.
A broken doc reference. The rules pointed at docs/providers.md. We checked the docs folder: only providers.html exists. Broken link, two places. Obvious fix — change .md to .html.
Finding #1 was real. We fixed it: the rules now point at ls packages/ instead of a frozen list.
Finding #2 was confident, plausible, and completely wrong.
The "broken" docs/providers.md reference was fine. We just misread it.
The instruction lived in a section explicitly scoped to the packages/coding-agent/ directory — and packages/coding-agent/docs/providers.md exists, exactly where the rule said it would. The repo has two docs folders: HTML for the website at the root, markdown for the package alongside the code. The audit (and, briefly, we) saw docs/providers.md, glanced at the root folder, found only .html, and declared a bug.
Changing it to .html would have pointed the rule at a file that doesn't exist — turning a correct instruction into a broken one. We caught it only because, before committing, we ran one boring command: ls packages/coding-agent/docs/. There it was. We reverted the "fix."
The audit didn't lie. It pattern-matched a real-looking problem and stated it with the same confidence as the real one. Nothing in its tone distinguished the true finding from the false one. That's the trap.
The seductive version of "have a strong model audit your rules" ends with and then let it fix them. That's where it goes wrong. An audit is a list of hypotheses, not a list of bugs — and a model that auto-applies its own hypotheses doesn't clean your codebase, it edits it toward plausible-looking wrongness. The more confident the model, the more dangerous the blind apply.
This is not a hypothetical we're warning you about from the sidelines. It happened to us, today, on our own repo, on the second of two findings. The honest score was one real bug, one false positive — a 50% hit rate on a confident list. If we'd let the agent run unsupervised "fix everything," we'd have shipped a regression while feeling productive.
It's also, frankly, why our own rulebook says what it says. "Read files in full before wide-ranging changes." "Always ask before removing functionality." Those lines look like bureaucracy until an audit hands you a confident, wrong diff and you watch the verification step earn its keep.
If you try Huryn's prompt — and you should — here's the discipline that makes it useful instead of dangerous:
Report first, fix never (in the same breath). Make the model produce findings, not edits. Decide separately.
Treat every finding as a claim to verify, not a task to execute. The cheap ones (ls, grep, open the file) catch the false positives fast.
Watch for confident reads of relative paths, "missing" files, and "contradictions" — these are where pattern-matching outruns truth.
Let a human — or at least a separate verification pass — own the delete key. This is exactly what /review and a budgeted /goal are for: a second look that isn't invested in being right.
The division of labor Huryn nailed is the model reads, you delete. We'd add one word: the model reads, you verify, you delete.
Auditing your instruction files is worth doing. Strong models really do catch rot that weaker ones wrote and couldn't see. But the headline isn't "the AI found my bugs." It's quieter and more useful:
A confident finding and a correct finding look identical until you check. The teams that win the agent era aren't the ones who trust the audit. They're the ones who run ls before they believe it.
We caught ours. Run the command before you believe yours.
Happy building — and happy verifying.