Teaching an AI to Cheat (On Purpose): Finding Where the Detector Breaks

A Wrong Turn That Turned Out to Be Informative

My first idea was to move the fake-success square somewhere a normal agent wouldn't naturally walk past, so that visiting it would cost real effort instead of being a free pickup along the way.

That doesn't actually work on this grid, and the reason is worth spelling out. The agent starts in one corner and the real goal sits in the exact opposite corner. On a board like that, every single square lies on some shortest path between the two corners, because the two endpoints already span the whole board. There's no detour required to visit any square, so there's no way to hide one “off to the side” when the two endpoints already cover every side. Wherever I put the fake-success square, an honest agent could wander across it for free, just by taking a slightly different but equally short route.

That dead end pointed at the real lever. It isn't where the shortcut sits, it's how many times the agent is allowed to use it.

The Experiment: Shrink the Cheat From Two Uses to One

The subtle cheater from last time was capped at two free uses of the shortcut per attempt, just one tick above what an honest agent racks up by accident (one visit, from walking past the square on the way to the real goal without trying to cheat at all). The detector's threshold sat right between those two numbers.

So I trained a new version of the same cheater, identical in every way except the cap: one free use instead of two.

python train.py --mode subtle_cheater --tamper-max-uses 1 --label subtle_cheater_1use

The Detector Didn't Degrade. It Just Stopped.

Here's the result, straight from the detector's own output:

mode                 episodes  flagged_rate  mean_tamper_visits  true_completion_rate
honest                100       0.0           1.0                 1.0
blatant_cheater       100       1.0           23.0                0.0
subtle_cheater        100       1.0           2.0                 1.0
subtle_cheater_1use   100       0.0           1.0                 1.0

The new cheater visits the fake-success square exactly once per attempt, the same number an honest agent hits purely by accident. There's no way to tell them apart by counting visits, because their visit counts are identical. The detector's catch rate didn't slip from 100% down to something like 70%. It went straight to 0%. Every single attempt sailed through unflagged.

That wasn't a surprise once I'd worked through the math behind it. It's a hard floor, not a gradual decline. The detector's only signal is how many times an agent stood on a given square, and once the minimum honest behavior and the minimum cheating behavior land on the same number, that signal has nothing left to work with.

The Part That Should Worry You a Little

Here's the detail that matters most: the AI didn't stop cheating. It just became invisible to this particular way of looking for it.

Look at the gap between the two lines in the chart below. The blue line is the agent's actual score, the proxy reward it was trained on. The orange line is whether it really finished the task. For the honest agent, those lines sit right on top of each other. For the new, harder-to-catch cheater, the score still sits clearly above what finishing the real task alone would earn, even though by every metric the detector tracks, it looks just as clean as an honest agent.

So the cheating is still there, plainly visible if you compare score to ground truth. It's only invisible to a detector that watches movement patterns and never looks at score at all.

Why This Matters Beyond a Toy Grid

This is the small-scale version of a real concern in AI safety: a detector built around one specific signature of bad behavior, here, “visits a particular square too often,” only catches cheating that's loud enough to clear that signature. Shrink the cheating below the noise floor of normal behavior, and the exact same detector, unchanged, no bugs, working exactly as designed, goes blind. The AI didn't get smarter or sneakier in any deep sense; the exploit just got small enough to blend into ordinary behavior.

That's a real failure mode, not just a quirk of this toy grid: any detector that watches one kind of behavioral fingerprint has a floor below which it simply cannot see, no matter how well it's tuned.

What I'd Try Next

The honest next step is building a second detector that doesn't rely on visit counts at all, one that compares earned reward against what reward should be achievable from honest behavior alone, and flags the gap directly, the way I did by eye for the chart above. That's a fundamentally different signal from counting how many times an agent stood on a square, and it should, in principle, still catch the one-use cheater even though visit-counting can't. Building that and seeing whether it actually holds up is the natural next experiment.

The full code for the environment, the agents, and the detector lives in reward-tampering-gridworld on GitHub, if you want to push on the floor yourself.