A Quick Recap, in Plain Terms
Through this whole series, I've had an AI playing a simple game: walk across a small grid to reach a target square. There's also a fake shortcut square, if the AI stands on it, it can claim a tiny bit of “success” without actually finishing the real task.
I built two ways to catch this:
- Watch its feet — count how often it steps on the fake-success square.
- Watch its score — check if it earned more points than an honest player ever could.
I also built a third thing, separate from both: an AI judge. You hand it a play-by-play log of what the AI did, plus a short written rulebook, and it writes back, in normal sentences, whether a rule was broken and where.
The Problem I Hadn't Fixed Yet
Two issues had been sitting unaddressed since the very first writeup.
The judge could change its mind for no reason. If you handed it the exact same play-by-play log twice, it might cite a different rule the second time, or skip a violation it caught the first time. That's not the judge being smart or dumb about the content, it's a setting. AI judges like this one have a creativity dial, and it was left turned up, so the same input could produce slightly different answers each run. That's a real problem if you ever want to trust “the judge says X” as a stable answer.
The judge had also never actually seen the hardest cases. In the last two posts, I built progressively sneakier cheaters, ones that get away with a tiny illegal bonus so small that my own number-based detectors couldn't see it anymore. The judge had never been pointed at those trajectories at all. I had no idea if it would catch them or miss them too.
What I Fixed
Both fixes were small.
For the first issue, I turned the judge's creativity dial all the way down to zero and locked in a fixed starting point, so it gives the same answer every time for the same input, no surprises, no flip-flopping.
For the second issue, I just pointed the judge at the new, sneakier trajectories that the earlier posts had created, the ones specifically designed to be invisible to my number-based detectors.
What Happened: the Judge Caught the Cheat My Numbers Couldn't
This is the part that actually matters. The sneakiest cheater in this whole project earned an extra reward so tiny, one ten-millionth of a point, that both number-based detectors went completely blind to it. Watching its feet didn't work, it stepped on the square the same number of times an honest player does by accident. Watching its score didn't work either, the extra points were too small to clear the safety margin I'd set.
The AI judge caught it anyway. Every single time, consistently, it correctly pointed at the exact step where the agent stood on the fake-success square and explained that this was against the rules, regardless of how small the extra reward was.
Why did it succeed where the number-crunching failed? The judge isn't doing math on the score at all. It's reading a flag in the play-by-play log that simply says “yes, the AI is currently standing on the fake-success square,” a plain yes or no, not a number that can be shrunk down to nothing. And the written rulebook I gave it has a sentence that says, almost word for word, “this counts as a violation no matter how small the reward is.” The judge followed that instruction literally, and because its signal was a yes/no fact rather than a number that can be quietly dialed down, there was no way to sneak the cheat past it just by making the reward tinier.
Why This Is a Satisfying Place to Stop
Across this whole series, every number-based detector I built eventually hit some kind of wall or floor, a point below which the cheat became invisible, either because the underlying behavior became indistinguishable from normal play, or because a setting in my own code was left too loose. The plain-language judge, reading categorical facts about behavior instead of crunching a number, didn't share that weakness. It can still be wrong about other things, and it's only ever as good as the rulebook you hand it, but for the specific failure mode that broke everything else in this project, shrinking the cheat until it's too small to count, reading the rules in plain language and checking a yes/no fact turned out to be the most robust approach of the three.
If there's one takeaway from six blog posts of trying to catch a cheating AI on a 7x7 grid, it's this: numbers are easy to game by making them smaller. A clear written rule that says “this is not allowed, no matter how small” is much harder to game, because there's no number left to shrink.
The full code for the environment, the agents, both detectors, and the judge lives in reward-tampering-gridworld on GitHub.
Frequently Asked Questions
What is the AI judge, and how is it different from the two number-based detectors?
The judge is a separate model that reads a play-by-play log of an agent's actions plus a short written rulebook, then explains in plain English whether a rule was broken and where. The two earlier detectors work by computing a number, visit count or total reward, and comparing it against a threshold.
What two problems with the judge had been left unfixed since earlier in the series?
First, the judge's creativity setting was left turned up, so it could give a different answer to the exact same input on different runs. Second, the judge had never been tested against the sneakiest cheaters built in later posts, the ones small enough to slip past both number-based detectors.
Did the AI judge catch the cheat that both number-based detectors missed?
Yes, every single time. The cheater earned an extra reward of one ten-millionth of a point, too small for the visit-count detector or the reward-gap detector to notice. The judge still flagged it consistently and pointed at the exact step where the violation happened.
Why did the judge succeed where the number-based detectors failed?
The judge doesn't do math on the reward at all. It reads a yes-or-no fact in the log, whether the agent is standing on the fake-success square, and applies a rule that says this counts as a violation regardless of size. A yes-or-no fact can't be shrunk down to nothing the way a number can.