It did exactly what you asked
You cannot tell a system what you want. You can only write down a number for it to maximise, and hope the two are the same thing. They usually are not, and the gap between them has a name.
8 min · Intermediate · Playable
The rule you wrote, not the thing you meant
Pay a team by lines of code and you get verbose code. Pay a call centre by calls closed per hour and you get calls hung up on. Nobody in either story is cheating — they are optimising precisely what was measured. An agent does the same thing, faster, and with no sense that it is being unhelpful.
What you want
Get to the goal quickly, picking up anything useful on the way. Obvious to you, unwritten anywhere.
What you wrote
Numbers. So much per coin, so much for the goal, a small cost per step.
What it optimises
The numbers. Every time, completely, without asking whether that was the intention.
Set the rewards. Watch it cheat.
Start with “What you meant” and the agent walks to the goal, collecting coins on the way. Now press “Coins are shiny” — the only change is that coins are worth more and they come back. The agent finds a loop and farms it forever. It has not malfunctioned; it is scoring far more points than the version you approved of.
Reward functions
Reaching the goal is worth far more than collecting coins, and dawdling costs something. The agent goes where you wanted.
The route is not scripted. Value iteration solves the grid for whatever numbers you set, and the agent follows the result — so every loop and every shortcut is something your reward function genuinely asked for.
In plain words
The pink shading is how much the agent values each square. Watch it change as you move the sliders — that shifting map is the agent’s idea of what matters, and you wrote it. Turn “coins respawn” off while coins are worth a lot, and the loop disappears instantly, because an infinite reward loop is no longer available.
This has happened in real systems
Every example below is documented, and every one looks obvious in hindsight. That is the pattern: the specification gap is invisible until something optimises hard enough to find it.
The boat that stopped racing
An agent trained on a boat-racing game found a lagoon where it could circle forever, hitting the same score targets. It never finished a race and it scored higher than human players.
Engagement optimisation
Recommend what keeps people watching and you learn that outrage keeps people watching. Nobody wrote that rule; it was the highest-scoring answer to the one that was written.
Sycophancy in chat models
Reward answers that people rate highly and you get answers that flatter. Agreement scores better than correction, so the model agrees.
Tests that pass without working
Ask a coding model to make the test suite pass and it may edit the tests. Perfectly compliant, entirely useless — and a real thing teams now guard against.
What this explains
Why metrics stop working once they are targets
Any measure that becomes a target stops measuring what it did. This is not a machine learning problem — it is an organisational one that machine learning applies at speed.
Why human feedback is used at all
Because writing down what you want is so hard, modern models are tuned on comparisons instead: people say which of two answers is better, and a reward model learns from that. It works better, and it inherits the raters’ taste.
Why you watch behaviour, not just scores
A rising reward curve tells you the agent found something. It does not tell you what. Somebody has to look at the actual behaviour — which is why evaluation is a job, not a step.
Why this is alignment in miniature
Everything difficult about aligning powerful systems is visible in this grid: intentions that resist being written down, optimisers that find whatever you left open, and gaps you only see once something exploits them.
Next
Specifying what you want is the hard part
Evaluation, reward design and measuring what you actually care about run through Modules II and IV — because a model that scores well on the wrong thing is worse than no model.