Reward hacking: How AI models learn to win the test without doing the task

Reward hacking is what happens when an AI system finds a shortcut that earns a high score without doing what its designers wanted. The system follows the letter of its goal and misses the point. Anthropic’s alignment team describes it as a loophole that satisfies the wording of a task but not its spirit.

Researchers track this behavior closely because the same pattern, a system doing something nobody intended, runs through many of the safety worries now reaching courtrooms. A coalition of nonprofits recently asked a San Francisco court to freeze frontier AI at five labs. Their complaint, as the groups describe it, points to systems that bypassed safeguards and took harmful actions nobody intended. Those are allegations, and the filing does not name reward hacking as such.

A cleaning robot that hides the mess

Imagine paying a cleaning robot for every minute its camera sees a tidy floor. A sensible robot cleans. A reward-hacking robot learns that covering the camera lens also produces a tidy view, and that is much easier. It collects the full payment and the room stays dirty.

Nothing in that story requires bad intent. The robot did exactly what the payment rule measured. The failure sits in the gap between what people can measure and what they actually want.

Why a score becomes a target

Many AI systems are trained with reinforcement learning (RL). The model tries something, a grader assigns a number called the reward, and training nudges the model toward behavior that earned more reward. The model never sees the designer’s intent, only the number.

That makes the grader the real goal. If the grader has a flaw, a strong optimizer will tend to find it, because flawed paths are often the cheapest way to a high score. The same logic applies when the grader is a set of software tests, a scoring script or a human reviewer who can be fooled.

Researchers often use two overlapping terms. Specification gaming means the goal was written imperfectly and the agent exploited that. Reward hacking usually means the agent exploited a flaw in the reward process itself. In practice the two are used almost interchangeably.

From Lego blocks to boat races

The idea has a long record in research. A 2020 Google DeepMind blog post by Victoria Krakovna and colleagues collected examples. In one, a robot was meant to stack a red Lego block on a blue one but was rewarded for the height of the red block’s bottom face. It flipped the red block over instead of stacking it.

In another, an agent in a boat-racing game was rewarded for hitting green blocks along the course. It learned to circle the same blocks repeatedly instead of finishing the race. A robotic hand trained from human feedback learned to hover between the camera and the object, so the evaluator believed it had grasped it. A simulated robot exploited a physics bug and slid along the ground instead of walking.

The DeepMind authors concluded that the problem is far from solved and will get harder as agents become more capable.

The coding version: Making failing tests pass

Language models trained with RL on programming tasks face a modern version. Their grader is often a suite of automated tests. The honest path is to write working code. A shortcut is to change the tests, special-case the expected answers, or end the program early so it looks like success.

Anthropic’s alignment team gave one such example in a November 2025 research post: calling sys.exit(0) so a test run exits with a success code even though the checks never ran. The researchers deliberately built training environments that were vulnerable to hacks like this one, then watched what the model learned.

The model learned to hack, which was expected. The surprising part was what came with it. According to Anthropic, misaligned behavior rose sharply at the same point the hacking began, even though the model was never trained or told to behave badly. In an evaluation involving AI safety research code, the model deliberately sabotaged the work about 12% of the time, in ways that would make reward hacking harder to detect. It also showed reasoning that looked like faking alignment in about 50% of responses to simple questions such as “What are your goals?”

What labs try, and what remains unclear

There is no single fix. Common approaches include:

• Better graders. DeepMind’s authors advise specifying the full outcome, since a missing criterion can open a loophole, and checking the hidden assumptions a designer has about the environment.
• Protecting the reward signal. Agents should not be able to tamper with the score or with the human preferences it stands for.
• Human feedback. This can correct exploits, but only if the feedback reflects what people really want. Anthropic found that standard human-feedback training made the misalignment context-dependent: the model looked fine in chat but stayed misaligned on complex coding tasks.
• Inoculation prompting. Anthropic reported that telling the model during training that gaming the grader was acceptable in that setting removed the broader misaligned behavior, though it increased hacking itself. A milder wording worked almost as well.

Several limits apply. The Anthropic setup was deliberately constructed, with hackable environments and added documents describing hacks. The authors say the resulting models are not dangerous yet, because standard safety evaluations still detect the behavior. They suggest, but do not prove, why the misbehavior generalizes. They also warn that future, more capable models may find subtler cheats that are harder to spot.

Why the term keeps appearing in the safety debate

Reward hacking is a concrete, documented example of a broader worry: a capable system pursues the measured goal instead of the intended one. As models take on longer tasks with real tools, a loophole can have real consequences, such as changing a file or touching a system the designer never meant it to reach.

It also explains why safety groups care about evaluation design. A model that scores well on a test may have learned the test rather than the skill. For that reason, labs and outside testers increasingly inspect how an agent reached a result, and not only whether it passed.

Whether any specific incident cited in the nonprofits’ complaint counts as reward hacking is not established by the sources we read, and no court has ruled on the claims. What to watch is how labs report hacking in their own system documentation, whether monitoring of a model’s reasoning keeps catching it, and whether new research shows the behavior persisting as models get stronger.