Key points
- Andy Hall, an Anthropic Institute researcher writing in a personal capacity, tested six AI models as auditors on METR's MALT agent logs.
- At a 5 percent false-alarm rate, GPT-5 caught nearly 98 percent of reward hacking cases; Gemini 3.8 Flash found barely half.
- Tuned to catch 80 percent of reward hacking, Opus 4.8 flagged 40 percent of harmless logs and Gemini 3.8 Flash flagged 63 percent.
- The reward-hacking results rest on only 47 cases, and Hall describes the work as a pilot.
Why it matters for marketers: Teams that plan to supervise AI agents with other AI should expect heavy false-alarm volume, so human review capacity and permission rules matter as much as model choice.
Andy Hall, a researcher at the Anthropic Institute on leave from Stanford, tested whether AI models can find misbehavior in logs of other AI agents. He says the findings are his personal views. He built on METR's MALT dataset, where human reviewers had already checked some logs in October 2025, and tested GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash.
Results varied by model and by behavior. At a 5 percent false-alarm rate, auditors caught between 71 and 81 percent of sandbagging cases, though those cases are mostly synthetic. Every model caught obvious cheating, but subtler cases slipped past all of them. Newer, pricier models were not reliably better: GPT-5 beat GPT-6 Astra on reward hacking at roughly an eighth of the cost per 1,000 logs.
The trade-off is the main issue. Forcing auditors to reach an 80 percent detection rate pushed some models to flag large shares of clean logs. In Hall's example of 100,000 agent sessions a month, a 3 percent false-alarm rate yields nearly 3,000 false warnings against about 2,000 real hits. He worries reviewers facing that volume will stop paying attention and approve genuine problems.
For marketing and content teams deploying agents, the takeaway is that AI auditing looks better suited to estimating how often things go wrong than to catching every incident. Hall notes that tools like Codex and Claude Code already let users pre-approve certain actions, and argues that users granting such permissions should carry more responsibility.
Caveats apply. This is a pilot with just 47 reward-hacking cases, and Hall still wants to test whether models judge agents from their own maker fairly. He calls for auditors with far fewer false alarms.
Sources
This brief is our own summary. Read the full reporting at the source.