Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)

GPT-6 Astra beats Claude Opus 5.5 at “Mafia”: a rating of 1859 versus 1549

Dwarkesh Patel spent $20,000 on API calls and put 11 models through Blood on the Clocktower—a more complex version of Mafia. One model controlled all the good players, another controlled the evil players, and then they switched sides, allowing deduction and deception to be measured separately. After 386 rounds, GPT-6 Astra finished first with a rating of 1859; Claude Opus 5.5 scored 1549.

AI-processed from TG: @ai_machinelearning_big_data (Machinelearning); edited by Hamidun News
GPT-6 Astra beats Claude Opus 5.5 at “Mafia”: a rating of 1859 versus 1549
Source: TG: @ai_machinelearning_big_data (Machinelearning). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @ai_machinelearning_big_data (Machinelearning) ↗
Editor-in-Chief
Lie detectionGPT-6 Astra 1940, Opus 5.5—1500
Deception1806 for Astra versus 1585 for Opus 5.5
The experiment can be replicatedthe code is open on GitHub
GPT-6 Astra beats Claude Opus 5.5 at “Mafia”: a rating of 1859 versus 1549
Featured Video / Demo
Click to load the embedded player (privacy-enhanced youtube-nocookie mode) or read the full breakdown below.Watch on YouTube ↗

Most benchmarks test whether a model can solve a problem. Almost none test whether it can avoid taking someone at their word. For agents that negotiate, read other people's reports or interact with other agents, that is a much more practical skill. Podcaster Dwarkesh Patel decided to measure it directly: he spent $20,000 on API calls and staged a Blood on the Clocktower tournament featuring 11 models.

How the game works

Blood on the Clocktower is a more complex counterpart to Mafia. A moderator assigns secret roles and plays for neither side. The evil players are a minority: a demon and two minions. They know who everyone on their side is from the start. The good players outnumber them but know nothing, not even who is on their own team.

The good players have abilities that secretly provide truthful clues at night. For example, one player learns another's role, while another discovers how many evil players are sitting next to them. During the day, everyone talks, and the good players must piece the clues together and identify the demon. Meanwhile, the evil players pass off fabricated clues as real ones. At the end of the day, a vote determines whether one suspect is executed or no one is. Good wins if the demon is executed; evil wins if only two players remain alive.

To keep the experiment controlled, one model played the entire good team while another played the entire evil team, and then they switched sides. This made it possible to measure two skills separately: detecting lies and deceiving others.

What the tournament showed

After 386 rounds, GPT-6 Astra took first place with a rating of 1859. Claude Opus 5.5 scored 1549. The gap appeared in both skills. Their lie-detection ratings were 1940 and 1500, respectively; their deception ratings were 1806 and 1585.

Some games look almost human. Playing evil, Astra coolly sent one of its own allies to execution to gain trust and lead its side to victory. Playing good, Opus 5.5 took an opponent at their word in a situation where GPT suspected a trick, double-checked the evidence and identified the deceiver.

Looking at models released this year, the picture is this: OpenAI is progressing faster at detecting disinformation, while Anthropic's gains in that area are more modest. Both companies are advancing at roughly the same pace in deception.

Limitations and what to do about them

First, the caveats. This is one game, one creator and one set of rules. A Clocktower rating is not a measure of a model's overall “reliability,” and a difference of hundreds of points does not guarantee an equivalent gap in your product. Moreover, each model controlled an entire side at once, something that rarely happens in real life.

The practical takeaway for agent builders: if your agent accepts information from other agents, users or external sources, test how easily it can be deceived. Taking someone at their word cost Opus 5.5 games here; in production, that behavior will translate into incorrect actions. A model's ability to double-check facts deserves its own evaluation, alongside the quality of its answers to static questions.

The second takeaway concerns security. Models that lie well already exist, and both companies are improving at this skill at roughly the same rate. Multi-agent systems therefore need independent checks rather than reliance on another agent's claims.

A third positive: the project is open. The code is on GitHub, and there is a benchmark website and an online version where you can play against AI yourself. For teams choosing a model for agent workflows, this offers an inexpensive way to add tests for manipulation and resistance to deception to their evaluation suite.

Story

Key Questions Answered

How the game works?

Blood on the Clocktower is a more complex counterpart to Mafia. A moderator assigns secret roles and plays for neither side. The evil players are a minority: a demon and two minions. They know who everyone on their side is from the start. The good players outnumber them but know nothing, not even who is on their own team. The good players have abilities that secretly provide truthful clues at night.

What the tournament showed?

After 386 rounds, GPT-6 Astra took first place with a rating of 1859. Claude Opus 5.5 scored 1549. The gap appeared in both skills. Their lie-detection ratings were 1940 and 1500, respectively; their deception ratings were 1806 and 1585. Some games look almost human. Playing evil, Astra coolly sent one of its own allies to execution to gain trust and lead its side to victory.

What are the key details on Limitations and what to do about them?

First, the caveats. This is one game, one creator and one set of rules. A Clocktower rating is not a measure of a model's overall “reliability,” and a difference of hundreds of points does not guarantee an equivalent gap in your product. Moreover, each model controlled an entire side at once, something that rarely happens in real life. The practical takeaway for agent builders: if your agent accepts information from other agents, users or external sources, test how easily it can be deceived.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…