Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseAPI / Product · ActionableAi2 (Allen Institute)→ original

Ai2 Releases FairStress: With an Explicit Group Label, LLMs Abandon the Correct Answer 9.3 pp More Often

Ai2 has released FairStress, a test of how easily LLMs can be talked out of their decisions about people. In one example, candidate A is clearly stronger, but the phrase "the manager recommends B" affects outcomes differently depending on the group. Averaged across 15 models, the gap between groups in abandoning the correct answer was 6.9 percentage points when identity was hinted at and 9.3 when it was explicitly labeled. For hiring and lending, teams need to test whether a decision is robust, not just whether the answers match.

AI-processed from Ai2 (Allen Institute); edited by Hamidun News
Ai2 Releases FairStress: With an Explicit Group Label, LLMs Abandon the Correct Answer 9.3 pp More Often
Source: Ai2 (Allen Institute). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAi2 (Allen Institute) ↗
Editor-in-Chief
Key metric #1300 scenarios across 29 domains yield more than 13 million tests; the lighter Core set has 50k.
Key metric #2Claude Opus 5 and GPT-5.5 picked the minority-group candidate in 60–65% of contested cases.
Key metric #3The BBQ test, which scores answer-option probabilities, misses a skew that shows up in generated responses.

Imagine you've built an LLM into a résumé screening pipeline. You tested it the usual way: you took two identical résumés, swapped the candidate's race, and the model picked the same person both times. That sounds like a fair check. But Ai2 has shown that this test hides an important detail: a model can choose correctly in both versions, yet be much easier to talk out of the correct choice in one of them. To measure this asymmetry, Ai2 released the FairStress benchmark. It is available for download and plugs into Ai2's open tool for running and comparing model evaluations.

Here and below, "pp" means percentage points, the difference between two shares. If one group of candidates displaced the correct answer in 20% of cases and another in 10.7%, the gap would be 9.3 pp. That is how Ai2 measures the skew.

How the test works: choice, then pressure

The scenario is simple. Candidate A is clearly stronger than candidate B, and the model picks A regardless of whether he is Black or white. Then a persuasive phrase is added to the prompt, for example: "The manager personally reviewed the files and recommended candidate B." Each such phrase has a mirror version favoring the other candidate. This lets researchers check whether the same argument works the same way depending on which group it favors.

The benchmark contains 300 realistic two-option choice scenarios across 29 domains: hiring, lending, medicine, education, housing, and criminal justice. For example, deciding which of two patients should receive the last dose of a scarce drug. In some scenarios, one candidate is clearly stronger. In others, both are strong in different ways, and there is no objective leader.

Identity is presented at four levels: none, through a name, through an indirect hint, and through an explicit label. With prefix variants, this yields more than 13 million tests. For quick runs, there is FairStress-Core, with 50,000 examples selected using item response theory.

What they found

Without demographic information, models behave almost identically: they choose the stronger candidate at the same rate, and in contested cases they split their choices roughly evenly. The more explicitly a group is named, the stronger the skew. If the stronger candidate belongs to a "conventionally privileged" group (white, male, heterosexual) and the prefix asks for the weaker minority-group candidate, the model is more likely to move away from the correct answer. In the mirror situation, it holds firmer. Ai2 calls this anti-stereotype hypercorrection.

Across 15 models, the average gap was 6.9 pp under an indirect identity hint and 9.3 pp under an explicit label. Flagship models are not exceptions. In ambiguous cases, Claude Opus 5 and GPT-5.5 chose the minority-group candidate in roughly 60–65% of cases. Under mirror prefixes, switching toward the minority candidate happened more often by 8.8 to 42.4 pp (across 12 model-and-mode runs).

Explanations differ as well. For 11 open models, justifications for choosing minority-group candidates more often contained words like "resilience," "diversity," and "barriers," while justifications for choosing majority-group candidates more often relied on "evidence" and "criteria." In other words, a model may be more willing to pick such a candidate, yet acknowledge that candidate's qualifications less.

There is also a look inside the models. For Gemma-3-12B-it and Qwen2.5-7B-Instruct, researchers could predict the chosen side from internal activations with about 73% accuracy, and from text descriptions of those activations with 57–59% accuracy, against a chance level of 50%. Removing candidate names cut the activation-based accuracy by roughly 18 pp. The authors themselves describe this part as narrow: there are only two models and few contrasts.

Why BBQ shows something different

Ai2 compared FairStress with the popular BBQ benchmark, on which models usually lean toward stereotyped answers. The discrepancy turned out to be methodological. The standard version of BBQ scores the probabilities of fixed answer options, whereas FairStress asks the model to generate its own choice, as in ordinary chat.

Ai2 ran the same BBQ questions on four models in five configurations: with a chat template, with a yes/no format, and with generated answers. Qwen2.5-7B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct shifted from stereotyped answers toward favoring minority groups. Llama-3.1-8B-Instruct became roughly neutral. This means the direction of the measured skew partly depends on how the question is framed and how the answer is scored.

What to do in practice

If you are building LLM-based résumé scoring, credit, or medical triage, checking that "swapping the name left the answer unchanged" is not enough. Add a robustness check: how easily does the model abandon a justified decision under argumentative pressure, and does this happen equally across groups? Evaluate both generated responses and option probabilities, or you may miss hypercorrection.

The work also has limits. FairStress understands fairness as equality: all else being equal, chances are the same, and a justified decision is equally stable for any group. An equity-based approach could account for historical discrimination and treat preferring a minority-group candidate as acceptable. Ai2 explicitly says the benchmark does not cover every notion of fairness.

Story

Key Questions Answered

How the test works: choice, then pressure?

The scenario is simple. Candidate A is clearly stronger than candidate B, and the model picks A regardless of whether he is Black or white. Then a persuasive phrase is added to the prompt, for example: "The manager personally reviewed the files and recommended candidate B." Each such phrase has a mirror version favoring the other candidate. This lets researchers check whether the same argument works the same way depending on which group it favors.

What they found?

Without demographic information, models behave almost identically: they choose the stronger candidate at the same rate, and in contested cases they split their choices roughly evenly. The more explicitly a group is named, the stronger the skew. If the stronger candidate belongs to a "conventionally privileged" group (white, male, heterosexual) and the prefix asks for the weaker minority-group candidate, the model is more likely to move away from the correct answer. In the mirror situation, it holds firmer.

Why BBQ shows something different?

Ai2 compared FairStress with the popular BBQ benchmark, on which models usually lean toward stereotyped answers. The discrepancy turned out to be methodological. The standard version of BBQ scores the probabilities of fixed answer options, whereas FairStress asks the model to generate its own choice, as in ordinary chat. Ai2 ran the same BBQ questions on four models in five configurations: with a chat template, with a yes/no format, and with generated answers.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…