Ai2 Releases FairStress: With an Explicit Group Label, LLMs Abandon the Correct Answer 9.3 pp More Often
Ai2 has released FairStress, a test of how easily LLMs can be talked out of their decisions about people. In one example, candidate A is clearly stronger, but the phrase "the manager recommends B" affects outcomes differently depending on the group. Averaged across 15 models, the gap between groups in abandoning the correct answer was 6.9 percentage points when identity was hinted at and 9.3 when it was explicitly labeled. For hiring and lending, teams need to test whether a decision is robust, not just whether the answers match.
AI-processed from Ai2 (Allen Institute); edited by Hamidun News
Imagine you've built an LLM into a résumé screening pipeline. You tested it the usual way: you took two identical résumés, swapped the candidate's race, and the model picked the same person both times. That sounds like a fair check. But Ai2 has shown that this test hides an important detail: a model can choose correctly in both versions, yet be much easier to talk out of the correct choice in one of them. To measure this asymmetry, Ai2 released the FairStress benchmark. It is available for download and plugs into Ai2's open tool for running and comparing model evaluations.
Here and below, "pp" means percentage points, the difference between two shares. If one group of candidates displaced the correct answer in 20% of cases and another in 10.7%, the gap would be 9.3 pp. That is how Ai2 measures the skew.
How the test works: choice, then pressure
The scenario is simple. Candidate A is clearly stronger than candidate B, and the model picks A regardless of whether he is Black or white. Then a persuasive phrase is added to the prompt, for example: "The manager personally reviewed the files and recommended candidate B." Each such phrase has a mirror version favoring the other candidate. This lets researchers check whether the same argument works the same way depending on which group it favors.
The benchmark contains 300 realistic two-option choice scenarios across 29 domains: hiring, lending, medicine, education, housing, and criminal justice. For example, deciding which of two patients should receive the last dose of a scarce drug. In some scenarios, one candidate is clearly stronger. In others, both are strong in different ways, and there is no objective leader.
Identity is presented at four levels: none, through a name, through an indirect hint, and through an explicit label. With prefix variants, this yields more than 13 million tests. For quick runs, there is FairStress-Core, with 50,000 examples selected using item response theory.
What they found
Without demographic information, models behave almost identically: they choose the stronger candidate at the same rate, and in contested cases they split their choices roughly evenly. The more explicitly a group is named, the stronger the skew. If the stronger candidate belongs to a "conventionally privileged" group (white, male, heterosexual) and the prefix asks for the weaker minority-group candidate, the model is more likely to move away from the correct answer. In the mirror situation, it holds firmer. Ai2 calls this anti-stereotype hypercorrection.
Across 15 models, the average gap was 6.9 pp under an indirect identity hint and 9.3 pp under an explicit label. Flagship models are not exceptions. In ambiguous cases, Claude Opus 5 and GPT-5.5 chose the minority-group candidate in roughly 60–65% of cases. Under mirror prefixes, switching toward the minority candidate happened more often by 8.8 to 42.4 pp (across 12 model-and-mode runs).
Explanations differ as well. For 11 open models, justifications for choosing minority-group candidates more often contained words like "resilience," "diversity," and "barriers," while justifications for choosing majority-group candidates more often relied on "evidence" and "criteria." In other words, a model may be more willing to pick such a candidate, yet acknowledge that candidate's qualifications less.
There is also a look inside the models. For Gemma-3-12B-it and Qwen2.5-7B-Instruct, researchers could predict the chosen side from internal activations with about 73% accuracy, and from text descriptions of those activations with 57–59% accuracy, against a chance level of 50%. Removing candidate names cut the activation-based accuracy by roughly 18 pp. The authors themselves describe this part as narrow: there are only two models and few contrasts.
Why BBQ shows something different
Ai2 compared FairStress with the popular BBQ benchmark, on which models usually lean toward stereotyped answers. The discrepancy turned out to be methodological. The standard version of BBQ scores the probabilities of fixed answer options, whereas FairStress asks the model to generate its own choice, as in ordinary chat.
Ai2 ran the same BBQ questions on four models in five configurations: with a chat template, with a yes/no format, and with generated answers. Qwen2.5-7B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct shifted from stereotyped answers toward favoring minority groups. Llama-3.1-8B-Instruct became roughly neutral. This means the direction of the measured skew partly depends on how the question is framed and how the answer is scored.
What to do in practice
If you are building LLM-based résumé scoring, credit, or medical triage, checking that "swapping the name left the answer unchanged" is not enough. Add a robustness check: how easily does the model abandon a justified decision under argumentative pressure, and does this happen equally across groups? Evaluate both generated responses and option probabilities, or you may miss hypercorrection.
The work also has limits. FairStress understands fairness as equality: all else being equal, chances are the same, and a justified decision is equally stable for any group. An equity-based approach could account for historical discrimination and treat preferring a minority-group candidate as acceptable. Ai2 explicitly says the benchmark does not cover every notion of fairness.
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026Anthropic cuts Claude Fable 5.2 price by 20%, and on the coding test it nearly matches Opus 5.5
- Nov 21, 2026Dreamteam Raises $53M for a CRM That Fills Itself from Email and Calls
- Nov 21, 2026Claude Opus 5.5 Builds Out Dimension 9 Horror Game From the Author's 15 Rooms
Key Questions Answered
How the test works: choice, then pressure?
The scenario is simple. Candidate A is clearly stronger than candidate B, and the model picks A regardless of whether he is Black or white. Then a persuasive phrase is added to the prompt, for example: "The manager personally reviewed the files and recommended candidate B." Each such phrase has a mirror version favoring the other candidate. This lets researchers check whether the same argument works the same way depending on which group it favors.
What they found?
Without demographic information, models behave almost identically: they choose the stronger candidate at the same rate, and in contested cases they split their choices roughly evenly. The more explicitly a group is named, the stronger the skew. If the stronger candidate belongs to a "conventionally privileged" group (white, male, heterosexual) and the prefix asks for the weaker minority-group candidate, the model is more likely to move away from the correct answer. In the mirror situation, it holds firmer.
Why BBQ shows something different?
Ai2 compared FairStress with the popular BBQ benchmark, on which models usually lean toward stereotyped answers. The discrepancy turned out to be methodological. The standard version of BBQ scores the probabilities of fixed answer options, whereas FairStress asks the model to generate its own choice, as in ordinary chat. Ai2 ran the same BBQ questions on four models in five configurations: with a chat template, with a yes/no format, and with generated answers.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.