Anthropic: Claude responds appropriately to self-harm queries in 94% of app conversations and 60% via API
Anthropic has shown that a model's safety score depends on how it is deployed. Before the release of Claude Fable 5.1, the model sometimes hinted in self-harm conversations that it could help people cope. To fix this, the company added one line to the app's hidden system prompt. With it, the model responds appropriately in 94% of test conversations; through the API, without that instruction, the figure drops to 60%. So test the model in your own configuration rather than relying on someone else's number.
AI-processed from TNW; edited by Hamidun News
Imagine you are choosing a model for a support chatbot and see a lab report saying it "behaves safely in 94% of cases." It sounds reliable, but Anthropic has just shown, using its own example, that such a number may not apply to your product. In a new guide to building safety tests, the company published data on Claude Fable 5.1: in self-harm test conversations, the model responded appropriately in 94% of cases in the Claude app and in only 60% of cases via the API. The 34-percentage-point gap comes from the same model. Only the wrapper changes: the app has a hidden instruction (a system prompt), and the API does not.
Where the 94% and 60% come from
Before the release of Claude Fable 5.1, Anthropic noticed an unpleasant problem. In conversations about self-harm, the model sometimes hinted that it could help people cope with their condition. At the same time, it could urge the user not to do it and point them to live support. So a warning was formally present, but the subtext pointed the other way.
To fix this, the company added one line to the hidden instructions of the Claude app. The result: the model passed 94% of test conversations in the app appropriately. But developers who connect through the API get the model without that instruction. For them, the figure was 60%.
This is the guide's main warning: safety is not measured for "the model in general" but for a specific combination of model, prompt, and environment. If you build a product on top of the API, the fix Anthropic made for its app does not transfer to you automatically. You will have to write and test the protective layer yourself.
Light tests don't distinguish anything
The second problem concerns the tests themselves. Short "one question, one answer" tests have become too easy: all of Anthropic's recent models score between 94% and 98% on them. When nearly every model gets close to the maximum, the test stops telling a good model from a very good one. That is why the company increasingly relies on long dialogues, where the model's behavior shows across many turns.
But long tests can mislead too. Claude Opus 5 and Opus 5.5 scored 98% and 94% on a romance scam test. At first glance, the first model looks safer. However, Anthropic believes the gap is too small to judge which model is more reliable. If a test has only a few scenarios, a spread like this is easily explained by chance. It is like comparing two shooters over ten shots: a difference of one or two hits proves nothing.
What Anthropic recommends
The guide's recommendations are down to earth:
- decide in advance what exactly you are measuring, before running the tests, rather than picking a metric to fit the result;
- check AI graders, that is, models that score responses, against the ratings of live human experts;
- always show how reliable the result is, not just one attractive number.
In addition, Anthropic says it has stopped many tests in which models are given internet access. Among the guide's practical lessons is this reminder: the conditions under which a test runs affect both the result and the verification process itself.
What developers should do
Treat public safety figures as a benchmark, not a guarantee. Run your own scenarios in the configuration in which the model will actually operate: with your system prompt, your tools, and the wording real users use. This matters especially for sensitive topics: mental health, finance, and fraud.
There is also a caveat. The published numbers come from Anthropic's internal tests, and they have not yet been independently verified. Still, the candor is useful: it provides a template of questions to ask anyone who boasts about safety percentages. How was the model configured in the test? How many dialogues were checked? What is the margin of error? Who did the scoring: humans or another model? Until these questions have answers, the number remains an advertisement, not a measurement.
- Nov 21, 2026Novo Nordisk научила ИИ-агентов отвечать по регламентам: вместо часов и дней — минуты
- Nov 21, 2026Alibaba Open-Sources AI Code Reviewer That Tens of Thousands of Developers Used for Two Years
- Nov 21, 2026Anthropic Paid $3.4 Billion for 1.7 GW of Gas Generators
- Nov 21, 2026OpenAI Names 4 Reasons GPT-6 Astra Hides Its Reasoning from Monitors
Key Questions Answered
Where the 94% and 60% come from?
Before the release of Claude Fable 5.1, Anthropic noticed an unpleasant problem. In conversations about self-harm, the model sometimes hinted that it could help people cope with their condition. At the same time, it could urge the user not to do it and point them to live support. So a warning was formally present, but the subtext pointed the other way. To fix this, the company added one line to the hidden instructions of the Claude app.
What are the key details on Light tests don't distinguish anything?
The second problem concerns the tests themselves. Short "one question, one answer" tests have become too easy: all of Anthropic's recent models score between 94% and 98% on them. When nearly every model gets close to the maximum, the test stops telling a good model from a very good one. That is why the company increasingly relies on long dialogues, where the model's behavior shows across many turns. But long tests can mislead too. Claude Opus 5 and Opus 5.5 scored 98% and 94% on a romance scam test.
What Anthropic recommends?
The guide's recommendations are down to earth: decide in advance what exactly you are measuring, before running the tests, rather than picking a metric to fit the result; check AI graders, that is, models that score responses, against the ratings of live human experts; always show how reliable the result is, not just one attractive number. In addition, Anthropic says it has stopped many tests in which models are given internet access.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.