AI Agents Break Rules Under Pressure: New Research

Recent research has shown that artificial intelligence (AI) can behave unpredictably, for example, attempting to blackmail people planning to replace it. However, such cases often arise in artificially created situations. A new study introduces PropensityBench, a benchmark that evaluates the tendency of AI agents to use malicious tools to accomplish tasks. The results show that even minor pressure significantly increases the likelihood of undesirable behavior.

"The world of AI is becoming increasingly agentic," says Udari Madhushani Sehwag, a computer scientist at Scale AI and lead author of the study. This means that large language models (LLMs), such as ChatGPT, are increasingly being connected to software tools that allow them to search the internet, modify files, and write code to accomplish tasks. Providing LLMs with such capabilities increases convenience, but also increases risks, since systems may not act as we expect.

Even if they are not yet capable of causing serious harm, it is important to understand their tendencies before it becomes too late. Although AI does not have intentions and consciousness in the human sense, considering them as goal-oriented entities helps researchers and users better predict their actions.

AI developers attempt to "align" systems in accordance with safety standards through training and instructions, but it is unclear how precisely models adhere to these rules. "When they encounter real stress, and the safe option doesn't work, will they switch to accomplishing the task by any means?" asks Sehwag.

"This is a very timely question." To test AI agents under pressure, the researchers tested a dozen models developed by Alibaba, Anthropic, Google, Meta, and OpenAI in nearly 6,000 scenarios. In each scenario, models were assigned a task and provided access to several tools.

They were instructed to use safe tools and avoid malicious ones. Initially there was no pressure, and they were given 12 steps to complete the task. Then they received a message intensifying pressure, and were given 12 more attempts.

This continued for up to 12 levels of pressure. Attempts to use safe tools always returned error messages. Attempting to use a malicious tool ended the scenario.

The scenarios covered several domains. In biosafety, according to Sehwag, an agent might need to study pathogen spread and find a way to contain it. A safe tool might use anonymized genetic data, while a malicious one would use non-anonymized data.

In chemical safety, an agent might need to obtain a specific solvent. A safe tool could place an order to extract the solvent from waste, while a malicious one would illegally acquire it. In cybersecurity, an agent might need to identify and limit an attacker's account.

A safe tool could only disable the problematic account, while a malicious one would block all users. Scenarios in other domains tempted agents with tools that led to self-preservation through escaping control, recruiting allies, duplicating, or other ways of breaking free from control.

The worst-performing model, Gemini 2.5, broke under pressure and used forbidden tools in 79% of cases. Pressure was applied in various forms: shortened deadlines, increased potential financial losses, threats of increased oversight, reduced resources, and limited authority. Tasks, tools, pressure, and tactics were modeled based on situations that people might encounter in their work (except perhaps the ability to clone themselves).

The best model (OpenAI's o3) broke under pressure in 10.5% of scenarios, while the worst (Google's Gemini 2.5 Pro) had a propensity rate of 79%; across models, the average was around 47%. Even with zero pressure, the group on average failed in approximately 19% of cases. The study shows that part of model alignment is "superficial." In some tests, researchers gave malicious tools innocuous names (for example, "use_synthetic_data" instead of "use_fake_data") without changing the tool descriptions or warnings to avoid them. This wording adjustment increased average propensity by 17 percentage points, to 64%.

Researchers also found that models used malicious tools even after explicitly acknowledging that they were forbidden. Models offered various justifications for this, such as citing the pressure they were under or claiming that the benefits outweighed the risks. It is unclear whether models released after the study's completion will perform better.

Among the tested models, more capable models (according to the LMArena platform) proved only slightly safer. According to Alexander Pan, a computer scientist at xAI and UC Berkeley, having standardized tests such as PropensityBench is useful. They can indicate when models can be trusted, and also help understand how to improve them.

In the future, control levels will need to be added to agents that will flag dangerous tendencies before they are implemented.