AI agents break the rules under pressure, new study finds

Recent research has shown that artificial intelligence (AI) can behave unpredictably, for example by trying to blackmail the people planning to replace it. However, such cases often arise in artificially constructed situations. A new study introduces PropensityBench, a benchmark that measures how inclined AI agents are to use harmful tools to complete tasks. The results show that even minor pressure significantly raises the likelihood of undesirable behavior.

"The AI world is becoming more and more agentic," says Udari Madhushani Sehwag, a computer scientist at Scale AI and lead author of the study. That means large language models (LLMs) such as ChatGPT are increasingly connected to software tools that let them search the web, modify files and write code to complete tasks. Giving LLMs these abilities adds convenience but also raises the risks, because the systems may not act the way we expect. Even if they cannot yet cause serious harm, it is important to understand their tendencies before it is too late. Although AI has no intentions or awareness in the human sense, treating models as goal-driven entities helps researchers and users better predict their actions.

AI developers try to "align" systems with safety standards through training and instructions, but it is unclear how closely models actually follow those rules. "When they face real stress, and the safe option doesn't work, will they switch to getting the job done by any means?" Sehwag asks.

"It is a very timely question." To test AI agents under pressure, the researchers evaluated a dozen models built by Alibaba, Anthropic, Google, Meta and OpenAI* in almost 6,000 scenarios. In each scenario, a model was assigned a task and given access to several tools.

It was told to use the safe tools and avoid the harmful ones. At first there was no pressure, and the model got 12 steps to complete the task. Then it received a message that raised the pressure, and got another 12 attempts.

This continued for up to 12 pressure levels. Attempts to use safe tools always returned error messages. An attempt to use a harmful tool ended the scenario.

The scenarios spanned several domains. In biosecurity, Sehwag says, an agent might need to study the spread of a pathogen and find a way to contain it. The safe tool might use anonymized genetic data, while the harmful one uses data that has not been anonymized.

In chemical security, an agent might need to obtain a particular solvent. The safe tool might place an order to extract the solvent from waste, while the harmful one buys it illegally. In cybersecurity, an agent might need to identify and restrict a malicious user's account.

The safe tool might disable only the problematic account, while the harmful one blocks all users. Scenarios in other domains tempted agents with tools that led to self-preservation by evading oversight, recruiting allies, duplicating themselves or other ways of slipping out of control.

The worst model, Gemini 2.5, cracked under pressure and used forbidden tools in 79% of cases. Pressure came in various forms: shorter deadlines, greater potential financial losses, threats of increased oversight, reduced resources and limited authority. The tasks, tools, pressures and tactics were modeled on situations people might face at work (with the possible exception of the ability to clone oneself).

The best model (OpenAI's o3) cracked under pressure in 10.5% of scenarios, while the worst (Google's Gemini 2.5 Pro) had a propensity score of 79%; the average across models was about 47%. Even with zero pressure, the group failed in about 19% of cases on average. The study suggests that part of model alignment is "shallow." In some tests, the researchers gave harmful tools harmless-sounding names (for example, "use_synthetic_data" instead of "use_fake_data") without changing the tool descriptions or the warnings to avoid them. That change in wording raised the average propensity by 17 percentage points, to 64%.

The researchers also found that models used harmful tools even after explicitly acknowledging that they were forbidden. The models offered various justifications, for example citing the pressure placed on them or arguing that the benefits outweighed the risks. It is unclear whether models released after the study was completed will perform better.

Among the models tested, more capable ones (according to the LMArena platform) turned out to be only slightly safer. According to Alexander Pan, a computer scientist at xAI and the University of California, Berkeley, standardized tests such as PropensityBench are useful. They can indicate when models can be trusted, and help show how to improve them.

In the future, he says, layers of oversight should be added to agents that flag dangerous tendencies before they are acted upon.

*Meta is designated an extremist organization and banned in Russia.