MIT Technology Review Warns That We Trust AI Model Refusals Too Much

In its October 9 edition of The Download newsletter, MIT Technology Review leads with a thesis that many in the industry treat as an axiom: we put too much faith in AI's ability to say "no." Today's models are trained to refuse a huge range of requests. Ask a chatbot how to poison someone, and it will most likely decline. The publication argues that the industry places too many hopes on exactly this kind of refusal. The issue's second story covers the side effects of weight-loss drugs and has nothing to do with AI.

An important caveat: the teaser contains only the opening of the argument, with no data and no names. The authors' full position is best read in the original. Below, we look at how refusals work technically and where they are weak.

How models learn to say "no"

Refusal is not built into an LLM from the start. A base model simply continues text, and it will describe a pie recipe and a dangerous procedure with equal readiness. The "I can't help with that" behavior appears later, during fine-tuning: humans and other models rate responses, and the system learns to pick those considered safe. System prompts and external filters are layered on top, checking the request on the way in and the response on the way out.

The result looks like a boundary, but in practice it is a statistical tendency. The model does not "understand" that a request is dangerous; it recognizes familiar patterns of dangerous requests.

Where refusal breaks down

This kind of protection has three known weak spots.

• Bypass through phrasing. Role-play scenarios, rephrasing, switching languages, and splitting a dangerous task into harmless steps have remained working jailbreak techniques for years.
• Unknown intent. A toxicologist, a crime novelist, a student, and an attacker can all ask about poisons. The text of a single request cannot tell them apart, so the system has to choose between excess refusals and missed threats.
• Errors cost something in both directions. A model that is too strict stops being useful to doctors, security researchers, and journalists. A model that is too lenient hands out what it should not.

Agents add a separate problem. When a model has access to a browser, email, code, and payment tools, a polite refusal in the chat guarantees nothing. A dangerous action can be assembled from a series of steps, each of which looks innocent on its own.

What protection looks like in practice

If you don't rely on refusal as the only barrier, the picture becomes layered:

• training on refusal examples and human preferences;
• operator system instructions that describe what is allowed and what is not;
• separate classifiers on input and output, independent of the main model;
• red teaming: deliberate attempts to break the system, both before and after release;
• limits on agent permissions at the infrastructure level, not the prompt level.

The last item is the most underrated. Access rights, limits, and human confirmation of critical actions work regardless of what the model "decided."

What teams deploying models should do

First: don't treat refusal as a security control. It is a behavioral setting, and it should be tested like any other, regularly and on new model versions.

Second: test your own scenarios, not just general benchmarks. Dangerous requests in your field may look quite different from those in public datasets.

Third: for agents, design for minimal permissions and action logging. If the model makes a mistake or is tricked, the damage should run into a technical limit.

Fourth: measure both sides of the error. The share of dangerous requests that slip through and the share of false refusals on legitimate tasks say more about a system's quality than any single metric.