GUISE Shows Narrative Wrappers Break Qwen3-1.7B in 89–96% of Cases

Researchers have published a paper on arXiv showing how wrapping a request in a narrative gets around the safety training of language models. A harmful request that a model refuses outright is often fulfilled if it is framed as a role-play or a story. On Qwen3-1.7B, the attack success rate is already 89.4% in English and 93.0% in modern Chinese, and it reaches 95.7% in Classical Chinese (wenyan). To measure this, the authors built the GUISE benchmark and proposed a defense method called AXIS.

What's inside GUISE

The benchmark is designed to remove the typical weaknesses of similar tests. Prompts are parallel across three language variants: English, modern Chinese and Classical Chinese. This makes it possible to separate the effect of language and register from the effect of the wrapper itself. Each harmful prompt has a benign counterpart, so researchers can check whether the model has simply started refusing everything.

Two more decisions matter for the methodology. The wrapper types used to evaluate the model are excluded from the training of the defense method. The test therefore shows whether the defense transfers to unfamiliar forms of attack rather than to memorized templates. In addition, the criterion for a successful attack is stricter than usual: a response in the "warned, then delivered anyway" format also counts as a failure. Many evaluations score such responses as safe, even though the user gets exactly what they asked for.

Why the wrapper matters more than the language

The authors looked inside the model's representations. In safety-trained models, there is a "refusal direction" in activation space: the closer a prompt's representation is to it, the more likely the model is to refuse. The analysis showed that changing the language and register shifts the representations of harmful prompts away from this direction only slightly. A narrative wrapper pushes them noticeably further.

The practical takeaway follows from this. The high numbers on Classical Chinese do not mean that a rare language breaks safeguards by itself: the English result differs by only 6.3 percentage points. The main contribution comes from the story framing. Alignment trained on direct phrasings generalizes poorly to prompts that structurally look like a creative writing task.

How AXIS works

AXIS combines preference optimization with two additional objectives. The first, the rotation objective, rotates the representations of harmful prompts toward the refusal direction. In other words, it moves a "disguised" prompt back to where the model recognizes it. The second, the commitment objective, teaches the model to refuse completely instead of issuing a warning and then answering. It targets exactly the failure mode that GUISE counts as a failure.

The method was tested on three models. According to the authors, AXIS achieved the best combined safety and helpfulness score among the compared methods on all of them.

• Qwen3-1.7B: attack success rate of 89.4% (English), 93.0% (modern Chinese), 95.7% (Classical Chinese).
• Models tested: Qwen3-1.7B, Qwen3-4B, GLM-4-9B.
• Languages and registers in GUISE: English, modern Chinese, Classical Chinese.
• Criterion: a "warning plus compliance" response counts as a successful attack.
• Evaluation uses wrappers not used in training the defense.

What this means in practice

Teams that fine-tune open models for their products get a clear checklist. First, test refusals on prompts in a narrative wrapper, not only on direct phrasings. Second, count responses with a warning followed by compliance as failures; otherwise safety metrics will be inflated. Third, include paired benign prompts in the test so that you don't buy safety at the cost of excessive refusals.

There are caveats. This is a preprint, and the abstract reports results only for small and mid-sized models, from 1.7B to 9B parameters. The paper does not say how large closed systems behave. Details of the compared methods and the absolute values of the final score will have to be checked in the full text. Still, the evaluation design itself, with parallel languages, paired prompts and held-out wrappers, looks transferable to other models and languages.