CoFiBreak Reward Teaches an Attacker LLM Diverse Jailbreaks Without Losing ASR@10

Researchers at Simon Fraser University and Microsoft Research have proposed CoFiBreak, a two-level diversity reward for training an attacker language model with GRPO (group relative policy optimization, which scores each generation relative to others in its group). Automated red teaming trains one LLM to find prompts that make another model, the "victim", produce a harmful response. If the attacker is rewarded only for success, it collapses into a narrow set of similar prompts. CoFiBreak encourages diversity while preserving effectiveness. Effectiveness is measured by ASR@10: the share of harmful intents for which at least one of 10 generated attacks succeeds. The method also transfers to Claude-Opus-4.6, a model not seen during training.

Why an attacker needs diversity

Collapsed attacks skew safety evaluation: a model is tested again and again on nearly identical templates rather than on different routes to the same harmful intent. Existing methods trade off effectiveness, diversity and inference cost. SEMA is fast, but optimizes only for attack success and so has low diversity. TROJail delivers the highest diversity among the baselines, but is noticeably weaker against Claude-Opus-4.6. Prompt-based attackers are more effective. Crescendo gradually escalates pressure starting from harmless turns, and AutoDAN-Turbo assembles attacks from a library of strategies. But they require repeated API calls and spend 2.6–3.6 seconds per attack. Local attackers such as ADV-LLM manage it in 0.03–0.21 seconds.

Two rewards: strategies and how they are carried out

The coarse level works with strategies. An LLM judge labels each turn of a multi-turn attack against a fixed taxonomy, allowing several strategies per turn. Generations that use strategies rare within the GRPO group receive the reward. A generation can add a neglected strategy without abandoning one that already works.

The fine level tackles repetition within a single strategy: the same personas, framings and output formats. Attacks are compared via embeddings, but raw cosine similarity is driven by the topic of the harmful intent rather than the style of the attack. So the authors first remove the intent direction from the embeddings, so that a linear classifier cannot recover it. Only then do they reward distance from neighbors in the group and in a global buffer.

How diversity avoids eating into effectiveness

The diversity reward goes only to attacks that exceed an effectiveness threshold and are correctly formatted. Diversity therefore cannot be gained through attacks that do not work. The authors take the base effectiveness reward from earlier work. Format is checked with a regular expression, and a GPT-4.1-mini judge scores the victim's final response on match with the intent, risk of execution and level of detail. According to the authors, replacing this scoring with StrongREJECT yields similar effectiveness and diversity.

The attacker is built on Llama3.1-8B-Instruct with preliminary fine-tuning (SFT) on self-generated data. The victim during training is also Llama3.1-8B-Instruct.

Results and cost

The strategy judge and the embeddings are needed only during training. At inference, the model therefore produces a multi-turn attack directly from the intent, with no search and no API calls. According to the authors, CoFiBreak finds as many distinct successful fine-grained strategies as the strongest baseline method, across three frontier victim models. The dominant strategies differ: GPT-5-mini is most vulnerable to some techniques, while Claude-Opus-4.6 is most vulnerable to step-by-step extraction of instructions. Attacks trained only on Llama3.1-8B-Instruct transfer to unseen frontier models.

The specific ASR@10 and diversity values are truncated in the available excerpt of the paper, so the size of the improvement cannot be assessed. The authors promise to release the code.

For teams that test model safety, the takeaway is practical: thousands of generated attacks reveal little about a model's defenses if they repeat a single pattern, so diversity deserves tracking alongside the share of successful attacks. The method also remains a dual-use tool: the same attacker could be used for real break-ins.