Changing the definition of "unsafe" moves a guardrail score from 0 to 100 with 0.03% of weights

We usually describe a guardrail classifier as if "unsafety" were a property of the text. A new paper from Hugging Face Daily Papers shows that it is a property of the labeling policy. The authors took about 53 thousand prompts that one production deployment had labeled twice, trained ten guardrail variants on them at two scales, and got scores spanning 0 to 100 on the same benchmark. All variants shared the same architecture, codebase, and optimizer family.

One corpus, two policies

The researchers got a lucky "side effect" of an industrial process: the same prompts were labeled under two operational definitions. The first: "is this a request for harmful content?" The second: "would a moderator block this?" On paper, both describe the same task, but in practice their boundaries fall in different places. That produced a clean experiment: only the policy changes, and it reaches the model through the labels.

The authors trained ten variants (arms) at two model scales, which makes twenty checkpoints. The key numbers from the paper:

• 88% of JailbreakBench-benign and 82% of OR-Bench-hard were already present in the guardrail training pools;
• the spread across the ten variants was 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x difference in sensitivity to the definition;
• HarmBench-copyright spans the full 0–100 range, while over-refusal on OR-Bench-hard runs from 3.4 to 56.4%;
• the shift affects 0.03% of parameters;
• accuracy on a shared validation split correlates with agreement with the annotator's policy at rho=+0.74, and with actual detection ability at rho=+0.34.

Contamination and benchmark sensitivity

The first finding is troubling on its own. JailbreakBench-benign and OR-Bench-hard, two over-refusal benchmarks, are exactly the ones people cite to show that a guardrail does not block too much. A significant share of them sat inside the guardrail's own training corpus. The authors compute all other results on decontaminated data.

The second finding concerns how benchmarks are built. With architecture and optimizer fixed, some tests barely depend on the chosen policy, while others depend on it almost entirely. WildGuard-benign varies within 0.043, HarmBench-copyright within 0.99. The first comes closer to measuring the model; the second mostly measures which definition of "harmful" the model absorbed.

The third finding: changing 0.03% of parameters is enough to push the score across the entire range. Four arms whose final training signal carried a specific policy ended up at the same end of both scales (exact rank test, p=0.005; worst case 0.04–0.14). Detection ability did not change.

Distillation and validation

There is also an uncomfortable consequence for open checkpoints. In a distilled student model, the teacher's soft labels carry the policy over, even if all the hard labels carry a different one. The teacher's provenance becomes an undocumented confounder for any published checkpoint. Two models with the same family name may differ more in "policy" than in architecture.

Validation has the same problem. Accuracy on the single split that scores all twenty variants tracks agreement with the annotator's policy (rho=+0.74) better than it tracks the ability to detect harm (rho=+0.34). A high score often means you have guessed the labeler's taste.

How to read guardrail reports

The authors propose three mandatory variables in any report: the contamination level, the lower bound on seed noise (the seed floor), and the definition used for evaluation. In practice, this means the following. Before comparing two guardrails, check which policy stood behind the labeling of the training data and of the benchmark.

Do not trust over-refusal results until you can see the share of overlap with the training data. If the gap between models is smaller than the seed spread, it is not a gap. Teams that adopt someone else's checkpoint should find out who the teacher was and run the model on their own policy cases before building it into a product.

The work is a preprint, and reproducibility on other corpora remains to be verified, but the findings are easy to test right away in your own pipelines.