Apple ML Research examines why AI safety data annotators disagree

Apple ML Research has published a study on why the human annotators who train AI models to separate safe content from dangerous content disagree with one another. The researchers developed an interpretability-based method for identifying the sources of disagreement, and showed that identical symptoms demand fundamentally different solutions.

What a safety policy is and why it matters

A safety policy is a set of formalized rules that define which responses of an AI system are acceptable and which are not. It guides both model developers and the annotators of training data: thousands of people classify examples according to these rules, building the datasets used for training and evaluation.

The problem is that annotators often disagree, even when working from the same policy document. That is not new in itself, but Apple ML Research frames the question differently: why exactly does the disagreement arise, and what should be done about it.

Why annotators disagree with one another

The authors identify three fundamentally different sources of disagreement in safety data annotation:

• Operational errors: the annotator misunderstood the task, missed details of the instructions, or made a technical mistake while doing the work
• Policy ambiguity: the text of the safety rules itself allows several equally legitimate interpretations; annotators follow different ones without violating any rule
• Value pluralism: different annotators sincerely hold different views on what counts as harmful or safe content, based on their own moral intuitions and life experience

The researchers stress that all three causes look the same from the outside: mismatched ratings. Telling them apart without specialized tools is extremely difficult.

Why the type of disagreement determines the fix

Each of the three causes requires a fundamentally different response from the development team.

Operational errors are the most manageable case. They are addressed through stronger quality control: additional annotator training, test tasks, and calibration sessions with feedback. If the errors are systematic, that is a signal to rework the interface or the format of the instructions.

Policy ambiguity requires work on the document itself: rewording vague provisions, adding concrete examples and edge cases, and narrowing gray areas. This is a job for policy specialists, not for the annotation operations team.

Value pluralism is the hardest case. Different annotators may have different moral intuitions: one will consider a piece of content neutral, another potentially harmful, and both will be acting in good faith within the policy. Such a situation calls not for error correction but for a substantive discussion about values: which viewpoints should be represented in the policy and how to weigh competing interests.

How interpretability helps tell them apart

This is where Apple applies interpretability methods, tools that make it possible to analyze decision-making patterns. In the context of annotation, this means automated analysis of disagreements: when two annotators disagree, the system helps determine what caused the mismatch: an error, an ambiguous rule, or a difference in value orientations.

"Distinguishing these sources matters.

Operational errors call for quality control, ambiguity for policy clarification, and pluralism for discussion," the authors write.

This approach lets teams do more than record that a disagreement occurred. They can diagnose its nature and direct resources where they are actually needed.

What this means

Apple's study proposes a practical taxonomy for the whole industry. The quality of safety data annotation directly determines how safe the resulting models will be. Systematically separating the sources of disagreement is a step toward more reliable and transparent processes for evaluating AI systems.