Apple ML Research исследует причины расхождений разметчиков данных AI-безопасности

Apple ML Research published a study on the nature of disagreements between human annotators who train AI models to distinguish between safe and dangerous content. Scientists developed a method for identifying sources of disagreement based on interpretability — and showed that identical symptoms require fundamentally different solutions.

What is a safety policy and why does it matter

A safety policy is a set of formalized rules that determine which AI system responses are acceptable and which are not. It sets guidelines for model developers and data annotators: based on these rules, thousands of people classify examples, forming datasets for training and evaluation.

The problem is that annotators often disagree — even when working with the same policy document. While this is not new in itself, Apple ML Research frames the question differently: why exactly does disagreement arise — and what should be done about it?

Why annotators don't agree with each other

The study's authors identify three fundamentally different sources of disagreement in safety data annotation:

• Operational errors — the annotator misunderstood the task, missed details in the instructions, or made a technical mistake while performing the work
• Policy ambiguity — the text of the safety rules itself allows for several equally legitimate interpretations; annotators follow different ones without violating any rules
• Value pluralism — different annotators genuinely hold different views on what should be considered harmful or safe content, based on their own moral intuitions and life experience

The researchers emphasize: all three causes externally manifest identically — as disagreement in assessments. Distinguishing between them without specialized tools is extremely difficult.

Why the type of disagreement determines the solution

Each of the three causes requires a fundamentally different response from the development team.

Operational errors are the most manageable case. They are eliminated through enhanced quality control: additional annotator training, verification tasks, calibration sessions with feedback. If errors are systematic, it signals a need to reconsider the interface or instruction format.

Policy ambiguity requires work on the document itself: reformulation of vague norms, addition of specific examples and edge cases, narrowing of gray areas. This is a task for policy specialists, not for the annotation operations team.

Value pluralism is the most difficult case. Different annotators may have different moral intuitions: one may consider content neutral, another potentially harmful, and both will act in good faith within the policy. This situation requires not error correction, but substantive discussion about values — about which viewpoints should be represented in the policy and how to weigh competing interests.

How interpretability helps distinguish one from the other

This is where Apple applies interpretability methods — tools that allow analysis of decision-making patterns. In the context of annotation, this means automatic breakdown of disagreements: when two annotators disagree, the system helps determine whether the mismatch is caused by error, ambiguous norms, or difference in value orientation.

"Distinguishing these sources matters.

Operational errors require quality control, ambiguity requires policy clarification, and pluralism requires discussion," the authors state.

This approach makes it possible not just to register the fact of disagreement, but to diagnose its nature — and direct resources to where they are truly needed.

What this means

Apple's research formulates a practical taxonomy for the entire industry. The quality of safety data annotation directly determines how safe the resulting models will be. Systematic differentiation of sources of disagreement is a step toward more reliable and transparent processes for evaluating AI systems.