AXLE Method Primes Annotators' Own Experience Before Hate Speech Labeling: 104 People, 1,040 Ratings

A demographic questionnaire filled out when an annotator is hired is the standard way to account for their position. A new paper on arXiv (cs.CL) tests whether such a questionnaire works at all, and proposes an alternative called AXLE. In an experiment with 104 annotators, 30 examples of ambiguous hate speech and 1040 annotations, a preliminary conversation increased sensitivity to harm but noticeably reduced agreement between annotators. The authors treat this as the method's goal.

The problem with a static questionnaire

Perspectivist NLP starts from the premise that on subjective tasks, a label depends on the annotator's position. People judge toxicity, offensiveness and discriminatory subtext differently, and averaging these judgments into a single "correct" answer is not always fair. In practice, though, position is captured once: age, gender, religion, migration experience. These data enter the dataset as a covariate. This rests on an assumption that the knowledge the questionnaire indexes is actually active in a person's mind when they read a specific example. The authors call this assumption a design problem and propose solving it at the level of the task itself.

How AXLE works

AXLE adds a step before labeling. First, the annotator sees a scenario vignette that sets the context. Then comes a short conversation guided by "axes", that is, predefined dimensions of experience. From the answers, the system synthesizes a first-person micro-narrative and returns it to the annotator before they see the first example. The person reads their own experience, reassembled into a coherent text, and only then moves on to the labels.

What the experiment showed

The design is a 2x2 between-subjects setup: 104 participants, 30 examples, 1040 annotations, or an average of 10 ratings per person. The material is ambiguous hate speech on three topics: immigration, religion and gender. The authors write cautiously about association rather than proven causality.

• Identification resonance with the material is higher among participants who went through the elicitation step.
• Sensitivity to harm is also higher.
• Label entropy rose: responses are distributed more widely.
• Agreement between annotators fell.

The last two findings are usually read as a drop in quality. The authors interpret them differently: the step reveals pluralism rather than fabricating consensus. This is what sets AXLE apart from annotator training, where the success criterion is the opposite, namely convergence of ratings. If a task is ambiguous by nature, disagreement carries information, not noise.

What to change in the labeling pipeline

For teams that collect data for moderation and for training toxicity classifiers, the takeaway is practical. If your only quality metric is inter-annotator agreement, AXLE will show "degradation", even though the labeling has moved closer to the real spread of opinion. Before adopting such a step, decide what you want: a single label for a hard moderation rule, or a distribution of opinions for training models that account for different points of view.

In the second case, you need to store the raw labels and their entropy, not just the majority vote. The scale of the study is modest: 30 examples, three topics, one task format. So test the method first on your own material and your own pool of annotators before building it into a production pipeline.