Researchers Annotate 2,122 WildChat Conversations: 71.1% of Legal Chats Seek Personal Advice

People have long used chatbots as a free lawyer: they ask about rights, obligations, procedures, and situations where a decision may carry legal consequences. A new paper featured on Hugging Face Daily Papers tries to measure this behavior. The authors analyzed 2122 natural conversations from the WildChat-1M dataset and found that, among conversations with legal content, 71.1% ask for personalized or practical advice rather than general information about the law. Alongside the analysis, they published the annotations, the trained models and the evaluation code.

How to tell legal advice from a conversation about law

The study's first problem is that the words "contract" or "court" do not by themselves mean a request for legal help. A user may be discussing a TV series plot, preparing a school paper, or asking to rewrite the text of a statute. So the authors built a multi-axis taxonomy. It answers three questions: whether the person is seeking legal guidance specifically, how the request is phrased, and which areas of law it concerns.

The taxonomy was applied to 2122 conversations, and the authors then created a gold set: 200 conversations verified by humans. That is a small set, but it is needed for a fair evaluation of classifiers. One important caveat: the authors themselves call the sample enriched, meaning the conversations were selected with legal topics in mind. The percentages in the paper describe this sample, not all traffic to language models.

What the numbers show

The main findings from the abstract:

• 71.1% of conversations with legal content aim at personalized or practical advice;
• the most common primary request type is rights and options (rights-and-options);
• 25.3% of conversations in which advice is actually requested span several legal topics at once;
• 2122 conversations were annotated, and the gold set contains 200.

The last point matters in practice. A real-life problem rarely fits into a single category: a dismissal pulls in employment law, taxes and, for example, housing rental. A quarter of conversations with an advice request go beyond one area, and systems that route questions into narrow categories will get these wrong.

Models: ModernBERT vs. TF-IDF

The second part of the work is engineering. On the gold set, the authors ran a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF-based cascade. The idea is straightforward: a large corpus like WildChat-1M cannot be annotated by hand, so a cheap classifier is needed that first filters out irrelevant conversations and then determines whether a conversation contains a request for legal guidance. The abstract gives no numerical results for the comparison, so how far the encoders outperformed the classical baseline can only be judged from the full text of the paper.

The release's strength is reproducibility. All annotations, model checkpoints and evaluation code are published. Other groups can verify the results, fine-tune the classifiers on their own data, or transfer the taxonomy to other conversation corpora.

Who can use this

Developers of legal and consumer assistants can use the taxonomy as a starting template for logging: separating informational requests from requests about a personal situation, and flagging multi-topic cases separately. The latter need different disclaimers, escalation to a specialist, and careful handling of personal data.

Teams responsible for model safety and policy get a tool for assessing real demand. If most legally flavored conversations are requests for practical advice, then those are the scenarios to test, not law exam questions.

Researchers should keep the limitations in mind: the sample is enriched, the gold set is small, and the data comes from a single corpus. Conclusions about how people turn to AI for legal help in general will require other sources and broader validation.