Sociolinguistics Instead of LLM Judges: Testing User-Simulator Personas on 5 Agents

User simulators have become a working tool for testing agents: there are not enough live people for tests, and the number of commercial agents keeps growing. The problem is that the simulators differ noticeably from real people, and their fidelity is usually judged by another LLM, which is expensive and subjective. The authors of a new arXiv preprint (cs.CL, 2610.10988) propose taking the judge out of the process and measuring persona fidelity with deterministic methods. They ran a pilot experiment on five customer-support agents.

A persona as a set of style frequencies

A user persona is usually described in words: "an elderly customer, annoyed, writes briefly." The model decides on its own how to turn the label into behavior, and the result is hard to verify. The researchers take a different route and draw on sociolinguistics. Here a persona is a social type that shows up in observable language style. So they define it as concrete style indicators, that is, frequencies of linguistic features.

This representation makes it possible to use two long-established tools that do not need a trained evaluator model:

• stylometry for authorship verification: it checks whether texts from different personas can be told apart by style;
• lexical content analysis: it counts how closely the replies match the specified frequencies.

Both methods become diagnostics of fidelity. They are reproducible and cheap, and they produce numbers rather than another model's opinion.

What the A/B test showed

The sociolinguistic scheme was compared with a flat descriptive baseline, in which the persona is written as an ordinary text description. The test was run on five task-oriented support agents. According to the abstract, the scheme improved both style match and stylometric distinguishability for most of the models tested. That means for some models there is no effect or a weaker one, and the abstract does not say which.

The authors themselves add a caveat: a persona's stylistic fidelity is not the same as its "naturalness." A reply can hit the specified frequencies precisely and still sound machine-written. For anyone planning to replace live testers with simulators, this is an important limitation. The metrics show how well the specified style was followed, but they do not guarantee that a human would have written the same way.

Another point in the paper concerns how to use the metrics. They are most useful for error attribution: you can see at which level the style breaks down, for example in vocabulary, sentence length, or punctuation. This is a step toward interventions that bring a simulator closer to the real diversity of users' language.

How to apply this in agent testing

The work is a pilot and so far only a preprint, without peer review, and a sample of five agents is small. But the practical takeaway for teams building eval pipelines for agents is clear:

• Describe test users with numbers. The share of abbreviations, typos, and politeness formulas, along with reply length, can be verified; a "somewhat rude customer" cannot.
• Add a deterministic check layer alongside the LLM judge. It is cheaper and lets you catch simulator regressions when the model changes.
• Break down failures by style component. If a persona "blurred" on vocabulary rather than syntax, you need to fix a specific piece of the prompt.
• Do not mistake style accuracy for realism. Final validation still requires a sample of real dialogues or human evaluation.

If the approach holds up across more models and domains, it will give the industry a cheap, reproducible yardstick for one of the weakest spots in agent evaluation.