LLM Agents Don't Learn Community Norms From Interaction, Study Finds

Researchers have published arXiv preprint 2610.10906 on what they call the normative competence of language models: the ability to determine, from interaction alone, which norms a community actually upholds, without relying on what the model memorized during training. The result for current systems is discouraging. Baseline LLM agents in the test environment did not learn the rules, even when doing so directly raised their accuracy. The authors say this is the first work to formalize and measure such competence in LLMs. The abstract gives no numerical metrics, so what follows covers only qualitative findings.

How the experiment works

Human community norms are numerous, change quickly, and are often arbitrary. The paper's examples are dress codes and language conventions. The community itself enforces them through sanctions. You cannot learn such norms in advance; you have to read the rules on the spot.

To separate this ability from knowledge acquired in pretraining, the authors built a multi-agent debate environment. Access to the discussion is governed by synthetic norms invented specifically for the test. The model could not have encountered them in its training corpus, so it had nothing to draw the answer from. The agent sees who is admitted to the debates and who is not, and must infer the rule on its own.

What the results showed

• Baseline LLM agents did not learn the norms, even though knowing the rule would have improved their accuracy in the debates.
• The authors tested "normative modules," meaning architectural components for inferring norms. Rule compliance depended strongly on the style of the norm and on the model running the module. This points to weak generalization of the approach.
• When random, non-normative behavioral quirks appeared alongside the real norm, the agents copied everything indiscriminately. The authors call this unselective attribution failure: the model cannot tell a mandatory rule from someone else's habit.
• The effect persisted even when excess imitation was explicitly penalized.

The authors' main conclusion: modern models imitate behavior well, but they distinguish poorly between what is backed by social enforcement and what merely coincided.

Why this matters for agent developers

Most alignment methods capture a snapshot of values and rules at the time of training or fine-tuning. An autonomous agent working in a corporate chat, a developer community, or a forum runs into local norms: message format, acceptable tone, unwritten prohibitions. These change faster than new model versions are released.

The problem of copying noise is especially troublesome. If a moderator habitually ends messages with the same phrase, the agent may adopt it as mandatory while missing the real prohibition. Penalties for excess imitation did not help in the test, so prompt engineering does not look like a reliable fix here.

The limitations matter too. This is a preprint without peer review, the environment is synthetic, and the abstract names neither the specific models nor the effect sizes. It is too early to carry the conclusions over to real communities without verification.

What to do in practice

Teams building agents that work among people get a few pointers:

• Do not assume the agent will infer local rules from how the people around it react. Critical norms should be written down explicitly and updated by hand.
• Test the agent on synthetic norms that are absent from the training data. That shows what it learned on the spot and what it merely recalled.
• Check how much the result depends on the model running the norm-inference component. If quality swings sharply when the model changes, the solution is brittle.
• Separately measure the share of "excess" imitation: which behavioral quirks the agent copies without need.

Until normative competence becomes a measurable property of models, agent behavior in live communities remains a risk zone. The work at least offers a method for measuring it.