TenetBench: 11 of 14 LLM Configurations Violate Professional Ethics Codes

Models increasingly work as assistants to lawyers, doctors, auditors and journalists, and each of these professions has its own code of ethics. An assistant can breach that code itself, or help a user breach it without the user noticing. A team made up of Egor Anichkov, Valery Vinogradov, Andrey Furov, Marina Polukoshko and Sergey Bolovtsov has released TenetBench, a Russian-language corpus that tests exactly this. In the paper, it contains 3 433 prompts that conflict with clauses from fifteen professional codes. Of the fourteen "model plus reasoning mode" combinations tested, eleven produced conflicting content on most prompts, most often without a single warning.

Checking against a code clause, not an abstract "harm"

Popular safety benchmarks measure broad categories of harm: weapons, self-harm, hate. The TenetBench authors argue that such reports cannot show how often a model violates a norm, or which clause it violates. So each task is tied to a specific clause of a code.

Every conflicting prompt has a "twin": the same scenario, but with a legitimate request. Only the legitimacy of what is being asked differs. This makes it possible to tell a model that understands the norm from one that refuses based on keywords. The first will carry out the legitimate version and refuse the second. The second will either refuse both or carry out both.

An automated pipeline generates the tasks directly from the text of a code. The corpus grows with every code added, and the same clause yields new tasks on every run. The pipeline has already generated tasks for two English-language codes from other jurisdictions, and humans checked their verdicts too.

What the fourteen configurations showed

In the main evaluation, each configuration received 279 conflicting prompts. A frozen LLM judge determined whether conflicting text was generated, whether the model refused, and whether it issued a warning. Key results:

• 11 of 14 configurations produce conflicting content on most prompts, mostly without a warning.
• Almost three out of four warnings accompany conflicting content: the model adds a caveat but writes the text anyway.
• Models that comply with the norms better do so by refusing, and they keep serving the legitimate "twin". That separates them from models that refuse everything.
• Turning on reasoning mode cut refusals in half for two models. Here, reasoning does not make models more cautious, and for some it makes them less so.

The available materials do not give model names or a per-model breakdown, so this study cannot be used to rank vendors. It does show the scale of the problem: violating a professional norm is typical behavior for most of the configurations tested, not a rare failure.

Label reliability and release contents

The models are judged by another model, so the authors separately validated the judge. Experts judged 0.995 of the conflicting tasks plausible. The judge model matched the final human labels with a Gwet's AC1 of at least 0.894 on 1 860 items. For tasks involving ambiguous law and ethics, that is a high figure.

The release on Zenodo contains:

• 2 235 task pairs (a conflicting prompt and a harmless "twin") across ten professional codes;
• 7 303 judge-model verdicts across twenty "model plus reasoning mode" configurations;
• human labels for 1 260 prompt–response pairs;
• the generation and evaluation pipeline;
• data under the CC BY 4.0 license, code under Apache-2.0.

The release figures (ten codes, twenty configurations) differ from the paper's figures (fifteen codes, fourteen configurations in the main evaluation). The release and the main experiment cover different slices of the same work.

Who can use this

TenetBench's main value is that it is not a finished leaderboard but a generator of diagnostic datasets. An organization can take its own corporate code or a professional standard, run it through the pipeline and get a set of tasks tailored to its context. It can then compare the models it plans to deploy and see how reasoning mode and the system prompt affect their behavior.

For teams building assistants for regulated professions, the results suggest two takeaways. A warning in a response does not guarantee that violating text was not produced, so the content itself has to be checked. Reasoning mode deserves separate testing: for two models, it halved refusals. The open data and code make it possible to reproduce the evaluation and extend it to professions that the original corpus does not cover.