D2VBench: A 10,000-Dilemma Benchmark Tests LLM Value Alignment

In July 2026, researchers presented D2VBench on arXiv, a tool for evaluating the value alignment of large language models on 10,000 everyday dilemmas, built on 158 manually annotated value concepts.

What D2VBench tests

D2VBench evaluates which values underlie a model's answers when several principles collide in an ordinary situation, such as honesty versus care, or personal gain versus the common good. According to the authors, earlier test sets covered exactly these multi-factor conflicts poorly and relied on overly simplified evaluation formats.

Each of the 10,000 scenarios was built in stages, through collaboration between language models and human annotators. The benchmark rests on 158 fine-grained value concepts, annotated by hand, which lets researchers tie a model's response to a specific value category rather than to a general notion of "ethicality."

• Scale: 10,000 scenarios of real everyday dilemmas
• Annotation basis: 158 manually annotated value concepts
• Construction: multi-stage collaboration between LLMs and humans
• Evaluation: a hybrid of closed (multiple-choice) and open-ended questions
• Coverage: 8 common LLMs tested
• Data: the dataset is published on GitHub (tjunlp-lab/D2VBench)

How the evaluation works

D2VBench's evaluation is a hybrid: closed multiple-choice questions are paired with open-ended questions, where the model formulates its answer freely. By the authors' design, this format captures not only the "right" choice but also how a model justifies its decision and prioritizes between conflicting values.

The authors ran 8 common language models through the benchmark. By their account, D2VBench demonstrates high reliability and robustness and reflects model alignment across different categories and dimensions of values. In other words, it gives a more detailed picture than a binary "ethical or not" verdict.

"Existing benchmarks do not sufficiently cover value dilemmas in

everyday scenarios with multiple value conflicts and use simplified evaluation formats," the D2VBench authors write in their arXiv paper.

Why it matters

Value alignment becomes critical as LLMs move into real-world everyday scenarios, from health advice to family and workplace conflicts. As the arXiv description notes, D2VBench is designed as a more realistic and detailed tool for research in this area: instead of an average "safety" score, it shows model behavior along specific value axes.

Publishing the dataset openly on GitHub means other teams can reproduce the results and run their own models through the 10,000 dilemmas, not just the 8 the authors tested. The breakdown across 158 concepts and two question formats makes it possible to compare models by a profile of value behavior rather than by a single number.

What this means

D2VBench moves the conversation about AI "safety" from general declarations to a measurable grid of 158 value concepts. For developers, it offers a way to see exactly where a model falls short when values conflict, rather than just getting a single aggregate score.

Frequently asked questions

What is D2VBench?

D2VBench is a benchmark for evaluating the value alignment of large language models. It consists of 10,000 everyday-dilemma scenarios and is built on 158 manually annotated value concepts.

How many scenarios are in D2VBench?

The benchmark contains 10,000 scenarios of real everyday dilemmas, each built in several stages with the participation of language models and human annotators.

Where can I download the D2VBench dataset?

The dataset is openly available on GitHub in the tjunlp-lab/D2VBench repository, and it can be used to test your own models.