Малые языковые модели Qwen игнорируют инструкции, но выглядят точными в тестах

In July 2026, researchers published a paper on arXiv showing that small Qwen language models remain accurate on tasks but systematically ignore instructions that contradict their habitual behavior — and the standard accuracy metric fully masks this failure.

What the authors tested

Instruction tuning is meant to make a model follow user requests, but it's unclear whether small models comply when a command conflicts with their usual behavior. The authors took instruction-tuned Qwen models of different sizes and confronted them with a "counter" instruction across three types of tasks.

A conflicting instruction was added to the standard task: choose a deliberately incorrect option in a multiple-choice test, produce the opposite sentiment in a sentiment analysis, or return a doubled result in a math problem. This cross-task design made it possible to check whether resistance was tied to a specific task or was a more general behavioral trait.

• Three tasks: answer selection (MCQA), sentiment classification, math questions
• Conflicting instruction: choose the wrong option, give the opposite sentiment, or double the answer
• Three metrics: standard accuracy, non-standard accuracy, and Instruction-Following Failure Rate (IFFR)
• Models: instruction-tuned Qwen of several sizes

Why accuracy is misleading

The trick in the design is that all answers are checked against the original correct solution. A model that ignored the "counter" instruction and solved the task as usual looks successful by the standard metric — even though it did not carry out the user's command. This is counterintuitive: two models with the same accuracy score on a benchmark can behave completely differently once given a command that runs against expectations. That's exactly why reporting standard accuracy alone hides failures in instruction following, and the authors introduce a separate metric, IFFR, to catch them.

What the results showed

The results diverged by model size. Small Qwen models remained competent — they solved the underlying task — but routinely ignored the non-standard instruction. Larger models showed a clear gap between the two modes: they actually switched to the "counter" command noticeably more often.

Both standard accuracy and overall instruction following grew with model scale, but, as the authors note, the pattern was inconsistent across different tasks and datasets. Larger size did not guarantee equally reliable compliance everywhere.

"Task competence and instruction following are distinct abilities, and

reporting standard accuracy alone conceals instruction-following failures," the study's abstract on arXiv states.

What it means

For those deploying small models, the takeaway is practical: a high benchmark score doesn't mean the model will do exactly what you ask. Growing task-solving ability doesn't automatically deliver reliable control over behavior, so compliance should be measured separately — with a metric like IFFR — rather than inferred from overall accuracy. For applications where controllability and safety matter, this is one more reason to test a model on conflicting commands, not just on benchmark tasks.

Frequently asked questions

What is Instruction-Following Failure Rate (IFFR)?

IFFR is a metric proposed by the authors for the share of cases where a model failed to carry out a given instruction. Unlike standard accuracy, it specifically shows failures to follow a command rather than the quality of the task solution.

Why does a model look accurate but not obey?

Because in the experiment, answers are checked against the original correct solution. If a model ignored the "counter" instruction — for example, "choose the wrong option" — and solved the task as usual, the standard metric counts this as a success, even though the model violated the user's command.