Small Qwen Language Models Ignore Instructions but Look Accurate on Benchmarks

In July 2026, researchers published a paper on arXiv showing that small Qwen language models stay accurate on tasks but systematically ignore instructions that contradict their usual behavior, and that the standard accuracy metric masks this failure completely.

What the authors tested

Instruction tuning exists so that a model carries out the user's requests, but it is unclear whether small models comply when a command conflicts with their usual behavior. The authors took instruction-tuned Qwen models of several sizes and confronted them with a "counter" instruction on three types of tasks.

They added a conflicting directive to a standard assignment: choose a deliberately wrong option in a multiple-choice test, produce the opposite sentiment in sentiment analysis, or return a doubled result in a math problem. This cross-task design made it possible to check whether the resistance is tied to a specific task or is a more general behavioral trait.

• Three tasks: multiple-choice QA (MCQA), sentiment classification, math questions
• Conflicting instruction: choose the wrong option, give the opposite sentiment, or double the answer
• Three metrics: standard accuracy, non-standard accuracy, and Instruction-Following Failure Rate (IFFR)
• Models: instruction-tuned Qwen models of several sizes

Why accuracy is misleading

The catch in the design is that every answer is checked against the original correct solution. A model that ignored the "counter" instruction and solved the task as usual looks successful by the standard metric, even though it did not carry out the user's command. This is counterintuitive: two models with the same accuracy score on a benchmark can behave completely differently when given a command that runs against expectations. That is why reporting standard accuracy alone hides failures in instruction execution, and why the authors introduce a separate metric, IFFR, to catch them.

What the results showed

The results diverged by model size. Small Qwen models stayed competent, solving the original task, but routinely ignored the non-standard instruction. Large models showed a clear gap between the two modes: they were noticeably more likely to actually switch to the "counter" command.

Both standard accuracy and instruction following generally rose with model scale, but, as the authors note, the pattern was inconsistent across tasks and datasets. Larger size did not guarantee obedience equally everywhere.

"Task competence and instruction following are distinct capabilities,

and reporting only standard accuracy hides instruction-following failures," the abstract of the arXiv paper states.

What this means

For anyone deploying small models, the takeaway is practical: a high benchmark score does not mean the model will do exactly what you ask. Gains in problem-solving ability do not automatically provide reliable control over behavior, so obedience should be measured separately with a metric like IFFR rather than inferred from overall accuracy. For applications where controllability and safety matter, this is one more argument for testing a model on conflicting commands, not just on reference tasks.

Frequently asked questions

What is Instruction-Following Failure Rate (IFFR)?

IFFR is a metric proposed by the authors that measures the share of cases in which a model failed to carry out the given instruction. Unlike standard accuracy, it captures failures to follow the command rather than the quality of the task solution.

Why does a model look accurate but not obey?

Because in the experiment, answers are checked against the original correct solution. If the model ignored the "counter" instruction, for example "choose the wrong option," and solved the task as usual, the standard metric counts it as a success even though the model violated the user's command.