Researchers Propose CBE: Six GPT and Qwen Models Fail a Competence Test

The authors of a preprint spotted on Hugging Face Daily Papers propose rethinking how we measure the logical reasoning of large language models. Instead of a single accuracy figure on a chosen set of tasks, they introduce competence-based evaluation (CBE). Under CBE, a model counts as competent only if it applies the same skill consistently across an entire family of tasks. Testing on six GPT and Qwen models produced a clear result. Quality is quite decent within limited complexity ranges, but even the strong models do not meet the competence condition.

What's wrong with conventional benchmarks

A standard benchmark returns a single percentage of correct answers. The authors argue that this number says almost nothing about generalization. A model can solve a task confidently in its familiar phrasing and then fail when the same logic is presented differently. A separate problem is that a benchmark fixes its tasks in advance. If they fall in a zone where the model is strong, the final figure looks convincing, even though quality outside that zone is different.

How CBE works

The approach relies on two mechanisms. The first is a parameterized generator that increases complexity while keeping the required competence the same. The second is a set of equivalent representations that preserve the answer: the same task is written in different ways, and the correct result stays unchanged. A model passes the test only if it succeeds across the whole grid of sizes and phrasings, not at a single point.

The authors built their first practical example on elementary ordering tasks. The input size grows, and the ordering rule is presented in different forms. The task is deliberately simple: if a model cannot withstand variations here, it is harder to expect reliability on complex reasoning.

• What was tested: six models from the GPT and Qwen families.
• Task: elementary ordering with growing input size.
• Variations: different phrasings of the same rule with the same correct answer.
• Result: acceptable quality within limited complexity ranges, but no model met the competence condition, including the strong ones.
• Chain-of-thought and GRPO: widen the zone of strong performance but do not remove all the failures found.
• Supervised LoRA: weak transfer to held-out data.

What the improvement methods showed

The authors also tested the methods usually used to fix weak reasoning. Chain-of-thought and GRPO, a reinforcement learning method, push back the boundary beyond which a model starts to fail, but they do not close all the gaps found. Supervised LoRA fared worse: transfer to held-out data is minimal. This fits a common observation: supervised fine-tuning on a narrow distribution easily leads to memorizing patterns instead of learning the rule. That, however, is our own interpretation, not a conclusion from the abstract.

How to use these ideas in practice

Teams choosing a model for logic-heavy tasks should add their own parameterized tests to standard benchmarks. Take a rule that matters for your product, generate tasks of growing size, and write each one in several phrasings. Then look not at the average score, but at exactly where the accuracy curve breaks. Fine-tuning results deserve the same treatment: if quality after LoRA improved only on familiar formats, you improved the metric, not the reasoning.

There are caveats. Ordering is a narrow class of tasks, and the abstract gives no specific percentages, so the scale of the failures will have to be checked in the full text. But the authors call the protocol itself extensible: it can be carried over to other types of logic and applied to new models.