Competence Over Accuracy: Why GPT and Qwen Lose Logic When Problems Get Rephrased

Researchers at the University of Illinois Urbana-Champaign have published a study that calls into question the reliability of current methods for evaluating logical reasoning in large language models. Their new methodology, Competence-Based Evaluation (CBE), reveals a fundamental mismatch: models that post high accuracy on standard benchmarks can't consistently apply the same logical rule when a task's wording or difficulty changes.

The problem with current reasoning evaluations

Despite impressive gains across a wide range of tasks, large language models still struggle with logical reasoning — in particular, with consistency across linguistic variations. Current benchmarks evaluate models using a single metric: accuracy on a chosen set of tasks. That approach doesn't reveal whether a model can generalize and transfer knowledge to new variants of the same problem.

The Competence-Based Evaluation methodology

The researchers propose an alternative evaluation paradigm — CBE — that tests whether models can reliably apply the same piece of knowledge across an entire family of related problems. The core idea is a parameterized task generator that scales difficulty while keeping the required competence fixed. Consistent answer representations test whether the underlying logic holds up beyond any single specific phrasing.

The methodology was applied to elementary ordering tasks, where input size was scaled and the ways of presenting the same ordering rule were varied.

Testing six models from the GPT and Qwen families

Six models from the GPT and Qwen families were tested. The results were telling:

• Performance within narrow ranges: models showed acceptable performance within limited difficulty ranges, but even strong models failed the competence check once the task family was expanded

• Chain-of-Thought (CoT) and GRPO: explicit reasoning techniques (chain-of-thought) and the GRPO method extended the range of strong performance but didn't eliminate all the identified errors in logical knowledge transfer

• Supervised LoRA: the supervised LoRA approach showed minimal transfer of knowledge to new data the model hadn't seen before

These findings point to a deeper problem: the standard accuracy metric doesn't correlate with the reliability of logical transfer and generalization.

Outlook and implications for LLM development

The CBE methodology provides detailed diagnostic profiles for each model and offers an evaluation protocol that can evolve alongside future generations of models. The study underscores the need to rethink how we measure LLMs' logical capabilities — not just by raising overall accuracy, but by ensuring reliable and stable application of knowledge across task variations.