TII unveils QIMMA, an Arabic LLM leaderboard that audits benchmarks before ranking models

TII has launched QIMMA, a new leaderboard for Arabic LLMs that changes how models are evaluated: the team checks the quality of the benchmarks first and only then publishes results. The authors show that even well-known Arabic datasets contain systematic errors that skew final scores.

What is QIMMA

QIMMA combines 109 subsets from 14 source benchmarks into a single evaluation suite of more than 52,000 examples. Coverage is broad: culture, STEM, law, medicine, safety, poetry and literature, and programming.

According to the authors, 99% of the content is originally in Arabic rather than translated from English. That matters because translated tests often break natural context, produce awkward phrasing, and give models tasks that poorly reflect how Arabic is actually used.

Against this backdrop, QIMMA positions itself not as just another table of scores, but as an attempt to close several long-standing problems in Arabic NLP: fragmented leaderboards, weak reproducibility, no per-sample results, and unverified gold answers.

The authors also stress another distinction: it is the first Arabic leaderboard with built-in code evaluation. To that end, they added adapted Arabic versions of HumanEval+ and MBPP+, so the suite tests not only language knowledge but also a model's ability to understand programming tasks written in Arabic.

How validation works

The core of the project is a two-stage validation pipeline. Before any model is run, two large models independently check each example: Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B. They rate tasks against ten binary criteria. If either model gives an example less than 7 out of 10, it is flagged as problematic: when both models agree, the example is dropped outright, while disputed cases go to manual review by native speakers familiar with regional and dialectal nuances.

QIMMA checks benchmarks before evaluating models, so that final scores reflect the real quality of

Arabic LLMs.

For code benchmarks, the team took a different route. Instead of removing tasks, the researchers rewrote the Arabic prompts without changing identifiers, reference solutions, or test suites. They fixed 145 of 164 prompts in HumanEval+ (88%) and 308 of 378 in MBPP+ (81%). The edits covered several areas:

• normalizing the language to natural Modern Standard Arabic
• removing ambiguities and clarifying constraints
• aligning terminology, punctuation, and example format
• fixing structural errors such as broken strings and corrupted text fragments
• clarifying meaning where ranges or conditions could be read in more than one way

What problems were found

The review showed these were not isolated slips but recurring defects in the datasets themselves. For example, the team discarded 436 examples from ArabicMMLU (3.1% of the dataset) and 41 from MizanQA (2.3%). Other sets had lower defect rates, but the same pattern repeated across them: errors in correct answers, unreadable text, duplicates, culturally disputed labels, and mismatches between the gold answer and the scoring method.

In other words, some popular Arabic benchmarks have been used as if they were error-free, when they are not.

On the cleaned set, Qwen3.5-397B-A17B-FP8 came out on top with an average score of 68.06. Karnak is second with 66.20, and Jais-2-70B-Chat is third with 65.81. The authors also note that model size does not guarantee a better result. Arabic-specialized models are often stronger on cultural and linguistic tasks, while multilingual systems do better at coding: Qwen3.5-397B posts the best results on both HumanEval+ and MBPP+. So QIMMA is useful not only as a ranking but also as a map of the strengths of different architectures.

What this means

QIMMA makes a simple but important shift: comparing LLMs without checking the tests themselves is no longer enough. For the Arabic market, this may become a new evaluation standard, and for developers it is a reminder that a benchmark's quality affects a model's reputation as much as the model itself.