BILAL-Bench: Six of Seven Open Models Confidently Fabricate Sources in 79–95% of Cases

When a language model invents a Quran verse (ayah) or a hadith (a report of the words and actions of the Prophet Muhammad), it lends a fabrication the authority of a sacred text. A team from the Islamic University of Technology measured the scale of the problem in Bengali. The BILAL-Bench benchmark includes 240 questions verified by an expert in Islamic sciences. Seven open models produced 1 680 answers, and six of the seven confidently fabricate sources in 79.3–95.3% of cases where a specific citation is needed. Only 40 of the 1 680 answers were both correct and properly backed by a source.

How the test works

The questions were written directly in Bengali, not translated. They fall into five categories: attribution of Quran verses, hadiths, fiqh (norms of Islamic law), contemporary questions with no single agreed answer, and adversarial prompts with a false premise. Reference answers came from the Islamic Foundation Bangladesh's Bengali translation of the Quran, Bengali editions of the six canonical hadith collections, and fatwa portals. A qualified expert then checked them. For fiqh, the reference follows the Hanafi position. The authors stress that this is a design choice, not a claim that other schools are wrong.

The annotation has two axes. The first records the type of source error: complete fabrication, incorrect attribution, unsupported claim, appropriate refusal, or correction of a false premise. The second shows whether the answer sounds confident or hedged. The main metric is the share of confident fabrications among all answers, and answers that make no claim stay in the denominator. Two native Bengali speakers with relevant training labeled all answers blind. Cohen's kappa (inter-annotator agreement) was 0.864 across eight classes and 0.893 after merging classes.

The models received a single neutral instruction, with no system message, no examples and no hint to be more careful. Generation used greedy decoding with a 512-token limit. The test covered TigerLLM-1B-it, TituLM-gemma-2-2b, Gemma-2-9B-it, GPT-OSS-120B and three BanglaLLaMA variants: 3.2-3B-instruct, 3.1-8B-instruct and 3.1-8B-no-pretrain.

Scale doesn't help, and hadiths are the worst

In the categories that require citations, GPT-OSS-120B confidently fabricates in 93.3% of answers, while BanglaLLaMA-3.2-3B-instruct, forty times smaller, does so in 90.0%. The confidence intervals overlap. The lowest rate among the six is TituLM-gemma-2-2b at 79.3%. TigerLLM-1B-it looks better on paper at 63.3%, but 43 of its 150 answers contain no claim at all, and the model has no answers that are correct and properly backed by a source. Its low fabrication rate reflects silence, not accuracy.

Models struggle most with the origin of hadiths. Of 315 answers, only two are correct and properly backed by a source. Three models fabricate a source in every answer in this category, and the lowest fabrication rate among all models is 88.9%. GPT-OSS-120B reached 100%.

The shape of the errors differs as well. GPT-OSS-120B has 166 cases of outright fabrication and only one unsupported claim. TituLM-gemma-2-2b has 16 fabrications but 90 unsupported claims. Both patterns are bad, but a precise citation to a nonexistent source is especially dangerous: the reader has something to look up and something to believe. Of the 1 320 answers with source errors, 1 316 (99.7%) sound confident, so hedging is almost absent.

Where the models do differ

The two 8B versions of BanglaLLaMA differ only in whether they underwent additional pretraining on Bengali. Their confident fabrication rates are close: 83.3% and 86.7%. The mix of errors differs: the version with the additional training has 93 fabrications and 8 incorrect attributions, while the version without it has 84 and 17.

The models diverge most in restraint. On false-premise prompts, Gemma-2-9B-it fabricates in 40.0% of cases and GPT-OSS-120B in 91.1%. The number of appropriate refusals or corrections across the whole benchmark ranges from 6 for GPT-OSS-120B to 46 for Gemma-2-9B-it. It does not depend on parameter count.

The authors' practical conclusion is that size does not reduce fabrication, so it is more useful to look for other levers. These include retrieval from verified Bengali collections, training models to refuse deliberately, and automatic checks that the cited collection, narrator and hadith number actually exist. The authors name hadiths as the sharpest point for further experiments.

The limitations are also significant. The benchmark contains 240 questions in a single language, one neutral instruction, greedy decoding and a Hanafi reference for fiqh, so the results cannot be transferred to other schools, languages or model families. The variation in abstention needs to be tested on more models and prompt styles. BILAL-Bench itself is an evaluation tool, not a source of religious rulings, and the authors explicitly warn against using current open models for religious questions, including searching for fatwas. The code and data are published on GitHub in the BILAL-Bench repository.