Gemma-3-12b is worse at spotting lies in RAG: detection rate drops from 25% to 10%

Researchers from the University of Texas at Dallas and the Vancouver campus of the New York Institute of Technology tested how well Gemma-3-12b notices false claims in a RAG context when those claims are repeated across several documents. When one document out of seven was poisoned, the model found the substitution in 25% of cases. When the same claim appeared in six of the seven documents, the detection rate fell to 10%. The authors call the result preliminary evidence of conformity: documents that agree with each other earn more trust, even when they all repeat the same fabrication.

How the experiment was built

The basis was 200 random questions from SciFact, a dataset of scientific claims that must be confirmed or refuted against sources, sampled with random seed 38. The poison came from an earlier paper by the same group. There, for each question, the researchers first fixed the correct answer, then generated a contradicting sentence and expanded it into a coherent text of about 500 words. An LLM judge checked that the text did not support the correct answer. GPT-5.6 was the main generator, and Grok 4.6 stepped in when GPT-5.6 refused to write.

For the new test, each poison text was kept and five paraphrases were made of it. The instructions required preserving the original claims and their level of confidence, including the plainly wrong premise, rather than correcting it. The researchers then assembled two contexts of seven documents each. In the "minority" condition, one document was poisoned and six were healthy. In the "majority" condition, the context held the original poison, its five paraphrases and one healthy document. The task was worded identically in both: review the passages and return a structured disinformation report naming the number of the suspicious passage.

What the numbers show

In the "minority" condition, Gemma-3-12b detected the poison in 50 of 200 cases (25.0%). In the "majority" condition, it did so in 20 of 200 (10.0%). The difference is 15 percentage points, a 60% relative decline. This alone shows the detector is weak: even in the best case, three questions out of four pass unnoticed.

The paired analysis is more interesting. The poison was found on 16 questions in both conditions and on none of 146 questions in either. The remaining 38 questions had different outcomes, and 34 of them flipped from "detected" to "not detected," while only four flipped the other way. An exact two-sided McNemar test (a paired test that looks only at questions whose outcome changed) confirmed that this imbalance is significant. For confidence intervals, the authors used the Wilson method.

Why the conclusion is cautious for now

The main practical risk is clear: an attacker does not need independent sources, only a way to multiply one false claim across paraphrased texts. A RAG system that treats every retrieved passage as a separate unit of evidence may mistake such duplication for confirmation. Six documents can be six different studies, or they can be six retellings of one unsupported assertion. A good disinformation detector must account for the provenance of sources and the dependencies between them, not just for matching wording.

But the authors themselves stress the limits of the work. One model was tested on one task with a fixed context of seven documents, and document retrieval (the retriever) played no part in the experiment at all. The switch to "majority" changes three things at once: the share of poisoned documents, the number of paraphrased repeats and the number of healthy counter-documents.

It is therefore impossible to claim that the drop is caused by conformity specifically, rather than by sensitivity to repetition or by a shortage of honest documents. In addition, the analysis used binary detection labels recorded in a table, with no independent human review of each written rationale and no check of whether the model pointed to the correct poisoned passage.

The next step is obvious: vary the share of poison, the number of paraphrases, the independence of sources, the order of documents and the number of healthy texts one at a time. For now, RAG developers should take away one point: deduplicating semantically similar passages and tracking the source before passing text to the model may be more useful than hoping the model will notice the trick on its own. The authors also note that the poisoned texts were deliberately fabricated to test robustness and must not be taken as scientific claims.