Ragas, DeepEval, RAGChecker, Opik: какие метрики RAG действительно работают

On July 12, 2026, French researchers published an empirical study on arXiv on metrics for RAG systems (Retrieval-Augmented Generation). Scientists compared how well four popular metrics libraries predict quality as assessed by humans on real data from business systems.

Which metrics were tested

Researchers selected four RAG metrics libraries:

• Ragas — metrics based on LLM-evaluation and classical IR-metrics
• DeepEval — focus on correlation with human assessments
• RAGChecker — oriented towards verification of context correctness
• Opik — monitoring and evaluation tool from OpenAI

Additionally, standard metrics were used: recall, precision, F1, and others.

Test data was created by the researchers themselves: human annotators labeled questions and answers based on real business documents. This allowed them to avoid synthetic data and test the metrics in a natural setting.

What correlates with quality

If a metric truly predicts quality well, its scores should correlate with human scores. It turned out that different metrics from the same library (and especially from different libraries) gave significantly different results.

Some metrics correlated well with human assessments on the full dataset, but predicted poorly on a subset. Others showed opposite correlations: one metric increased while human assessment decreased.

"We found that simple aggregation of several metrics does not

guarantee a better result," the study authors conclude.

Why results need to be clarified

Researchers honestly noted the limitations of their methodology: the dataset size was relatively small, data was taken from a single domain (business documents), and not all metrics from all four libraries were fully utilized. The assessment methodology itself depended on the choice of annotators — different people sometimes give different ratings for the same answer.

Additionally, the paper notes that some metrics perform poorly in cases where the answer is correct but paraphrased, or when the context contains necessary information but the metric doesn't notice it.

What it means

There is no universal RAG metric that works equally well for all types of systems. RAG application developers need to choose metrics for their specific task, and it's best to combine several approaches and regularly test them on real data with human annotation.

Frequently asked questions

Which metric should I choose for my RAG system?

There's no definitive answer. The authors recommend starting with simple metrics (recall, precision), then adding one of the four libraries (for example, Ragas if speed is needed; DeepEval if correlation with humans is important), and definitely validate on your own data.

Why do metrics from different libraries give different results?

Each library uses different formulas, different LLM models for evaluation, and different definitions of "quality." Ragas may consider alignment with original information important, while RAGChecker values correctness of logic in the answer. On some data this aligns, on others it doesn't.