RAGAS and RAG Metrics: Stop Guessing, Start Measuring Quality

RAG systems are gaining popularity, but they often give wrong answers, add invented facts or ignore relevant context. In the third part of our quality engineering series, we look at how to measure RAG instead of reading tea leaves, and how to use RAGAS, a framework that replaces manual review with automation.

Why RAG Metrics Are Harder Than LLM Metrics

A plain LLM can be evaluated on standard benchmarks: MMLU, HumanEval, TruthfulQA. RAG adds a context retrieval layer, which creates new points of failure. The question "What was Gates's salary in 1997?" can break in several ways:

• The search engine did not find a relevant document: retrieval returned noise
• The model found the document but ignored the key fact: inattention
• The model found the fact but added a hallucination on top: source and invention blended together
• The context was relevant, but the answer does not match the question: a logic error

Controlling these scenarios requires dedicated metrics. Manually checking every answer is expensive and does not scale to thousands of queries.

RAGAS: Four Metrics for Every Case

RAGAS is a framework from Basis AI for automatic RAG evaluation. Here is its core set of metrics:

• Faithfulness: is the generated answer faithful to the context? The model checks whether the LLM added facts that are not in the sources. The score ranges from 0 to 1.
• Answer Relevance: does the answer match the question? RAGAS generates a reverse question from the answer and compares the two semantically.
• Context Precision: are the retrieved fragments relevant? It checks whether the retriever went in the wrong direction and returned noise.
• Context Recall: is the context complete? Did the documents contain all the information needed to answer the question fully?

Each metric points to a bottleneck: bad retrieval, bad generation, or both.

How RAGAS Evaluates Under the Hood

There is no magic here: RAGAS uses an LLM itself as the judge. For Faithfulness, it takes the generated answer and the context, asks the model to extract statements that can be verified (factual claims), then checks each one in turn against the context. If a statement is supported by a fact in the source, the score goes up.

For Answer Relevance, it generates a hypothetical question from the answer (reverse), then computes the cosine similarity (semantic distance) to the original question. The higher the match, the more relevant the answer.

"If your LLM knows how to lie, it also knows how to detect a lie", is

the framework's logic.

The whole process requires LLM calls (at least 2-3 per answer), so RAGAS is considered expensive in tokens. But the alternative is hiring people to annotate, which is even more expensive and slower.

What This Means

RAGAS makes RAG engineering reproducible and traceable. Instead of a qualitative "seems to work", you get quantitative metrics that show how each update (new documents, a new model, a new prompt) affects quality.

For small pet projects, RAGAS may be overkill. For enterprise solutions, where a mistake costs money and customer trust, it is the periodic table that RAG engineers have long been missing.