GeopoliticalQA: bge-m3 Beats Commercial Embeddings on 5 525 Report-Based Questions

A group of researchers has published GeopoliticalQA on arXiv, a benchmark for RAG systems that answer questions about geopolitical and strategic analytical reports. The set contains 415 open reports and 5 525 question-answer pairs, each tied to a specific document. That works out to about 13 questions per report. The main result: at the chunk level, retrieval quality does not grow with embedding model size. The mid-sized open model bge-m3 matched models an order of magnitude larger and outperformed commercial embedding APIs. Generation, meanwhile, stays difficult even when the model is given the correct passages.

What's inside the benchmark

Analytical reports are a poor fit for standard RAG. Their formats vary, their structure is uneven, and the most valuable conclusions are often spread across several sections rather than sitting in a single paragraph. The authors split the questions into four types: factual, inferential, comparative, and risk-oriented.

A document-aware pipeline generated the questions. It first profiles the report, working out what the document is about and how it is organized, and only then generates questions. This makes the questions less dependent on a random snippet of text. The authors validated the set in two ways: expert annotation and an LLM-as-judge protocol, in which a language model does the scoring.

• Corpus: 415 open geopolitical and strategic reports
• Annotation: 5 525 questions tied to documents
• Question types: factual, inferential, comparative, risk-oriented
• Retrieval: 15 systems
• Generation: six open-weight LLMs
• Quality validation: experts and LLM-as-judge

Retrieval: scale doesn't settle it

At the document level, meaning selection of the right report, the systems produce similar results. Differences appear at the chunk level, where a system must find a specific passage inside a long text. Here, according to the authors, quality is not determined by embedder scale. bge-m3 holds its own against models an order of magnitude larger and outperforms commercial embedding APIs.

The abstract names neither all of the tested systems nor the specific metric values. The exact gaps will have to be checked in the paper's tables. The direction is clear, though: a paid API does not guarantee better retrieval on a specialized corpus.

Generation: a lower ceiling than expected

For the generation test, the authors gave the six open LLMs gold-standard passages, which removed retrieval errors from the equation. The models still perform poorly, especially on inferential and risk-assessment questions. Factual questions are easier because the answer is in the text. Inferences and risks require comparing claims, weighing arguments, and forming a judgment that may not appear verbatim in the report. These are exactly the answers people read analytical reports for.

If an answer is bad even with perfect context, the retriever is not the problem. Retrieval tuning won't fix it.

What teams building RAG should do

First, don't choose an embedder by size or price. Run several models on your own corpus, including bge-m3 as a self-hosted baseline. Your data also stays inside your perimeter, which matters for sensitive documents.

Second, evaluate retrieval and generation separately. A mode with gold-standard passages shows the generator's ceiling and helps pinpoint where the system breaks.

Third, break metrics down by question type. An average across all questions will hide failures on risk and inference queries. Human review is worth keeping for those.

There are limitations as well. The corpus consists of open reports rather than classified materials, and part of the validation relies on LLM-as-judge, with its inherent biases. Even so, it is a useful benchmark for comparing RAG approaches on analytical texts.