Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

The pplx Embedder Finds the Right Scan on the First Try 68% of the Time; EmbeddingGemma Manages 38%

Scanned contracts used to go through OCR first. Now a model can search for a page directly from its image. A Data Scientist at Raft Digital Solutions compared three such models. For a query like "which contract has a 0.1% per day penalty?", pplx-embed-late puts the right page first 68% of the time on DocVQA, while EmbeddingGemma-2 does so 38% of the time. Compressing a page to 1 MP barely hurts quality. For scans and tables, pplx is the one to pick, but budget more storage for the index.

AI-processed from Habr AI; edited by Hamidun News
The pplx Embedder Finds the Right Scan on the First Try 68% of the Time; EmbeddingGemma Manages 38%
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
TAT-DQA financial report tablespplx 65%, WeMM-2B 57.8%, EmbeddingGemma-2 30.6%
GDP charts, exact-number searchpplx 82.4%, WeMM-2B 39.7%
SlideVQA slidesWeMM-2B 94.6%, with pplx nearly level at 93.7%

Imagine an archive of thousands of contracts, invoices and reports. An employee asks, "Which contract has a 0.1% per day late-payment penalty?"

, and the system has to return the right page. Usually a scan is first turned into text with OCR, chopped into chunks and put into RAG. That works for running text.

In tables, charts and diagrams, though, the meaning lies in where the values sit, not in the words. Artem, a Data Scientist at Raft Digital Solutions, tested an alternative: multimodal embedders that build a numerical vector for a page directly from its image. The main metric of the test is hit@1: the share of questions for which the correct page came first in the results.

On DocVQA scans, the best model, pplx-embed-late (just pplx in the article), scored 68%, while EmbeddingGemma-2 scored 38%. If you are building document search, this tells you which model to try first.

Who took part and how it was measured

Three models were compared. WeMM-2B produces one 2048-dimensional vector per page and can accept an image together with text. EmbeddingGemma-2 also produces one vector per page, but only 768-dimensional; by default an image takes 280 tokens out of an 8192 budget. pplx-embed-late works differently: it produces a vector for every token (128 dimensions each) and searches with late interaction (MaxSim), where the closeness of a question and a page is computed from the best token pairs. It does not support mixed "image plus text" input.

The test is simple. All pages in a dataset are turned into vectors and stored in Qdrant. The question is also turned into a vector, and the database returns the 20 nearest pages. If a question has several matching pages, any of them counts. The 95% intervals were computed by bootstrap with 2000 resamples. Each run also included a control with shuffled questions: its hit@1 must stay below 10%, otherwise the model is guessing without regard to meaning.

Results by page type

On scans, pplx leads. DocVQA: 67.8% for pplx, 60.5% for WeMM-2B and 38% for EmbeddingGemma-2. On Russian business scans: 89%, 84.6% and 38.2%. On the company's internal corpus (46 questions): 78.3%, 60.9% and 21.7%. On TAT-DQA financial tables, pplx also wins: 65% versus 57.8% and 30.6%.

Charts give a mixed picture. For questions about the overall look, WeMM-2B is slightly better: on ChartQA it scores 69.1% versus 61.1% for pplx and 51.4% for EmbeddingGemma-2. But when the task is to find a chart by an exact value, pplx pulls ahead: on synthetic GDP charts it scores 82.4% versus 39.7% for WeMM-2B. For uniform synthetic tables that differ in only a few cells, pplx finds the right one 94.7% of the time, WeMM-2B 37.3%.

WeMM-2B is best at finding infographics and presentations: 85.2% on InfoVQA and 94.6% on SlideVQA, with pplx nearly level (84.6% and 93.7%). EmbeddingGemma-2 trails on all the real-world datasets.

Compression, hardware and caveats

You can cap a page at 1 megapixel, which cuts the number of tokens (the image chunks fed to the model) by a factor of 1.8–2.2. On scans, WeMM-2B's hit@1 does not change, while pplx's drops by 2 percentage points on DocVQA. At 0.25 MP, quality drops noticeably for both models.

The price of pplx's accuracy is index size: a vector for every token takes noticeably more space than a single vector per page, and you need to budget for that in storage and GPU. If resources are tight and speed matters, EmbeddingGemma-2 remains an option, but with a noticeable loss in quality.

Figures from different datasets cannot be compared directly: the page pools and questions differ. Diagrams and interfaces were tested only on synthetic data. That checks the mechanism; it is not an evaluation on real documents. The Russian dataset has only 10 independent pages, and the internal corpus has 46 questions, so conclusions drawn from them are less reliable.

Practical takeaway: for searching scans and tables, start with pplx-embed-late and estimate the index size in advance. For presentations and charts searched by overall look, WeMM-2B is enough. Cap images at 1 MP and measure quality on your own documents: the ranking of the models on them may differ from the public datasets.

Story

Key Questions Answered

Who took part and how it was measured?

Three models were compared. WeMM-2B produces one 2048-dimensional vector per page and can accept an image together with text. EmbeddingGemma-2 also produces one vector per page, but only 768-dimensional; by default an image takes 280 tokens out of an 8192 budget. pplx-embed-late works differently: it produces a vector for every token (128 dimensions each) and searches with late interaction (MaxSim), where the closeness of a question and a page is computed from the best token pairs.

What are the key details on Results by page type?

On scans, pplx leads. DocVQA: 67.8% for pplx, 60.5% for WeMM-2B and 38% for EmbeddingGemma-2. On Russian business scans: 89%, 84.6% and 38.2%. On the company's internal corpus (46 questions): 78.3%, 60.9% and 21.7%. On TAT-DQA financial tables, pplx also wins: 65% versus 57.8% and 30.6%. Charts give a mixed picture. For questions about the overall look, WeMM-2B is slightly better: on ChartQA it scores 69.1% versus 61.1% for pplx and 51.4% for EmbeddingGemma-2.

What are the key details on Compression, hardware and caveats?

You can cap a page at 1 megapixel, which cuts the number of tokens (the image chunks fed to the model) by a factor of 1.8–2.2. On scans, WeMM-2B's hit@1 does not change, while pplx's drops by 2 percentage points on DocVQA. At 0.25 MP, quality drops noticeably for both models. The price of pplx's accuracy is index size: a vector for every token takes noticeably more space than a single vector per page, and you need to budget for that in storage and GPU.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…