Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseResearch Paper · For R&DLanceDB Blog→ original

LanceDB: The IVF_PQ Index Compresses RAG Vectors 64x and Speeds Up Search

Vector search in RAG runs into a memory wall: a single embedding of 128 float32 numbers weighs 4096 bits, and a database may hold millions of them. The IVF_PQ index in LanceDB tackles this with two techniques. PQ splits each vector into chunks and stores, instead of each chunk, the ID of a similar "typical" chunk (8 IDs of 8 bits each, or 64 bits total, 64 times smaller). IVF divides the space into cells and searches only the nearest ones. That is why, as a database grows, IVF_PQ is worth testing on your own data.

AI-processed from LanceDB Blog; edited by Hamidun News
LanceDB: The IVF_PQ Index Compresses RAG Vectors 64x and Speeds Up Search
Source: LanceDB Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceLanceDB Blog ↗
Editor-in-Chief
Key metric #1128 float32 numbers (4096 bits) become eight 8-bit IDs, or 64 bits
Key metric #2nprobes defaults to 20; for high recall, 5–10% of the dataset is often enough
Key metric #3refine_factor recomputes candidates in memory: more accurate, but slower

Every RAG system eventually runs into simple arithmetic: embeddings pile up, and storing them as they are gets expensive. A single dense vector of 128 float32 numbers takes 4096 bits, and a real database holds millions of such vectors. The LanceDB team, in its blog, broke down the IVF_PQ index, which tackles two problems at once. The PQ part (Product Quantization) compresses each vector: in the authors' example, 4096 bits become 64 bits, 64 times less. The IVF part (Inverted File) speeds up search: the system doesn't scan the entire database but only looks at the nearest groups of vectors.

How PQ Compresses a Vector 64x

Imagine a long address that needs to be written in a short form. You cut it into parts, build a reference table of typical values for each part, and write not the part itself but the number of its row in the table. Product Quantization works in a similar way.

There are three steps. First, the large vector is split into equal chunks, or subvectors. The vector's length must divide evenly by the number of chunks. Then, for each chunk, the nearest centroid is found, meaning a typical representative of similar chunks (the authors call it a reproduction value). Finally, instead of the centroid itself, its unique ID is stored, which can later be used to restore the centroid.

In a toy example, a 12-number vector is split into four chunks of three numbers and becomes four IDs. The example is artificial, and the savings barely show. At realistic sizes, the picture changes. A 128-number float32 vector takes 128×32 = 4096 bits. Compress it into eight 8-bit IDs, and you get 8×8 = 64 bits. That's a 64x difference.

The cost is known: the restored vector doesn't match the original. Any lossy compression loses something, and PQ is no exception. That's why search over such vectors is approximate.

How IVF Narrows the Search Area

PQ saves memory but doesn't answer the question of how many vectors need to be scanned. That's where the Inverted File comes in. The space is divided into partitions, each holding vectors that are close to one another. The author illustrates this with a Voronoi diagram: a set of "cells" where each point belongs to its nearest center.

When a query arrives, the system doesn't scan the entire database. It finds the nearest cells and searches only within them. Within that subset, PQ is applied. The result is a coarse filter followed by a search among compressed vectors. The fewer the candidates, the faster the answer.

Which Parameters to Tune in Practice

In LanceDB, the index is created in a few lines: the data is loaded into a table, then create_index is called with the number of partitions. The blog's example uses 10,000 random vectors. An important caveat: training the index requires enough data in the table, otherwise the centroids will be poor.

There are three main parameters at search time:

  • limit, default 10: how many results to return.
  • nprobes, default 20: how many partitions to scan. A higher value improves accuracy but slows the search. According to the authors' advice, to get high recall (the share of relevant results found) with low latency, covering 5–10% of the dataset is usually enough.
  • refine_factor, not set by default: when enabled, the system reads additional items and re-sorts them in memory. Accuracy rises, speed drops.

What to Do With This and What Not to Expect

If you're building RAG or semantic search, and your database is growing along with your memory and infrastructure bills, IVF_PQ is worth considering. The nprobes and refine_factor parameters let you trade accuracy for speed without rebuilding the index.

There are also honest limitations. The material is educational: it contains no benchmarks on real data, and the 64x figure follows from vector sizes, not from measurements of search quality. How much recall drops on your embeddings depends on the model, dimensionality, and number of partitions, and you'll need to test that on your own sample. But as an introduction to what lies behind the term "vector index" and where the memory savings come from, the text is useful and engineering-concrete.

Story

Key Questions Answered

How PQ Compresses a Vector 64x?

Imagine a long address that needs to be written in a short form. You cut it into parts, build a reference table of typical values for each part, and write not the part itself but the number of its row in the table. Product Quantization works in a similar way. There are three steps. First, the large vector is split into equal chunks, or subvectors. The vector's length must divide evenly by the number of chunks.

How IVF Narrows the Search Area?

PQ saves memory but doesn't answer the question of how many vectors need to be scanned. That's where the Inverted File comes in. The space is divided into partitions, each holding vectors that are close to one another. The author illustrates this with a Voronoi diagram: a set of "cells" where each point belongs to its nearest center. When a query arrives, the system doesn't scan the entire database. It finds the nearest cells and searches only within them. Within that subset, PQ is applied.

Which Parameters to Tune in Practice?

In LanceDB, the index is created in a few lines: the data is loaded into a table, then create_index is called with the number of partitions. The blog's example uses 10,000 random vectors. An important caveat: training the index requires enough data in the table, otherwise the centroids will be poor. There are three main parameters at search time: limit, default 10: how many results to return. nprobes, default 20: how many partitions to scan. A higher value improves accuracy but slows the search.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…