LanceDB Ships Realtime Engine: Fresh Data Searchable in 52 ms
In typical data lakes, writes wait for a batch commit, so a frame saved a second ago cannot be found in search. LanceDB has added Realtime Engine to its Enterprise edition: a row is acknowledged once it is saved to S3 storage and can be read in 52 ms on average. Search over fresh rows takes 15–28 ms for text and 26–30 ms for vectors. For live robotics and RL datasets, teams can skip a separate search service.
AI-processed from LanceDB Blog; edited by Hamidun News
Picture an engineer examining a failed robotic arm grasp. The frame was saved a second ago, but it is not in the table yet: the system waits for a batch to fill and a commit, meaning the publication of a new table version, to finish. A researcher training a model with reinforcement learning (RL) faces the same problem when they want to check a run that just finished. LanceDB has released Realtime Engine for its paid LanceDB Enterprise edition, which removes this wait. By the company's measurements, the median time from when a write is durably stored in cloud storage (S3) to when the row can be read is 52 milliseconds.
The problem with batch data lakes
Modern lakehouse systems (data lakes where tables live as files in object storage such as S3) were built around large batch commits. Data is collected, written as columnar files, and then a new table version is published. Storing and scanning such large volumes is cheap, but streaming creates an awkward trade-off. If you commit often, small files and metadata updates pile up. If you wait for a large batch, readers wait along with writers.
For AI workloads, this is especially visible. Robot fleets continuously produce images and sensor readings, while labeling pipelines add labels and embeddings. A corrected label should immediately affect the next filter, and a recomputed embedding should appear in the next vector search. In RL training, workers write trajectories, graders add rewards, and the training process picks up finished examples. All of this runs in parallel and without pauses.
The usual fix is to put a write buffer or a separate search service in front of the lake. LanceDB points to the drawbacks of this approach: data ingestion often supports only row appends, vector and full-text indexes are poorly supported, and the data itself is tied to a vendor. While the data sits in a closed buffer, other engines cannot read it through the open table format. Durability, moreover, often depends on replication inside the buffer rather than on the object store.
How Realtime Engine works
The foundation is the MemWAL format, part of the Lance table format. It adds an LSM tree (a structure that first quickly accepts writes into memory and a log, then merges them into main storage) on top of object storage. A new write lands on an engine node. The node stores it in a write-ahead log in object storage and adds it to an indexed in-memory buffer, the MemTable. A success response is returned only after the data is safely persisted, so acknowledged writes are not lost if a node crashes.
Full MemTables are flushed into indexed Lance files (SSTables) and merged into the main table in the background. Each shard, meaning a separate part of the table, writes independently. As a result, there is no table-wide commit for every write, and ingestion scales by adding shards and nodes. Queries look at the MemTable, the SSTables, and the base table at once, so fresh and old data are searched in a single query. While the base table's indexes catch up with changes, queries use the SSTable indexes, and new rows do not drop out of the results.
Compaction (merging small files), index updates, and garbage collection run on separate workers and scale independently of ingestion and queries. If a table receives a heavy sustained load, maintenance workers are added for it.
What the measurements showed
The figures come from LanceDB itself, run on Amazon EKS with the public datasets FineWeb-Edu (web text), MS MARCO (search passages), and LAION (image-text pairs). This is not an independent verification. A write was counted only after it was saved to S3, and each run checked that exactly the acknowledged rows ended up in the table. One node had 30 vCPUs and 112 GiB of memory.
- The median time from a durable write to S3 until the row is readable is 52 ms.
- Writes with a live primary-key index grew from 79–83 MB/s on one shard to 312–334 MB/s on four shards of one node.
- The median latency of queries over fresh rows during ingestion was 15–28 ms for full-text search and 26–30 ms for vector search.
- Writing into a table that already held the full FineWeb-Edu or MS MARCO corpus was no slower than writing into a small one. Old data remains searchable while new data arrives.
Live indexes cost CPU time, and the cost depends on the data. On LAION, most of each row is the image, so supporting all indexes barely hurts: 289 MB/s versus 314 MB/s with a single primary-key index. Long text documents require more full-text index work per written byte. Therefore, you can choose which indexes to keep live for each table and balance write speed against query speed on new data. The engine can write images straight to object storage, keeping only a reference in memory to reduce node memory usage.
What this means for practitioners
If you build datasets from robotics data streams or run RL pipelines, Realtime Engine removes an extra layer: a separate search service and the synchronization with it. The limitations are also clear. This is a feature of the paid Enterprise edition, the best results are shown on a multi-shard configuration, and sustaining high write speed over hours depends on whether maintenance workers keep up. The MemWAL format itself is open and described in the Lance specification, so compatible readers and writers can work with the same data. Before switching, run your own workload: with long text, the cost of live indexes will be higher than with images.
- Nov 21, 2026Cassava Seeks Up to $1 Billion for Nvidia Chips for African AI Data Centers
- Nov 21, 2026Anthropic Goes Public: At $78 a Share, the Company Is Worth $1.4 Trillion
- Nov 21, 2026Anthropic's IPO Order Book Is Several Times Larger Than the Shares on Offer; Founders Set to Receive $1.75 Billion
- Nov 21, 2026AWS Connects Quick AI Assistant to Seller Central and Gives Sellers 12 Months of Plus
Key Questions Answered
What are the key details on The problem with batch data lakes?
Modern lakehouse systems (data lakes where tables live as files in object storage such as S3) were built around large batch commits. Data is collected, written as columnar files, and then a new table version is published. Storing and scanning such large volumes is cheap, but streaming creates an awkward trade-off. If you commit often, small files and metadata updates pile up. If you wait for a large batch, readers wait along with writers. For AI workloads, this is especially visible.
How Realtime Engine works?
The foundation is the MemWAL format, part of the Lance table format. It adds an LSM tree (a structure that first quickly accepts writes into memory and a log, then merges them into main storage) on top of object storage. A new write lands on an engine node. The node stores it in a write-ahead log in object storage and adds it to an indexed in-memory buffer, the MemTable. A success response is returned only after the data is safely persisted, so acknowledged writes are not lost if a node crashes.
What the measurements showed?
The figures come from LanceDB itself, run on Amazon EKS with the public datasets FineWeb-Edu (web text), MS MARCO (search passages), and LAION (image-text pairs). This is not an independent verification. A write was counted only after it was saved to S3, and each run checked that exactly the acknowledged rows ended up in the table. One node had 30 vCPUs and 112 GiB of memory. The median time from a durable write to S3 until the row is readable is 52 ms.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.