Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarLanceDB Blog→ original

Netflix Merges Video and Audio into One LanceDB Table: 5 Million Disk IOPS

Netflix stores frames, audio and subtitles in separate systems, so training teams have to assemble the data by hand. Now everything lives in one LanceDB table, with metadata and embeddings, the numerical "fingerprints" of a scene, stored alongside the video. The NVMe-backed cache handles more than 5 million read-write operations per second (IOPS) and serves 20,000 similar-frame searches per second. The takeaway: keep multimodal data in one place instead of copying it for each task.

AI-processed from LanceDB Blog; edited by Hamidun News
Netflix Merges Video and Audio into One LanceDB Table: 5 Million Disk IOPS
Source: LanceDB Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceLanceDB Blog ↗
Editor-in-Chief
Key metric #1Billions of vectors are indexed within hours, across tens of petabytes
Key metric #2New features are added without copying petabytes of data
Key metric #3Long GPU jobs can be paused and resumed without rerunning from scratch

Imagine you need to train a model on a Netflix series: find scenes similar to a given one, filter them by dialogue, and mix in production metadata. The frames sit in one storage system, the audio in another, the subtitles in a third, and the outputs of model annotations are scattered across scripts and tables. Netflix described how it brought order to this: it built a Media Data Lake on the Lance format and the LanceDB database.

The headline numbers are these: the NVMe cache delivers more than 5 million IOPS (read-write operations per second), and vector search, that is, finding similar frames and clips by their numerical meaning fingerprints (embeddings), sustains more than 20,000 queries per second. The description was published on LanceDB's own blog, so the figures come from the vendor, but the case is instructive for anyone working with video, audio and images.

What problem they were solving

Classic storage systems handle tables of numbers and text well: metrics, logs, events. Media is different. A single series produces millions of frames, hours of multichannel audio, subtitles and annotations. On top of that come derived data: embeddings, model labels, and scene-analysis results. All of it is tied to creative workflows and asset provenance.

Netflix's Media ML Data Engineering team created a dedicated specialization at the intersection of ordinary data engineering and ML for media. The result is called Media Tables. These are tables where canonical metadata, the raw files (blobs) and model outputs sit side by side. A researcher works from a single source of truth instead of assembling data from five systems.

How it works

At the core is the Lance format. It is built for fast random access, which matters in training, where data must be constantly shuffled and filtered. The second property is adding new columns without rewriting the data. Suppose a model appears that labels scenes by mood. Its results are simply appended to the table, and no one duplicates petabytes of video.

Different query types run on the same table: SQL, vector search and full-text search. A typical workflow: find frames visually similar to a given one, filter them by dialogue lines, and join structured metadata. All of this happens in one system.

Features are computed with user-defined functions (UDFs) that scale across hundreds of thousands of CPU cores and GPU slots. In the article's example, such a function takes a text column and, using a loaded model, converts sentences into 384-number vectors. Computation runs incrementally, with checkpoints and eviction support: an expensive GPU job can be paused and resumed without redoing completed work.

For training, data is read through named SQL views over the raw assets and features, while blobs are accessed directly through efficient APIs.

The numbers and what to do with them

According to the post, the system indexes billions of vectors within hours and manages tens of petabytes under a single abstraction. The distributed NVMe SSD cache delivers more than 5 million IOPS and sustains more than 20,000 vector search queries per second. According to the authors, this lowers cloud storage costs and makes it affordable to experiment with large multimodal models. Named applications include HDR remastering, localization quality metrics, plot understanding and multimodal search.

A practical takeaway for developers and product managers:

  • If you work with video, audio or images, design a unified table holding blobs, metadata and embeddings from the start, rather than a set of disconnected indexes.
  • Choose a format where new features are added without rewriting data. Otherwise every new model turns into copying terabytes.
  • Plan for pause and resume in expensive GPU jobs: this is a direct budget saving.

Limitations

The source is a vendor blog, not an independent benchmark. The infrastructure cost in dollars, the cluster size and any comparison with alternatives such as Iceberg or Delta Lake are not given in the text. ByteDance, Midjourney, Runway and Character AI are mentioned as LanceDB users, but without details about their workloads. Even so, the design is useful as a reference point: Netflix shows that a multimodal lake with unified access for SQL, vector search and training works in production at a scale of tens of petabytes. The technology itself is open, so the approach can be tested on your own data.

Story

Key Questions Answered

What problem they were solving?

Classic storage systems handle tables of numbers and text well: metrics, logs, events. Media is different. A single series produces millions of frames, hours of multichannel audio, subtitles and annotations. On top of that come derived data: embeddings, model labels, and scene-analysis results. All of it is tied to creative workflows and asset provenance. Netflix's Media ML Data Engineering team created a dedicated specialization at the intersection of ordinary data engineering and ML for media.

How it works?

At the core is the Lance format. It is built for fast random access, which matters in training, where data must be constantly shuffled and filtered. The second property is adding new columns without rewriting the data. Suppose a model appears that labels scenes by mood. Its results are simply appended to the table, and no one duplicates petabytes of video. Different query types run on the same table: SQL, vector search and full-text search.

What are the key details on The numbers and what to do with them?

According to the post, the system indexes billions of vectors within hours and manages tens of petabytes under a single abstraction. The distributed NVMe SSD cache delivers more than 5 million IOPS and sustains more than 20,000 vector search queries per second. According to the authors, this lowers cloud storage costs and makes it affordable to experiment with large multimodal models. Named applications include HDR remastering, localization quality metrics, plot understanding and multimodal search.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…