Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarLanceDB Blog→ original

Uber and LanceDB Teach the Lance Format to Split a Dataset Across Buckets: 5 Paths Instead of 10 Million

When thousands of readers pull training data from the same dataset at once, a single S3 bucket (a cloud storage container) becomes a bottleneck. Uber asked LanceDB to spread a dataset across several buckets without losing portability. The Lance manifest now holds a short list of "bases", which are shared path prefixes. Each file stores only a base number and a relative path. As a result, a dataset of 10 million files spread across 5 buckets can be moved by replacing 5 rows, not 10 million.

AI-processed from LanceDB Blog; edited by Hamidun News
Uber and LanceDB Teach the Lance Format to Split a Dataset Across Buckets: 5 Paths Instead of 10 Million
Source: LanceDB Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceLanceDB Blog ↗
Editor-in-Chief
Key metric #1Writes rotate across buckets, and reads run in parallel from all of them
Key metric #2Garbage collection and credential vending iterate over bases, not files
Key metric #3A base can be marked as a dataset root (for clones) or as a folder holding data files

Imagine you need to move from one cloud storage provider to another, and the table metadata records the full addresses of ten million files. You would have to rewrite every address. LanceDB and Uber's AI infrastructure team set out to solve exactly this problem. The goal was to spread one dataset across several S3 buckets (a bucket is the top-level container in cloud object storage) so that data reads faster during model training, while keeping the dataset cheap to move.

The problem: one bucket can't handle thousands of readers

Uber's multimodal data for model training and agentic search lives in datasets at petabyte scale. Thousands of readers access them at the same time, and a single bucket quickly becomes a bottleneck. So the engineers wanted to split one dataset across N buckets. New files are written in rotation across the buckets, and reads run in parallel from all of them. The aim is to raise aggregate throughput roughly N-fold.

The catch is in the paths. If all paths are relative, a file outside the dataset folder cannot be referenced. If the paths are absolute, every move requires rewriting the metadata in full. Uber needed both: multiple storage locations and maximum portability.

Lessons from Iceberg and Delta: hybrids are awkward

The author has worked on Apache Iceberg, Delta Lake, and now Lance. He notes that the two formats took opposite paths.

Iceberg started with absolute paths. They are unambiguous, but migrating a table means rewriting metadata that contains thousands or millions of paths. The discussion about relative paths began in 2021, and in late 2025 the community agreed on a design for version 4: paths without a URI scheme are treated as relative to the table's location.

Delta Lake went the other way. In September 2019 it started with relative paths, and in August 2021 it allowed absolute ones. This was needed for shallow clone (introduced in December 2022), where a new table references the files of an old one without copying them, and those files sit outside the new table's root.

As a result, both formats became hybrids, and the author sees three drawbacks:

  • The flexibility of absolute paths is rarely used, since files usually sit under the table root.
  • Garbage collection is harder because a directory cannot be safely deleted if files in it are referenced from somewhere else.
  • Issuing temporary credentials is difficult, because every prefix must be found among hundreds of millions of entries in the manifests.

The promise of "zero rewrites" holds only for new tables in one location with purely relative paths. Clones, file imports, tiered storage, and bucket splitting all break it.

How multi-base works in Lance

In Lance, all paths are relative, and the directory layout is defined by the specification: data files .lance, manifests .manifest, transactions *.txn, deletion vectors, and indexes. Copy a dataset from a local disk to s3://bucket/dataset, and it works immediately without editing any metadata.

The new idea rests on an observation: even when a table references millions of files, it contains only a few distinct shared prefixes. These are called "bases", and they are stored as a separate array in the manifest. Now each file holds a reference to a base plus a relative path. This yields several practical benefits:

  • Explicit structure: there is no need to scan millions of paths to find where the data lives.
  • Cheap moves: a dataset of 10 million files across 5 buckets requires replacing 5 rows, not 10 million.
  • Flexible migration: you can move only selected bases and leave the rest in place.
  • Simpler maintenance: garbage collection and credential vending iterate over the list of bases, not the list of files.

In the specification, each base has a flag that determines how its paths are resolved. If the base is a dataset root, the standard subdirectories apply inside it, so whole datasets can be referenced, for example for shallow clones. If not, the base points directly to a folder holding data files with no subdirectories, which suits splitting across buckets, as in Uber's case.

What to keep in mind

This is the account of the Lance developer about their own solution, not an independent format comparison. The portion of the post available here contains no measured throughput gains for Uber, only the target of "roughly N times". The source article is cut off, so the details of how writes work and how multi-base relates to clones are not covered here.

The practical takeaway for teams building datasets for training and agentic search: if data has to be split across buckets and regions, choose a format where prefixes are stored explicitly. Then migration and garbage collection operate on a short table of bases rather than on millions of rows.

Story

Key Questions Answered

What are the key details on The problem: one bucket can't handle thousands of readers?

Uber's multimodal data for model training and agentic search lives in datasets at petabyte scale. Thousands of readers access them at the same time, and a single bucket quickly becomes a bottleneck. So the engineers wanted to split one dataset across N buckets. New files are written in rotation across the buckets, and reads run in parallel from all of them. The aim is to raise aggregate throughput roughly N-fold. The catch is in the paths.

What are the key details on Lessons from Iceberg and Delta: hybrids are awkward?

The author has worked on Apache Iceberg, Delta Lake, and now Lance. He notes that the two formats took opposite paths. Iceberg started with absolute paths. They are unambiguous, but migrating a table means rewriting metadata that contains thousands or millions of paths. The discussion about relative paths began in 2021, and in late 2025 the community agreed on a design for version 4: paths without a URI scheme are treated as relative to the table's location. Delta Lake went the other way.

How multi-base works in Lance?

In Lance, all paths are relative, and the directory layout is defined by the specification: data files .lance, manifests .manifest, transactions .txn, deletion vectors, and indexes. Copy a dataset from a local disk to s3://bucket/dataset, and it works immediately without editing any metadata. The new idea rests on an observation: even when a table references millions of files, it contains only a few distinct shared prefixes. These are called "bases", and they are stored as a separate array in the manifest.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…