Lance Adds Git-Style Dataset Branches: Experiments Stay Out of Production's Way
Experimenting on a production dataset is nerve-wracking: you can corrupt something or slow down live queries. In Lance, each branch gets its own folder. New files are written there, while old data is pulled from the main dataset by reference, without copying. Writes to a branch don't conflict with production, and permissions can be split so the main dataset is read-only. For ML teams, this means experiments on live data without extra copies.
AI-processed from LanceDB Blog; edited by Hamidun News
Anyone who has prepared data for training a model knows this dilemma. The dataset is live, production runs on it, and you need to try new labeling, a different feature set or augmentation. Copying terabytes is inconvenient and expensive. Experimenting directly on the production table is risky. The author of Lance and one of the creators of branching in Apache Iceberg proposed an approach in the LanceDB blog that aims to resolve this dilemma: branches, tags and "shallow cloning" in a single model.
Why Iceberg branches and Delta clones fell short
In Iceberg, where the author came up with branches and tags back in 2021, the design is nearly identical to Git: the table's metadata file holds a list of named references to snapshots. Tags protect important versions from cleanup, and branches let you accumulate changes and check them before publishing. For auditing and the write-verify-publish workflow, this works well. For ML experiments, it fell short, for three reasons.
First, performance. Any operation on any branch updates the shared metadata file, so frequent writes to an experimental branch collide with production commits and invalidate read caches. Second, weak isolation: branch files sit in the same folder with the same access permissions, so you cannot allow production read-only access while granting write access only to a branch. Third, accounting: in audit logs, queries against a branch and against production are indistinguishable, and storage used for experiments is charged to the main table's budget.
Delta Lake has offered shallow clone since late 2022: a separate table is created that references the source table's files by absolute paths. It has its own permissions, quotas and audit trail, and new data is written to its own folder. The author admits he considered this the best model for several years.
How it works in Lance
His view changed when ByteDance and Netflix independently asked for branches in Lance for AI experiments. Their arguments are as follows. A branch is always linked to its parent table, so an experiment's origin is visible without a separate lineage system. All data stays within one dataset, and cleanup follows the retention rules for branches and tags, so a clone cannot be forgotten by accident, which matters, for example, for GDPR. Finally, "create a branch, experiment, merge or discard" is familiar to any developer.
The foundation is multi-base, a scheme in which the dataset manifest lists several base paths, and each file specifies which base to read it from. Three elements come out of it:
- Shallow clone. A new independent dataset with its own location and version history. Inherited fragments are read from the source through a single base path, not through an absolute path for each file as in Delta. No data is copied; only metadata is created.
- Tags. Immutable named pointers to versions, stored in separate files. In Iceberg, tags live in table metadata and are affected by rollback. In Lance, rolling back versions does not touch them, and tagged versions are not deleted by the garbage collector.
- Branches. In effect, a lightweight clone inside the source dataset's folder plus a record of where the branch diverged.
A key difference from Git and Iceberg: Lance tracks a branch not by its latest commit but by where it was created. Branch metadata stores only the parent branch and the version the branch started from. All subsequent commits are written to the branch's own folder, with its own manifests and files. The author considers this approach best for tables with petabytes of data and millions of files.
What this means in practice
Writing to a branch doesn't touch the main branch's metadata, so there are no commit conflicts and no cache invalidation. Branch data physically lives in a separate directory, so at the storage level you can grant "read production, write only to the branch" permissions. Audit logs show which activity belongs to which branch, and storage costs are tracked per branch. In addition, each branch has its own full version history, and time travel works within it, which Iceberg does not offer. You can create nested branches, for example two A/B test variants branching from a shared experimental branch.
In Python, the workflow looks familiar: write a dataset, tag a version ("baseline", "training-v1"), open the dataset at the tagged version, create a branch with create_branch, and append experimental data to it. The main dataset stays unchanged.
Limitations
This is a design description from the project's author, not an independent benchmark. The material contains no figures on speed or savings, and practice will show how the scheme holds up under heavy load. The gains will mainly go to teams that already store training data in Lance. If you run on Iceberg or Delta, the more useful takeaway is the principle itself: experiments need a separate physical folder with separate permissions, not just a logical label within a shared table.
- Nov 21, 2026LanceDB Ships Realtime Engine: Fresh Data Searchable in 52 ms
- May 26, 2026Lance: How ByteDance's All-in-One AI Model for Video Works
- Nov 21, 2026Uber and LanceDB Teach the Lance Format to Split a Dataset Across Buckets: 5 Paths Instead of 10 Million
- Oct 31, 2026SciGame Tests Five LLM Agents: Familiar Formulas Get in the Way of Finding Altered Laws
Key Questions Answered
Why Iceberg branches and Delta clones fell short?
In Iceberg, where the author came up with branches and tags back in 2021, the design is nearly identical to Git: the table's metadata file holds a list of named references to snapshots. Tags protect important versions from cleanup, and branches let you accumulate changes and check them before publishing. For auditing and the write-verify-publish workflow, this works well. For ML experiments, it fell short, for three reasons. First, performance.
How it works in Lance?
His view changed when ByteDance and Netflix independently asked for branches in Lance for AI experiments. Their arguments are as follows. A branch is always linked to its parent table, so an experiment's origin is visible without a separate lineage system. All data stays within one dataset, and cleanup follows the retention rules for branches and tags, so a clone cannot be forgotten by accident, which matters, for example, for GDPR.
What this means in practice?
Writing to a branch doesn't touch the main branch's metadata, so there are no commit conflicts and no cache invalidation. Branch data physically lives in a separate directory, so at the storage level you can grant "read production, write only to the branch" permissions. Audit logs show which activity belongs to which branch, and storage costs are tracked per branch. In addition, each branch has its own full version history, and time travel works within it, which Iceberg does not offer.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.