DuckLake 1.0: A Laptop Data Lake Built from DuckDB and a Folder of Parquet Files in an Hour
The DuckDB team released a stable DuckLake 1.0 back in April. Tables live in ordinary Parquet files, while everything else (the table list, schemas, snapshots) is stored in a regular SQL database. KDnuggets showed how to build such a lake on a laptop in six Python scripts: load 10,000 orders, update rows, change a schema, and roll back an accidental deletion. For small analytics projects, this is the shortest path to data versioning.
AI-processed from KDnuggets; edited by Hamidun News
The word lakehouse usually brings to mind clusters, cloud storage, and weeks of setup. The DuckDB team offers a different path: DuckLake, an open table format that can run on a laptop. Version 1.0 shipped in April 2026 with a stable specification and a backward compatibility guarantee. KDnuggets published a step-by-step walkthrough of how to build a working data lake in under an hour.
How DuckLake Works
The idea is simple. The data itself lives in Parquet files in a folder or in object storage. Everything else (the list of tables, schemas, snapshots, and per-file statistics) lives in the catalog, which is an ordinary SQL database. Other lakehouse formats store this information in auxiliary metadata files; here, finding the right files becomes a single SQL query.
The catalog can be any SQL database with ACID transactions and primary keys. The authors recommend choosing based on the scenario: DuckDB for a single client, another local database for several clients on one machine, and PostgreSQL for a multi-user lake. The tutorial uses the simplest option, a DuckDB file. You need Python 3 and the duckdb package, version 1.5.2 or higher. The first run requires internet access to download an extension.
What the Tutorial Shows
Six scripts run in sequence. First, DuckDB generates two months of synthetic orders, with 5,000 rows in each Parquet file. Then the lake is created and January is loaded into it. A snapshot helper function returns a table with a version number, a timestamp, and a short description of the changes. Where other formats force you to parse files, a regular query is enough.
Next, the data is changed. February is added, several hundred rows are updated (orders "in transit" are marked as delivered), and the table schema is altered. Each action creates a new snapshot. Updates work as a delete plus an insert: old rows are hidden by a delete file, and new versions are written separately. A schema change touches only metadata: Parquet files are not rewritten, and the catalog maps the old files to the new shape of the table.
The most striking feature is time travel. A query with the AT (VERSION => number) clause returns the table as it was at the required moment, along with the old column names. You can also query by time, for example "10 minutes ago." The author deliberately breaks the data: deletes all orders from the west region, then reinserts them from a snapshot where they still existed. The count returns to 10,000 rows. To avoid hardcoding version numbers in code, the scripts save snapshot markers to a small JSON file.
Maintenance comes down to a function that merges small adjacent files into larger ones. It merges only files that share the same schema version.
Pitfalls
The author flags several things separately.
- DuckLake never deletes old files on its own: they are needed for time travel. To free up space, you must first delete old snapshots and then clean up the files. But after that you can no longer return to those versions, so the retention period needs to be set deliberately.
- Small writes (up to 10 rows by default) go directly into the catalog rather than into a new Parquet file. They can be moved out later with a separate function.
- The format does not support indexes, primary and foreign keys, or UNIQUE and CHECK constraints.
- A catalog in the form of a DuckDB file is designed for a single client. For multiple users, you create the lake on PostgreSQL: only the connection string changes, and the other queries stay the same.
What's Next
The version 1.0 announcement mentions possible directions for DuckLake 2.0: git-like branches and incremental materialized views. These are plans for now, not promises. The author advises reading the 2.0 release notes before upgrading.
The practical takeaway for developers and analysts is this. If you need table versions, error rollback, and schema evolution, but don't want to stand up Spark or a separate metadata service, DuckLake gives you all of that on a single laptop. You can start with a local file and move the catalog to PostgreSQL when the team grows. The key is to plan your snapshot cleanup policy in advance.
- Nov 12, 2026AWS publishes a Bedrock data lake template that shows every prompt, token and its cost
- Apr 30, 2026Aleksey Seleznev releases a free R video course on building AI tools
- Oct 10, 2026ReviewScope hunts for anomalies in reviews using embeddings and graphs across 21,000 texts
- Nov 8, 2026A non-programmer recreated Karpathy's HackerTube for Habr: 388 videos in one evening
Key Questions Answered
How DuckLake Works?
The idea is simple. The data itself lives in Parquet files in a folder or in object storage. Everything else (the list of tables, schemas, snapshots, and per-file statistics) lives in the catalog, which is an ordinary SQL database. Other lakehouse formats store this information in auxiliary metadata files; here, finding the right files becomes a single SQL query. The catalog can be any SQL database with ACID transactions and primary keys.
What the Tutorial Shows?
Six scripts run in sequence. First, DuckDB generates two months of synthetic orders, with 5,000 rows in each Parquet file. Then the lake is created and January is loaded into it. A snapshot helper function returns a table with a version number, a timestamp, and a short description of the changes. Where other formats force you to parse files, a regular query is enough. Next, the data is changed.
What are the key details on Pitfalls?
The author flags several things separately. DuckLake never deletes old files on its own: they are needed for time travel. To free up space, you must first delete old snapshots and then clean up the files. But after that you can no longer return to those versions, so the retention period needs to be set deliberately. Small writes (up to 10 rows by default) go directly into the catalog rather than into a new Parquet file. They can be moved out later with a separate function.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.