Modal speeds up bun install in VM sandboxes 12x by replacing virtio-fs with a block disk

If you build sandboxes for agents, you know that a fast start doesn't mean fast work. An agent installs dependencies, runs grep across a repository, and changes permissions on thousands of files. All of this bottlenecks on the file system. Modal's engineers described how they got burned by this when launching VM Sandboxes and what they did about it.

Fast to start, slow to work

Modal has long had ImageFS, a distributed file system with lazy loading for containers. It pre-fetches only the metadata index and pulls file contents on demand from a multi-tier cache: memory, regional CDNs, and global storage. That is why containers start in under a second. OverlayFS sits on top for writes.

With VMs the logic is the same: a VM has its own guest kernel and no direct access to ImageFS on the host. The standard bridge is virtio-fs, where the virtiofsd daemon on the host applies guest requests to a shared directory. They pointed it at ImageFS, and startup stayed the same.

But in actual work everything fell apart. A recursive chown that used to take four seconds took more than a minute. bun install and pnpm install became roughly three times slower, and ripgrep suffered the most. In a microbenchmark of 100,000 calls, chown took 9.87 s versus 2.31 s under gVisor and 1.58 s on bare ImageFS.

Why a single chown turns into three round trips

Tracing revealed the key issue. A single chown in the guest kernel breaks down into three requests to the host: getxattr to check capabilities, getattr for setuid bits, and setattr for the ownership change itself. The daemon makes six system calls, and each of them hits ImageFS. The total is about 99 µs per operation, even though virtiofsd was already configured for maximum caching.

The root of the problem is architectural. virtio-fs is a shared file system: for metadata, you must always go to the authority on the other side of the VM boundary. Yet the host doesn't need the contents of the VM's root file system at all.

Speaking in blocks

The engineers proposed keeping the file system entirely inside the guest and giving the host only a block device via virtio-blk. A disk doesn't know what a file is: it stores and serves chunks of data. The guest ext4 checks permissions itself, clears bits, and writes the change to its journal. Everything reaches the host later, in a batch, once every few seconds. A single chown costs about 2.1 µs instead of 99.

The difficulty is that mkfs.ext4 requires the whole image to be present, which kills lazy loading. So the team writes the ext4 inode records itself from the image metadata and leaves the rest sparse. The ImageFS-blk daemon turns block requests into reads of file ranges from ImageFS, and puts written blocks into a sparse file and marks them in a bitmap. It is the equivalent of OverlayFS, only at the block level. It also removes an old pain point: copy-on-write operates on a single block rather than a whole file. In the worst case, OverlayFS copied gigabytes for the sake of a few bytes.

The results and the catch

The numbers are as follows (virtio-fs, ImageFS, ImageFS-blk):

• bun install: 12.5 s, 3.1 s, 1.07 s;
• ripgrep over the Linux repository with a warm cache: 6.03 s, 3.77 s, 0.18 s;
• SQLite speedtest1: 543 s, 290 s, 43 s;
• pgbench read-only: 98k, 137k, 230k TPS;
• pgbench read-write: 8.3k, 11.9k, 18.8k TPS.

ImageFS-blk even outperformed the original ImageFS: metadata no longer goes through FUSE. Cold start within a second was preserved.

The approach has a cost. The host no longer sees the files inside the VM: it knows only blocks, and snapshots are computed from the set of changed blocks. And with a cold cache, ripgrep over the repository takes 8.53 s on ImageFS-blk versus 4.47 s on regular ImageFS. The block scheme loses there. All figures were obtained by Modal itself on its own hardware; there is no independent verification.

What to do with this

If you are building micro-VMs for agents or CI, don't accept virtio-fs as the default root file system. For a single sandbox where the host doesn't read the guest's files, a shared file system is an unnecessary tax on every metadata operation. Run not only cold start but also your real workload: package installation, code search, chown, and a database. That is exactly where a 10x difference shows up.