For the last couple of weeks, I’ve been working on garbage collection for composefs repositories. A composefs repository is a content-addressed object store, and it needs a garbage collector to reclaim space from images that are no longer referenced. The hard part is deciding what “referenced” actually means when an image has no ref in the repository but is mounted somewhere, possibly from another process, possibly in a different mount namespace, possibly by the running system itself. On composefs-rs#346 was suggested to use flock() on the EROFS backing file. Mounters take a shared lock, the GC probes with an exclusive one, and if the exclusive lock fails, the image is still in use.

For a while now we’ve had composefs support in the container-libs. For each OCI layer in an image, we build a read-only EROFS blob that can be verified with fs-verity and that the kernel mounts directly. It works well if you look at a single container, but the moment you start thinking about composefs as the storage for a real container host where many containers are running, we have a problem. Many of them share the same image layers but with composefs every one of those containers mounts the same EROFS blobs, and each mount is a separate world as far as the kernel is concerned. The bytes on disk are shared, but the memory used to cache them is not. That is one of the reasons composefs is not yet the default way to store and run containers, and it is what I want to talk about here.

While working on crun, I got surprised by how much time the kernel spent in the copy_mount_options function. A container runtime issues a large number of mount(2) syscalls during startup — bind mounts, proc, sysfs, devtmpfs, and more — many of them with no extra options to pass. It turned out that passing an empty string instead of NULL for the data argument caused the kernel to allocate a full memory page and attempt a copy from user space on every one of those calls, adding measurable overhead.