*scratch*

Additional IDs that are allocated to a user through /etc/subuid and /etc/subgid must be considered as permanently allocated and never reused for any other user.

Even if the container/user namespace where they are used is destroyed, it is possible to forge a SUID binary that will keep access to any ID present in the user namespace.

This simple C program is enough to keep access to an UID that was allocated to a user namespace:

#define _GNU_SOURCE
#include <unistd.h>
#include <sys/types.h>

int main (int argc, char **argv)
{
	uid_t u = geteuid ();
	setresuid (u, u, u);
	execvp (argv[1], argv + 1);
}

with that in place, from the user namespace:

$ id -u # ID 0 is mapped to ID 1000 in the host
0
$ gcc program.c -o keep_id
$ chown 10:10 keep_id
$ chmod +s keep_id

even once the user namespace is destroyed and possibly the range of allocated subids changed for the user, from the host we can still get access to whatever ID was allocated to the user 10 in the user namespace:

$ id -u
1000
$ ls -l keep_id
-rwsr-sr-x. 1 100009 100009 18432 Jan 10 22:23 keep_id
$ ./keep_id id -u
100009

would be nice to have a way to “fork” the current session and be able to revert all the changes done, without any leftover on the file system.

Playing with fuse-overlayfs, a FUSE implementation of the overlay file system and thus usable by rootless users, I realized how that is so easy to achieve, just by setting the overlay lowerdir to ‘/’ and using a temporary directory for the upper dir.

The upper dir, where all the overlay changes are written can be deleted once the session is over, or re-used to get back the created session.

This simple setup also enables the use case of an unprivileged user that can install packages using the existing system as a base. With few caveats (e.g. /var/log must be writeable) I managed to run dnf and install a few packages on top of my system without the need of the root user. Obviously the rest of the system didn’t notice any change, as these files were visible only from the fuse-overlayfs mount and the mount namespace using it.

Perhaps a tool could help managing similar setups. The biggest problem is in how to address the assumption the lower layer won’t change, or at least not enough to cause any breakage in the layered session.

I was looking for an Emacs mode that could help me to hack on rust.

Rust-mode itself has not enough features to help me with a language I am not really proficient with yet.

I wanted to give a try to racer, which is available in the emacs packages list.

The rust toolchain available on Fedora 29 seems not able to build racer, so the first step was to install rust from rustup.rs and pretended it is completely fine to pipe curl into sh.

Once rustup was installed, I then needed the nightly toolchain and the rust-src component so that racer is able to navigate into the Rust source code.

$ curl https://sh.rustup.rs -sSf | sh
$ cargo +nightly install racer
$ rustup component add rust-src

At this point I installed the racer-mode package from Emacs.  To do so, M-xthen list-packages, in the new buffer find racer-mode, then press I to select it and X to install it.  If you don’t find it in the list of the packages, you might need to configure the packages archive to use.  This is what I am using from ~/.emacs:

(when (>= emacs-major-version 24)
  (require 'package)
  (add-to-list
   'package-archives
   '("melpa" . "http://melpa.org/packages/")
   t)
  (package-initialize))

Finally I had to configure in my ~/.emacs file the path for racer to look for source files:

(setq racer-rust-src-path "~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/")

At this point everything is configured to start using racer-mode.

You can open a .rs file, and set the mode with M-x racer-mode (if you are happy with it, you can configure it for any file with the .rs extension).

For a quick try, using M-. on a stdlib function name should bring you to the definition of the function, M-, to go back to its usage.

this is the recipe I use to build podman from upstream on Centos 7 and use rootless containers. We need an updated version of the shadow utils as newuidmap and newgidmap are not present on Centos 7. Using make install is not the correct way to install packages, and it will also overwrite existing The shadow utils are installed using “make install” which is not the clean way to install packages and it also overwrite the existing binaries, but it is fine on a development system. Podman is already present on Centos 7 and in facts we install it so we don’t have to worry about conmon and other dependencies.

$ sudo yum install -y golang runc git ostree-devel gpgme-devel device-mapper-devel btrfs-progs-devel libassuan-devel libseccomp-devel automake autoconf gettext-devel libtool libxslt libsemanage-devel bison libcap-devel podman
$ go get -u github.com/containers/libpod/cmd/podman

$ (git clone https://github.com/shadow-maint/shadow; cd shadow; ./autogen.sh --prefix=/usr --enable-man; make && sudo make -C src install)

$ (git clone https://github.com/rootless-containers/slirp4netns.git; cd slirp4netns; ./autogen.sh; ./configure --prefix=/usr; make -j $(nproc); sudo make install)

$ sudo bash -c 'echo 10000 > /proc/sys/user/max_user_namespaces'

$ sudo bash -c "echo $(whoami):110000:65536 > /etc/subuid"

$ sudo bash -c "echo $(whoami):110000:65536 > /etc/subgid"

and:

$ go/bin/podman pull alpine
$ go/bin/podman run --net host --rm -ti alpine echo hello
hello

a couple of weekends ago I’ve played with libslirp and put together slirp-forwarder.

SliRP emulates in userspace a TCP/IP stack. It can be used to circumvent the limitation of creating TAP/TUN devices in the host namespace for an unprivileged user. The program could run in the host namespace, receive messages from the network namespace where a TAP device is configured, and forward them to the outside world using unprivileged operations such as opening another connection to the destination host. Privileged operations are still not possible outside of the emulated network, as the helper program doesn’t gain any additional privilege that running as an unprivileged user.

Once the PoC was ready, I discovered there was already another tool by Akihiro Suda (@AkihiroSuda), slirp4netns that was doing exactly the same thing, and it was already using the better slirp implementation in QEMU, that is used for configuring unprivileged virtual machines.

slirp4netns was added to the rootlesscontainers github organization, and its repo can be found here: https://github.com/rootless-containers/slirp4netns

With some small changes to slirp4netns, it was possible to integrate slirp4netns into Podman for the configuration of an unprivileged network namespace. For example, we needed a way to terminate the slirp4netns program once the container exits, allow to configure the interface and notify Podman back once the configuration is done.

$ podman run --rm alpine ifconfig -a
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

tap0      Link encap:Ethernet  HWaddr CE:CE:E1:0A:4B:F9  
          inet addr:10.0.2.100  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::ccce:e1ff:fe0a:4bf9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:90 (90.0 B)

This is how it looks from the host, the arguments to slirp4netns in addition to some fd used for the synchronization, are the PID of a process in the network namespace to configure and the device name.

$ bin/podman run --rm alpine sleep 10 &
[1] 10360
$ pgrep -fa slirp
10460 /usr/bin/slirp4netns -c -e 3 -r 4 10447 tap0

I’ve cleaned up some C files I was using locally for hacking with user namespaces and uploaded them to a new repository on github: https://github.com/giuseppe/become-root.

Creating an user namespace can be easily done with unshare(1) and get the current user mapped to root with unshare -r COMMAND but it doesn’t support the mapping of multiple uids/gids. For doing that it is necessary to use the suid newuidmap and newgidmap tools, that allocates multiple uids/gids to unprivileged users accordingly to the configuration files:

• /etc/subuid: for additional UIDs
• /etc/subgid: for additional GIDs

$ grep gscrivano /etc/subuid
gscrivano:110000:65536

$ become-root cat /proc/self/uid_map 
         0       1000          1
         1     110000      65536

The uid_map file under /proc shows the mappings used by the process.

become-root doesn’t allow any customization, it statically maps the current user to the root in the user namespace and any additional uid/gid are mapped starting from 1.

One feature that might be nice to have is to allow the creation of other namespaces as part of the same unshare syscall, such as creating a mount or network namespace, but I’ve not added this feature as I am not using it, I rely on unshare(1) for more features. PR are welcome.

The project I was working on in the last weeks was moved under the github.com/containers umbrella.

With Linux 4.18 it will be possible to mount a FUSE file system in an user namespace. fuse-overlayfs is an implementation in user space of the overlay file system already present in the Linux kernel, but that can be mounted only by the root user. Union file systems were around for a long time, allowing multiple layers to be stacked on top of each other where usually the last one is the only writeable.
Overlay is an union file system widely used for mounting OCI image. Each OCI image is made up of different layers, each layer can be used by different images. A list of layers, stacked on each other gives the final image that is used by a container. The last level, that is writeable, is specific for the container. This model enables different containers to use the same image that is accessible as read-only from the lower layers of the overlay file system.

The current implementation of the overlay file system is done directly in the kernel, at a very low level, allowing non privileged users to use it directly poses some security risks. In the longer term, once the security aspect is resolved, non privileged users will probably be able to mount directly an overlay file system.

For now, given the new feature in Linux 5.18, having an implementation of the overlay union in user space will enable rootless containers to use the same storage as containers running as root.

On Fedora Rawhide, where Linux 4.18 is available, it is already possible to take a taste of it with:

podman --storage-opt overlay2.fuse_program=/usr/bin/fuse-overlayfs run ...

The previous command tells podman to mount an overlay file system using the specified FUSE helper instead of mounting it directly through the kernel.

Having Buildah running in an user namespace opens the possibility of building container images as a not root user. I’ve done some work to get Buildah running in an user container.

There are still some open issues to get it fully working. The biggest open one is that overlayfs cannot be currently used as non root user. There is some work going on, but this will require changes in the kernel and the way extended attributes work for overlay. The alternative is far from ideal and it is to use the vfs storage driver, but it is a good starting point to get things moving and see how far we get. (Another possibility that doesn’t require changes in the kernel would be an OSTree storage for Buildah, but that is a different story).

Circumvented the first obstacle, the other big issue was to get a container, that is created for every buildah run command, the Buildah version of the RUN directive in a Dockerfile. That means run a container inside of a container.

The default runtime for atomic –user is bwrap-oci, a tool that converts a subset of the OCI configuration file to a command line for bubblewrap, the real engine for running the container. There is an open issue with bubblewrap, that as part of the container setup, move the container in a chroot. This will prevent further containers to be created as for the unshare(2) man page, you can get an EPERM if:

EPERM (since Linux 3.9)
CLONE_NEWUSER was specified in flags and the caller is in a chroot environment (i.e., the caller’s root directory does not match the root directory of the mount namespace in which it resides).

This problem is tracked here: https://github.com/projectatomic/bubblewrap/pull/172. Once that is merged, together with some other small changes in bwrap-oci I got the container running and bubblewrap could be used both as the runtime for running the Buildah container that for the runtime for managing the containers created by Buildah.

I wanted to give it a try with runc as well as the container runtime. There is a lot of development going on upstream for running containers as not root user, but it also failed to run in an user namespace when it tried to setup the cgroups.

To get a better understanding of what could the solution for having a full OCI runtime managing these containers, I wrote some patches for crun, partly because it is my pet project and also as it is still experimental, it is much easier to quickly throw a bunch of patches at it and not be worried to make someone sad. I’ve added some code to detect when the container is running in an user namespace and relax some error conditions to deal with the limitations in such environment. Even if the user id is 0 the runtime doesn’t still have full control of the system.

The container image that I’ve prepared is hosted on Docker hub at docker.io/gscrivano/buildah.

Given you use the latest version crun from git and of the atomic CLI tool (that supports –runtime) you can run the container as:

$ atomic run --runtime /usr/bin/crun --storage ostree docker.io/gscrivano/buildah /host/$(pwd)/build.sh

The build.sh script looks very similar to the example on the Buildah github page. It is a shell script that looks like:

#!/bin/bash -x

export HOME=/host/$(pwd)

ctr1=`buildah --storage-driver vfs from --pull ${1:-docker.io/fedora:27}`

buildah --storage-driver vfs run --runtime /host/usr/bin/crun --runtime-flag systemd-cgroup $ctr1 -- dnf  upgrade -y

buildah --storage-driver vfs run --runtime /host/usr/bin/crun --runtime-flag systemd-cgroup $ctr1 -- dnf install -y lighttpd

buildah --storage-driver vfs config $ctr1 --annotation "com.example.build.host=fedora-27"

buildah --storage-driver vfs config $ctr1 --cmd "/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf"
buildah --storage-driver vfs config $ctr1 --port 80

buildah --storage-driver vfs commit $ctr1  giuseppe/lighttpd

We got very close, but it doesn’t work yet the last `commit` command fails as vfs got broken upstream: https://github.com/containers/storage/issues/96#issuecomment-368307230. We’ve built a container in an user namespace, but we cannot share it with anyone 🙂

I made a new COPR repository for CRUN so that it can be easily tested on Fedora:

https://copr.fedorainfracloud.org/coprs/gscrivano/crun/

To install crun on Fedora, it is enough to:

# dnf install 'dnf-command(copr)'
# dnf -y copr enable gscrivano/crun
# dnf install -y crun

a recent change in the atomic tool, which didn’t still get into a release, allows to easily override the OCI runtime for system containers. Assuming you are using atomic from the upstream repository, you can use crun as:

# atomic install --system ----runtime /usr/bin/crun registry.fedoraproject.org/f27/etcd
# systemctl start etcd

It will install etcd as a system container which runs through crun!

You might need to disable SELinux as the /usr/bin/crun executable is not yet labelled correctly.