OCI containers and Python

I’ve recently pushed Python bindings to the crun git repo. Handling JSON data is extremely easy in Python and makes it a great candidate to quickly customize the container configuration and launch it. Keep in mind that crun is only an OCI runtime for running a container, it doesn’t take care of pulling and storing images. You’ll need a different tool for that.

For more details, this is the commit that adds the Python bindings: https://github.com/giuseppe/crun/commit/a2b5faf88a4abde241ad813afdf6688e38fb7ca2

By default the Python bindings are not built, you’ll need to specify –with-python-bindings to the configure script to build them.

Once you have the Python bindings, a simple script can be used to customize and launch a container:


import python_crun
import json

# the spec() function generates a default configuration for
# running the OCI container, parse directly the JSON. 
spec = json.loads(python_crun.spec())

# Let's change the rootfs of the container to point to the correct
# path.  This is where the image is exploded on disk.
spec['root']['path'] = '/home/gscrivano/example-container/rootfs'

# We don't want the container to do much, just greet us
spec['process']['args'] = ['/bin/echo', 'hello from a container']

# From the customized configuration, we create the OCI container.
ctr = python_crun.load_from_memory(json.dumps(spec))

# The context says what ID the container will have, and optionally allows
# to tweak other settings as well such as the state root, using systemd
# for the cgroups
ctx = python_crun.make_context("test-container")

# We don't want to print any warning.
python_crun.set_verbosity(python_crun.VERBOSITY_ERROR)

# And finally run the container
python_crun.run(ctx, ctr)

Store it in a launch_container.py file.

Now, from the crun build root directory:


# PYTHONPATH=.libs python ./launch_container.py
hello from a container

Current status (and problems) of running Buildah as non root

Having Buildah running in an user namespace opens the possibility of building container images as a not root user. I’ve done some work to get Buildah running in an user container.

There are still some open issues to get it fully working. The biggest open one is that overlayfs cannot be currently used as non root user. There is some work going on, but this will require changes in the kernel and the way extended attributes work for overlay. The alternative is far from ideal and it is to use the vfs storage driver, but it is a good starting point to get things moving and see how far we get. (Another possibility that doesn’t require changes in the kernel would be an OSTree storage for Buildah, but that is a different story).

Circumvented the first obstacle, the other big issue was to get a container, that is created for every buildah run command, the Buildah version of the RUN directive in a Dockerfile. That means run a container inside of a container.

The default runtime for atomic –user is bwrap-oci, a tool that converts a subset of the OCI configuration file to a command line for bubblewrap, the real engine for running the container. There is an open issue with bubblewrap, that as part of the container setup, move the container in a chroot. This will prevent further containers to be created as for the unshare(2) man page, you can get an EPERM if:

EPERM (since Linux 3.9)
CLONE_NEWUSER was specified in flags and the caller is in a chroot environment (i.e., the caller’s root directory does not match the root directory of the mount namespace in which it resides).

This problem is tracked here: https://github.com/projectatomic/bubblewrap/pull/172. Once that is merged, together with some other small changes in bwrap-oci I got the container running and bubblewrap could be used both as the runtime for running the Buildah container that for the runtime for managing the containers created by Buildah.

I wanted to give it a try with runc as well as the container runtime. There is a lot of development going on upstream for running containers as not root user, but it also failed to run in an user namespace when it tried to setup the cgroups.

To get a better understanding of what could the solution for having a full OCI runtime managing these containers, I wrote some patches for crun, partly because it is my pet project and also as it is still experimental, it is much easier to quickly throw a bunch of patches at it and not be worried to make someone sad. I’ve added some code to detect when the container is running in an user namespace and relax some error conditions to deal with the limitations in such environment. Even if the user id is 0 the runtime doesn’t still have full control of the system.

The container image that I’ve prepared is hosted on Docker hub at docker.io/gscrivano/buildah.

Given you use the latest version crun from git and of the atomic CLI tool (that supports –runtime) you can run the container as:


$ atomic run --runtime /usr/bin/crun --storage ostree docker.io/gscrivano/buildah /host/$(pwd)/build.sh

The build.sh script looks very similar to the example on the Buildah github page. It is a shell script that looks like:


#!/bin/bash -x

export HOME=/host/$(pwd)

ctr1=`buildah --storage-driver vfs from --pull ${1:-docker.io/fedora:27}`

buildah --storage-driver vfs run --runtime /host/usr/bin/crun --runtime-flag systemd-cgroup $ctr1 -- dnf  upgrade -y

buildah --storage-driver vfs run --runtime /host/usr/bin/crun --runtime-flag systemd-cgroup $ctr1 -- dnf install -y lighttpd

buildah --storage-driver vfs config $ctr1 --annotation "com.example.build.host=fedora-27"

buildah --storage-driver vfs config $ctr1 --cmd "/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf"
buildah --storage-driver vfs config $ctr1 --port 80

buildah --storage-driver vfs commit $ctr1  giuseppe/lighttpd

We got very close, but it doesn’t work yet the last `commit` command fails as vfs got broken upstream: https://github.com/containers/storage/issues/96#issuecomment-368307230. We’ve built a container in an user namespace, but we cannot share it with anyone šŸ™‚

New COPR repository for crun

I made a new COPR repository for CRUN so that it can be easily tested on Fedora:

https://copr.fedorainfracloud.org/coprs/gscrivano/crun/

To install crun on Fedora, it is enough to:


# dnf install 'dnf-command(copr)'
# dnf -y copr enable gscrivano/crun
# dnf install -y crun

a recent change in the atomic tool, which didn’t still get into a release, allows to easily override the OCI runtime for system containers. Assuming you are using atomic from the upstream repository, you can use crun as:


# atomic install --system ----runtime /usr/bin/crun registry.fedoraproject.org/f27/etcd
# systemctl start etcd

It will install etcd as a system container which runs through crun!

You might need to disable SELinux as the /usr/bin/crun executable is not yet labelled correctly.

Facebook detox?Ā 

I have been using Facebook for the last years to fill every dead time:waiting for the bus, ads on TV, compiling, etc.  The quality of the information coming from Facebook is inferior to any other social network, at least to my experience (it can be I follow/know the wrong people), though the part of the brain that controls procrastination seems addicted to this lower quality information and the chattering there.  Also, I don’t want to simply delete my Facebook account and move on, most of the people I know are present only there, neither I want to be more “asocial”.

The Android market has always a solution.    An app let you define rules on how long are you permitted to use each app.  I am self limiting myself to ten minutes per day of Facebook.  Second day and the rule is still in place without exceptions! 

Refactoring a function name across several patches with git rebase

git rebase is one of my favorite git commands. It allows to update a set of local patches against another git branch and also to rework, trough the -i flag some previous patches.

The problem I had to deal with was quite simple, rename a function called notProperPythonCode to proper_python that was defined in the first patch and be sure that all other patches are using the correct name. The –exec flag allows to run a custom script after each patch is applied, so that I could run sed to process the Python files and replace the old function name with the new one. The process is quite simple, except that such changes would trigger a lot of merge conflicts, trivial to solve but quite annoying.
Fortunately git rebase allows to choose what merge strategy must be adopted for solving conflicts, the theirs strategy in case of a conflict, will take the previous version of the patch and silently use it. That is fine for this simple substitution case, where we process each file ending by *.py in the repository.

In the end, the final command looked like this:


git rebase -X theirs -i \
  --exec "sh -c \"git ls-files *.py \
  | xargs sed --follow-symlinks -i -e \
   's|notProperPythonCode|proper_python|g'\" \
  && git commit --amend -a -C HEAD" origin/master

System containers for Atomic

The main reason behind system containers was the inability to run Flannel in a Docker container as Flannel is required by Docker itself. CoreOS solved this chicken and egg problem by using another instance of Docker (called early-docker) that is used to setup only Etcd and Flannel.

Differently, Atomic system containers will be managed by runc and systemd.

The container images, even though being served through the Docker v2 registry, are slighty different than a regular Docker container in order to be used by Atomic. The installer expects that some files are present in the container rootfs under /exports, the OCI spec file for running the Runc container and the unit file for Systemd. Both these files are templates and some values are replaced by the installer.
The communication with the Docker registry is done through Skopeo, that is used internally by the atomic cli tool.

The demo shows the current status of the project, these features are not still part of any release and most likely there will be more changes before they will be released.

I wanted to give a try to asciinema, this is the result:

The images used in the demo are maintained here: https://github.com/giuseppe/atomic-oci-containers/

More details on the current design can be found here: https://github.com/projectatomic/atomic/issues/298.