I spend a lot of time working on crun and on how containers are kept separated from the host. Capabilities are one of the tools we use for that: instead of giving a container the full power of root, we hand out only the small pieces it needs. CAP_SYS_ADMIN gets most of the attention, since it is so powerful that people call it the new root, and dropping it is often treated as enough to make a container safe. I want to look at another capability that doesn’t sound so dangerous: CAP_SYS_PTRACE.

On its own it only lets a process debug another process, but ptrace is powerful enough to be dangerous.

We have received several security reports over the years that describe some container escape or takeover when CAP_SYS_PTRACE is granted. That is not a vulnerability, it is a configuration that is considered unsafe.

What ptrace can do#

ptrace is the system call that debuggers like gdb are built on.

Once a process is attached to another one, it can read and write its memory, read and change its registers, and make it run whatever code it wants. Attaching to a process is, for all practical purposes, the same as becoming that process.

The kernel decides who is allowed to attach in a function called ptrace_may_access(), here a simplified version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

static bool ptrace_has_cap (struct user_namespace *ns)
{
    return ns_capable (ns, CAP_SYS_PTRACE);
}

static bool task_still_dumpable (struct task_struct *task)
{
    const struct task_exec_state *exec_state;

    exec_state = task_exec_state_rcu (task);
    if (READ_ONCE (exec_state->dumpable) == TASK_DUMPABLE_OWNER)
        return true;
    return ptrace_has_cap (exec_state->user_ns);
}

static int __ptrace_may_access (struct task_struct *task)
{
    const struct cred *cred = current_cred ();
    const struct cred *tcred  = __task_cred (task);
    kuid_t caller_uid = cred->uid;
    kgid_t caller_gid = cred->gid;

    /* a thread can always look at itself */
    if (same_thread_group (task, current))
        return 0;

    if (uid_eq (caller_uid, tcred->euid) &&
        uid_eq (caller_uid, tcred->suid) &&
        uid_eq (caller_uid, tcred->uid)  &&
        gid_eq (caller_gid, tcred->egid) &&
        gid_eq (caller_gid, tcred->sgid) &&
        gid_eq (caller_gid, tcred->gid))
        goto ok;
    if (ptrace_has_cap (tcred->user_ns))
        goto ok;
    return -EPERM;
ok:
    if (!task_still_dumpable (task))
        return -EPERM;

    return security_ptrace_access_check (task);
}

ptrace_has_cap() is the helper that both capability checks go through: it just checks whether the caller has CAP_SYS_PTRACE in a given user namespace ns. Which namespace it is passed is the whole point.

__ptrace_may_access() itself has three checks:

  1. The tracer and the target run as the same user: every uid and gid of the target matches the caller. This is the common case, a debugger like gdb attaching to your own process.
  2. Or ptrace_has_cap(tcred->user_ns): the caller has CAP_SYS_PTRACE in the user namespace where the target’s credentials live. This is how a tracer running as root attaches to a process of another user.
  3. And even after one of the first two passed, task_still_dumpable() adds a last check. If the target is not dumpable, the caller needs CAP_SYS_PTRACE once more, but this time in exec_state->user_ns, the user namespace the target was in when it last called execve.

There are two different user namespaces in there: tcred->user_ns in the second check, and exec_state->user_ns in the third. That third one is where the whole story is.

What happens on exec#

When you run podman exec or docker exec, no new container is started. The OCI runtime creates a new process on the host and makes it join the container that is already running: it calls setns() for the mount, pid, network and user namespaces of the container, and only then runs your command.

The important part is that this does not happen all at once. For a short time the new process is already inside the container’s PID namespace, so every other process in the container can see it, but it is still the runtime’s own process, running as root with all the capabilities. Only after that does it drop the capabilities, switch to the container user, load the seccomp profile and finally call execv to run your command.

Sequence of a crun exec Time flows downward. crun creates the exec process, which joins the container namespaces and is briefly root with all capabilities. A malicious process in the container tries to attach to it during that window; whether the kernel allows it is the subject of the post. crun (on the host) new exec process malicious process in the container clone(), then PR_SET_DUMPABLE(0) the one step where it is root, has every capability, and is visible in the container's PID namespace setns(): now inside the container PID ns tries to attach drop caps, setuid, seccomp execv(cmd) time

So there is a small window where a fully privileged process is sitting inside the container’s PID namespace, visible to everything else in the container. It is exactly the kind of process you would want to attach to.

The attack you would expect#

A process that is already running in the container does not need to do anything clever. It scans /proc, waits for the new process to show up, and attaches to it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

/* runs inside the container, waiting for the exec */
for (;;) {
    DIR *d = opendir ("/proc");
    struct dirent *e;
    while ((e = readdir (d))) {
        pid_t pid = atoi (e->d_name);
        if (pid > 0 && is_the_exec_process (pid)) {
            ptrace (PTRACE_SEIZE, pid, 0, 0);
            /* now we can read and write its memory and registers */
        }
    }
    closedir (d);
}

If the attach succeeded, the process would be under the control of the attacker.

The good news is that with a user namespace this does not work.

Why it does not work, with a user namespace#

crun marks the exec process non-dumpable right before it joins the container, in libcrun_container_exec_with_options:

1
2
3
4
5
6

ret = prctl (PR_SET_DUMPABLE, 0, 0, 0, 0);
if (UNLIKELY (ret < 0))
  return crun_make_error (err, errno, "prctl unset dumpable");

pid = libcrun_join_process (context, container, status.pid, &status,
                            opts->cgroup, context->detach, process, ...);

execve is what sets both the dumpable flag and exec_state->user_ns: on a normal exec the kernel marks the process dumpable again and records the user namespace the process is in at that moment (fs/exec.c). crun’s exec process has not called execve yet during the window: it is still a fork of crun, which last exec’d on the host. So its exec_state->user_ns is the host’s, and the PR_SET_DUMPABLE(0) still stands. It calls execve only once it is already the unprivileged command it was asked to run, and only then becomes dumpable and tied to the container user namespace.

setns() moves the process’s credentials into the container user namespace, so the second check passes for a container process with CAP_SYS_PTRACE. But the third check does not use the credentials, it uses the exec-time user namespace, which is still the host’s, and no process in the container has CAP_SYS_PTRACE there.

ptrace_may_access on the non-dumpable exec process A process in the container is denied unless it holds CAP_SYS_PTRACE over the exec-time user namespace of the target, which for a runtime exec is the host's. process in container calls ptrace(PTRACE_SEIZE, new_pid) same creds and dumpable? no (dumpable=0) has CAP_SYS_PTRACE over the exec-time user namespace? attach succeeds yes yes (host) EPERM denied no a container

I checked this on a running kernel, with a small program that builds the same situation with plain namespaces: an attacker that is root in a new user namespace, and a non-dumpable target that exec’d on the host and then setns()’d into that user namespace. The attach is denied with EPERM. The same non-dumpable, host-owned process is also out of reach through /proc: opendir("/proc/<pid>/fd") from the container fails with EACCES, because for a non-dumpable process those files are owned by root as mapped in the exec-time user namespace, the host’s, not by anyone the container can be.

The real danger: containers without a user namespace#

All of this depends on the container having its own user namespace, and plenty of containers do not. A rootful podman or docker container runs in the host user namespace, and root in the container is root on the host.

Now the third check falls the other way. The exec process’s exec-time user namespace is the host, and a root process in such a container has CAP_SYS_PTRACE in the host user namespace, so it passes. The window is real: a process in the container can attach to the exec process while it is still privileged and take it over, read the secrets on its command line, or copy its file descriptors. The PR_SET_DUMPABLE(0) does not save you here.

What to do#

Two things. Give your containers a user namespace: it is what turns the exec process into something a compromised container cannot reach, and it does a lot more than that. And do not hand out CAP_SYS_PTRACE unless you absolutely trust the container payload that you’d run it on the host too.