Flatcar: All Docker containers fail to start after update to 2983.2.0

Description

After an update to 2983.2.0, all Docker containers fail to start with the message: standard_init_linux.go:228: exec user process caused: operation not permitted

Impact

Cannot start any containers. All containers fail to start at boot.

Environment and steps to reproduce

  1. Set-up: Flatcar version 2983.2.0
  2. Task: Booting up
  3. Action(s): None
  4. Error: All containers fail with the error: standard_init_linux.go:228: exec user process caused: operation not permitted

Expected behavior

Containers should start without error. I cannot start a basic container with bash from the CL either:

$ docker container run --interactive --tty --rm ubuntu bash
Unable to find image 'ubuntu:latest' locally
docker.io/library/ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c: Pulling from library/ubuntu
da7391352a9b: Pull complete
14428a6d4bcd: Pull complete
2c2d948710f2: Pull complete
Digest: sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
Status: Downloaded newer image for ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
Tagging ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c as ubuntu:latest
standard_init_linux.go:228: exec user process caused: operation not permitted

Additional information

$ docker info -f "{{json .}}" | jq ".SecurityOptions"
[
  "name=seccomp,profile=default",
  "name=selinux",
  "name=userns",
  "name=cgroupns"
]
$ systemctl cat docker.service
# /run/systemd/system/docker.service
[Unit]
Requires=torcx.target
After=torcx.target
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=containerd.service docker.socket network-online.target
Wants=network-online.target
Requires=containerd.service docker.socket

[Service]
EnvironmentFile=/run/metadata/torcx
Environment=TORCX_IMAGEDIR=/docker
Type=notify
EnvironmentFile=-/run/flannel/flannel_docker_opts.env
Environment=DOCKER_SELINUX=--selinux-enabled=true

# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/env PATH=${TORCX_BINDIR}:${PATH} ${TORCX_BINDIR}/dockerd --host=fd:// --containerd=/var/run/docker/libcontainerd/docker-containerd.sock $DOCKER_SELINUX $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/docker.service.d/docker-opts.conf
[Service]
Environment="DOCKER_OPTS="

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 33 (14 by maintainers)

Most upvoted comments

I’m 99% sure its https://www.aquasec.com/ enforcer that has created custom policies and that’s why some clusters have issues and others don’t.

@meltonbw do you know why your selinux config was customized and not directly upstream via links?

@serbaut thanks for your help and the data you provided - it’s really useful. You might be interested to run some Beta nodes in your cluster to try to catch this kind of behavior earlier.

We actually have that in another cluster but they didnt experience any issue 😛

This is how it looks on a bad node

# ls -ld /etc/selinux/* /var/lib/selinux
lrwxrwxrwx. 1 root root   28 Apr 27  2021 /etc/selinux/config -> ../../usr/lib/selinux/config
drwxr-xr-x. 4 root root 4096 May 18 15:55 /etc/selinux/mcs
lrwxrwxrwx. 1 root root   25 Apr 27  2021 /etc/selinux/mls -> ../../usr/lib/selinux/mls
lrwxrwxrwx. 1 root root   35 May 18 13:43 /etc/selinux/semanage.conf -> ../../usr/lib/selinux/semanage.conf
lrwxrwxrwx. 1 root root   30 Apr 27  2021 /etc/selinux/targeted -> ../../usr/lib/selinux/targeted
drwxr-xr-x. 5 root root 4096 May 18 15:55 /var/lib/selinux

vs

# ls -ld /etc/selinux/* /var/lib/selinux
lrwxrwxrwx. 1 root root 28 Apr 27  2021 /etc/selinux/config -> ../../usr/lib/selinux/config
lrwxrwxrwx. 1 root root 25 Apr 27  2021 /etc/selinux/mcs -> ../../usr/lib/selinux/mcs
lrwxrwxrwx. 1 root root 25 Apr 27  2021 /etc/selinux/mls -> ../../usr/lib/selinux/mls
lrwxrwxrwx. 1 root root 35 May 18 08:42 /etc/selinux/semanage.conf -> ../../usr/lib/selinux/semanage.conf
lrwxrwxrwx. 1 root root 30 Apr 27  2021 /etc/selinux/targeted -> ../../usr/lib/selinux/targeted
lrwxrwxrwx. 1 root root 28 Apr 27  2021 /var/lib/selinux -> ../../usr/lib/selinux/policy

So depending on which image you started with you get different selinux configs.

Do you have the same runtime spec: SELinux in enforcing mode, no-new-privileges and so on ?

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             mcs
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

We dont have any docker config (node is running k8s).