Flatcar: All Docker containers fail to start after update to 2983.2.0
Description
After an update to 2983.2.0, all Docker containers fail to start with the message: standard_init_linux.go:228: exec user process caused: operation not permitted
Impact
Cannot start any containers. All containers fail to start at boot.
Environment and steps to reproduce
- Set-up: Flatcar version 2983.2.0
- Task: Booting up
- Action(s): None
- Error: All containers fail with the error:
standard_init_linux.go:228: exec user process caused: operation not permitted
Expected behavior
Containers should start without error. I cannot start a basic container with bash from the CL either:
$ docker container run --interactive --tty --rm ubuntu bash
Unable to find image 'ubuntu:latest' locally
docker.io/library/ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c: Pulling from library/ubuntu
da7391352a9b: Pull complete
14428a6d4bcd: Pull complete
2c2d948710f2: Pull complete
Digest: sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
Status: Downloaded newer image for ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
Tagging ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c as ubuntu:latest
standard_init_linux.go:228: exec user process caused: operation not permitted
Additional information
$ docker info -f "{{json .}}" | jq ".SecurityOptions"
[
"name=seccomp,profile=default",
"name=selinux",
"name=userns",
"name=cgroupns"
]
$ systemctl cat docker.service
# /run/systemd/system/docker.service
[Unit]
Requires=torcx.target
After=torcx.target
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=containerd.service docker.socket network-online.target
Wants=network-online.target
Requires=containerd.service docker.socket
[Service]
EnvironmentFile=/run/metadata/torcx
Environment=TORCX_IMAGEDIR=/docker
Type=notify
EnvironmentFile=-/run/flannel/flannel_docker_opts.env
Environment=DOCKER_SELINUX=--selinux-enabled=true
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/env PATH=${TORCX_BINDIR}:${PATH} ${TORCX_BINDIR}/dockerd --host=fd:// --containerd=/var/run/docker/libcontainerd/docker-containerd.sock $DOCKER_SELINUX $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/docker-opts.conf
[Service]
Environment="DOCKER_OPTS="
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 33 (14 by maintainers)
I’m 99% sure its https://www.aquasec.com/ enforcer that has created custom policies and that’s why some clusters have issues and others don’t.
@meltonbw do you know why your selinux config was customized and not directly upstream via links?
We actually have that in another cluster but they didnt experience any issue 😛
This is how it looks on a bad node
vs
So depending on which image you started with you get different selinux configs.
We dont have any docker config (node is running k8s).