moby: cgroups: memory cgroup not supported on this system

Description

Hello, all Docker builds failing with message:

cgroups: memory cgroup not supported on this system

Log:

Running with gitlab-ci-multi-runner 9.5.0 (413da38)
  on e2fe1677997a (efc92a0b)
Using Docker executor with image docker:17.09
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:2b5312e84355e6e50dd36c61663df237e8f0431f5af003710a7b3d10cd73b6bc for docker service...
Waiting for services to be up and running...
Using docker image sha256:dc3f32273c6341f7fad7801c26fe9dad59f46755ff39b37eb3e7e9f5bb7301ea for predefined container...
Pulling docker image docker:17.09
Using docker image docker:17.09
ID=sha256:f5be151966b1589ddabd30599c8e20bd9188288077d592bbf38c9618bf33ff08 for build container...
Running on runner-efc92a0b-project-379-concurrent-0 via e2fe1677997a...

Fetching changes...
HEAD is now at 647972e Redacted
Checking out 647972e8 as master...
Skipping Git submodules setup
<redacted login>
Building image...$ docker build --pull --build-arg redacted/redacted:5030-647972e8b928224734451335b1c7df1511464040-master -f /builds/redacted/redacted/ci/../Dockerfile /builds/redacted/redacted/ci/../
Sending build context to Docker daemon  112.1MB
Step 1/8 : FROM ...
ee1a8002360e: Pulling fs layer
37e9fa1264a3: Pull complete
Digest: sha256:0cedc587e3ea3c52b649e39a51117db1cb04d06d0d5ab0a303b0aa558a11c728
Status: Downloaded newer image for ....:latest
 ---> 85cee6e6f424
Step 2/8 : ENV PROJECT_ROOT /www/
 ---> Running in aceac413a3f0
Removing intermediate container aceac413a3f0
 ---> 9683ddb81b27
Step 3/8 : WORKDIR $PROJECT_ROOT
Removing intermediate container 70dbeabe23a8
 ---> c232a3ab0f73
Step 4/8 : ADD ./dockerfs /
 ---> 1a55ef7ca2ba
Step 5/8 : ADD . ${PROJECT_ROOT}
 ---> 449308a91f70
Step 6/8 : RUN apt update ;      apt install -y         php7.1-bcmath     && apt clean ;    ;
 ---> Running in 3e18c573ff4d
cgroups: memory cgroup not supported on this system: unknown
1 docker build -t registry.redacted/redacted:5030-647972e8b928224734451335b1c7df1511464040-master -f 
Exited with code: 1

Log from Docker executor container:

time="2017-11-23T11:15:43.317887174Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2017-11-23T11:15:43.318082694Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2017-11-23T11:15:43.330084778Z" level=info msg="libcontainerd: started new docker-containerd process" pid=23
time="2017-11-23T11:15:43Z" level=info msg="starting containerd" module=containerd revision=992280e8e265f491f7a624ab82f3e238be086e49 version=v1.0.0-beta.2-53-g992280e8 
time="2017-11-23T11:15:43Z" level=info msg="changing OOM score to -500" module=containerd 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1 
time="2017-11-23T11:15:43Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1 
time="2017-11-23T11:15:43Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt" 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1 
time="2017-11-23T11:15:43Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug" 
time="2017-11-23T11:15:43Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc" 
time="2017-11-23T11:15:43Z" level=info msg="containerd successfully booted in 0.016153s" module=containerd 
time="2017-11-23T11:15:43.424522373Z" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."
time="2017-11-23T11:15:43.435014417Z" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."
time="2017-11-23T11:15:43.435087818Z" level=error msg="Failed to built-in GetDriver graph devicemapper /var/lib/docker"
time="2017-11-23T11:15:43.479063163Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2017-11-23T11:15:43.479282794Z" level=warning msg="Your kernel does not support cgroup memory limit"
time="2017-11-23T11:15:43.479328424Z" level=warning msg="Your kernel does not support cgroup rt period"
time="2017-11-23T11:15:43.479340099Z" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2017-11-23T11:15:43.479346301Z" level=warning msg="Unable to find blkio cgroup in mounts"
time="2017-11-23T11:15:43.479416144Z" level=warning msg="mountpoint for pids not found"
time="2017-11-23T11:15:43.480481046Z" level=info msg="Loading containers: start."
time="2017-11-23T11:15:43.494914625Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge                 81223  1 br_netfilter\nstp                     1693  1 bridge\nllc                     3377  2 bridge,stp\nipv6                  276616 279 bridge,[permanent]\nip: can't find device 'br_netfilter'\nbr_netfilter           11126  0 \nbridge                 81223  1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
time="2017-11-23T11:15:43.499519911Z" level=warning msg="Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'\nmodprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
time="2017-11-23T11:15:43.503488640Z" level=warning msg="Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'\nmodprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
time="2017-11-23T11:15:43.574772308Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2017-11-23T11:15:43.612856536Z" level=info msg="Loading containers: done."
time="2017-11-23T11:15:43.631723422Z" level=info msg="Docker daemon" commit=1caf76c graphdriver(s)=vfs version=17.11.0-ce
time="2017-11-23T11:15:43.632107586Z" level=info msg="Daemon has completed initialization"
time="2017-11-23T11:15:43.659701897Z" level=info msg="API listen on [::]:2375"
time="2017-11-23T11:15:43.659711341Z" level=info msg="API listen on /var/run/docker.sock"
time="2017-11-23T11:16:40.663628582Z" level=info msg="Layer sha256:394fcc204628ea57d1ea250bfcca2f1cfab16213c37930af03b866c1be32b3c3 cleaned up"
time="2017-11-23T11:17:27Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3e18c573ff4df0bca3e4f761b861c8bbaf46b3b047f4713049e6a5a13cd93310/shim.sock" debug=false module="containerd/tasks" pid=606 
time="2017-11-23T11:17:28.159477274Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2017-11-23T11:17:28Z" level=info msg="shim reaped" id=3e18c573ff4df0bca3e4f761b861c8bbaf46b3b047f4713049e6a5a13cd93310 module="containerd/tasks" 
time="2017-11-23T11:17:28Z" level=error msg="failed to kill shim" error="cgroups: memory cgroup not supported on this system" module="containerd/tasks" 
time="2017-11-23T11:17:28.333574135Z" level=error msg="3e18c573ff4df0bca3e4f761b861c8bbaf46b3b047f4713049e6a5a13cd93310 cleanup: failed to delete container from containerd: no such container"

Steps to reproduce the issue:

docker build.

Describe the results you received:

cgroups: memory cgroup not supported on this system

Image not builded.

Describe the results you expected:

Image builded successfully.

Additional information you deem important (e.g. issue happens only occasionally):

Tested on lot of projects where previous builds was successfull.

Output of docker version:

Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Tue Aug 22 17:04:27 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Tue Aug 22 19:03:58 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 27
 Running: 5
 Paused: 0
 Stopped: 22
Images: 71
Server Version: 17.06.1-ce
Storage Driver: devicemapper
 Pool Name: docker-8:4-3278552-pool
 Pool Blocksize: 65.54kB
 Base Device Size: 10.74GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 15GB
 Data Space Total: 107.4GB
 Data Space Available: 76.43GB
 Metadata Space Used: 22.13MB
 Metadata Space Total: 2.147GB
 Metadata Space Available: 2.125GB
 Thin Pool Minimum Free Space: 10.74GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93 (2015-01-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: 3wsk1z1i9cey1mxxhu63lhi2w
 Is Manager: true
 ClusterID: x7pkp9yojzrhwcn574cwgfv8k
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.234.4.29
 Manager Addresses:
  10.234.4.29:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190 (expected: 810190ceaa507aa2727d7ae6f4790c76ec150bd2)
init version: v0.15.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.6-gentoo-r1
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.69GiB
Name: docker-hosting-01
ID: FOOQ:GPIA:7BPD:WURM:GFUL:52PG:Q7AW:KOTA:2NIN:BCSB:P54J:O43K
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support

Server is used only for builds, all memory and CPU resources are unused.

Thank you

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 14
Comments: 26 (4 by maintainers)

Most upvoted comments

I had this problem with Raspbian Stretch / Debian 9.1 / Kernel 4.9.65+ / Docker 17.11.0-ce and was able to fix it by adding “cgroup_memory=1” in /boot/cmdline.txt & reboot. Looks like this now: dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=xxxxxxxx-xx rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=memory cgroup_memory=1 swapaccount=1

So, no need to downgrade docker ; )

More info: https://archlinuxarm.org/forum/viewtopic.php?f=15&t=12086

+56

mephune on Nov 24, 2017

@acidDrain Thanks, this solved my issue on Raspbian Stretch:

sudo apt-get install -y docker-ce=17.09.0~ce-0~raspbian --allow-downgrades

klein0r on Nov 24, 2017

FYI to anyone else that randomly comes across this - I’m running docker on Raspbian and had this same issue. Downgrading from docker-ce=17.11.0 to docker-ce=17.09 fixed my issue apt-get install -y docker-ce=17.09.0~ce-0~raspbian

acidDrain on Nov 25, 2017

Using the instructions provided by @mephune in this comment above on this same thread, here are a set of commands to copy and paste.

It checks if the settings cgroup_enable=memory and cgroup_memory=1 are already set and if not it adds them.

# Go to your home directory
cd $HOME
# Copy the file to edit it in your home directory
sudo cp /boot/cmdline.txt ./cmdline.txt
# Create a backup, just in case
cp ./cmdline.txt ./cmdline.txt.backup

# If you feel risky, you could replace the lines above uncommenting the ones below

# sudo su
# cd /boot

# This long single line has everything. Check for both settings and set them only if they are not set
if [ $(grep -c cgroup_enable=memory cmdline.txt) == 0 ] ; then echo "$(cat cmdline.txt) cgroup_enable=memory" > cmdline.txt ; fi && if [ $(grep -c cgroup_memory=1 cmdline.txt) == 0 ] ; then echo "$(cat cmdline.txt) cgroup_memory=1" > cmdline.txt ; fi

# If you ran it in your home directory, now you need to copy the modified file to the boot directory
sudo cp ./cmdline.txt /boot/cmdline.txt

It was tested in a Raspberry Pi 3 B running: OS: Raspbian GNU/Linux 9.3 (stretch) Docker: Docker version 17.11.0-ce, build 1caf76c

tiangolo on Dec 26, 2017

Just ran into this on Devuan Jessie w/ vanilla 3.16.0 kernel image using the official docker-ce edge image for Debian Jessie. Problem started after upgrade to 17.11.0~ce-0~debian. Even something as simple as

docker run hello-world

hangs. Downgrading to 17.10.0~ce-0~debian fixes the problem for me.

I’m not seeing this on another Devuan Jessie system w/ a 4.9.0 vanilla kernel image from jessie-backports using the same docker-ce edge 17.11.0~ce-0~debian package.

Waiting for a bug fix release 😉

paddy-hack on Nov 28, 2017

https://github.com/containerd/containerd/pull/1803 merged in containerd which should restore the old behaviour of just warning if the setting is not enabled.

dnephin on Nov 27, 2017

In my case it was problem with latest docker:dind image - docker:17.11.0-ce-dind. Switch to previous version docker:17.09 solved my problem.

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 4135203..e8bc8d0 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,7 +4,7 @@
 
 image: docker:17.09
 services:
-    - docker:dind
+    - docker:17.09-dind

0x6B386F on Nov 23, 2017

Downgrading to 17.10.0~ce-0~debian also worked for me. (running on a Xen DomU)

jelle-e on Nov 30, 2017

Future raspberrypi/linux (including Raspbian) kernel releases - anything dated today or later - will only require cgroup_enable=memory, not cgroup_memory=1. Keep both options for now, but cgroup_memory will be dropped in 4.14.

pelwell on Nov 29, 2017

Downgrading to 17.10.0~ce-0~debian also worked for me.

zenire on Nov 28, 2017

Yes, the problem was in containerd, not docker (basically; docker only printed a “warning”, but containerd 1.0 made this an “error”). This pull-request fixed it; https://github.com/containerd/containerd/pull/1803 and is part of Docker 17.12

Important even though this issue is fixed and Docker now successfully starts without memory cgroup being supported, it is still highly recommended to make sure your system does have support for memory cgroups. If memory cgroups are not enabled on your system, docker cannot restrict memory for containers, so containers can use unlimited memory, and easily cause the host to run out of memory if something went wrong inside a container, or a container is too greedy.

thaJeztah on Dec 28, 2017

Waiting for a bug fix release 😉

Looks like 17.12.0~ce fixed it for me 🎉 🎊

paddy-hack on Dec 28, 2017

ping @crosbymichael @stevvooe @mlaventure

cpuguy83 on Nov 23, 2017

I upgraded to 17.11.0-ce because I thought it would fix for #35310, now I get this bug. None of my services are starting and the ones I’ve checked are having the cgroups error. Thankfully it is just a hobby setup on raspberry pi’s.

RobertMurry on Nov 23, 2017

@1dal, thx for the tip 🥇 Today all our build servers were red.

nerro on Nov 23, 2017