circleci-cli: [circleci local execute] error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Meta:

CircleCI CLI Version: 0.1.15195+595cc9e (release)

Operating System: Arch Linux (latest)

CircleCI CLI Diagnostic:

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /home/ldorau/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Lukasz Dorau.

Current behavior:

$ circleci local execute
Docker image digest: sha256:fc6b14595a231dd8417c97fad26f5f5f294f64b6f58443d93dfba17186f9f738
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.9.14-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Expected behavior:

“circleci local execute” works

When did this begin?

Now

Was this previously working?:

No, never.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 28
Comments: 68 (10 by maintainers)

Most upvoted comments

Hey folks, I’m a product manager here at CircleCI. Thank you for making us aware of the problem and apologies for this loss of functionality. Our engineering team is investigating a fix. I unfortunately do not have an estimate at this time on when we’ll get this fixed but as soon as I have more information I’ll update this thread.

We’ll be updating the local CLI docs page to call out this limitation in the time being.

Thanks all for your patience.

+25

sebastian-lerner on Dec 22, 2021

We believe we have a fix for this issue in the CircleCI build-agent (Docker image circleci/picard) now.

To accelerate your rollout of the agent, you can run: rm ~/.circleci/build_agent_settings.json to get the latest version.

+23

pete-woods on Jan 5, 2022

Looks like this error from this library.

These docs say that with a recent version of the library and with a recent Linux kernel, the kernel parameter systemd.unified_cgroup_hierarchy=1 will cause cgroups v2 to be used by the library. However, I have this parameter on my system (as part of GRUB_CMDLINE_LINUX and GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub), but I still get the error mentioned in this issue.

It looks like circleci-cli needs to update its version of runc [edit: I did some digging around, and it seems more likely that the non-public circleci/picard Docker image is what needs to update runc.]. See, for example, a similar error message in another issue:

docker: Error response from daemon: OCI runtime create failed: this version of runc doesn’t work on cgroups v2: unknown.

Edit: Update

I pulled circleci/picard and poked around inside:


$ docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard

/ #

/ # cd /opt/circleci

/opt/circleci # grep 'runc@' ./linux/arm64/circleci-agent
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/user/lookup.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.0-rc92/libcontainer/cgroups/v1_utils.go

It looks like the latest circleci/picard is using runc@v1.0.0-rc92. In that release, cgroupv2 support was still considered experimental per the previous release (rc91), and wasn’t considered “fully ready for production” until the following release (rc93).

+17

dgoffredo on Jul 9, 2021

I’m experiencing this same issue on OSX Big Sur. Happens whether I use brew install circlecli or use the provided curl command for Mac/Linux

$ circleci diagnostic

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /Users/greg.wilburn/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Greg Wilburn.

$ circleci local execute

Docker image digest: sha256:130f2f80984be6c2336ae57c48cd2b666061a82660740e027ac63e96eed8c0ad
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

+16

snowinferno on Dec 7, 2021

We believe we have rolled out a fix for this recently reported issue.

To accelerate your rollout of the agent, you can run: rm ~/.circleci/build_agent_settings.json to get the latest version.

Please let us know if this still does not fix the issue for you.

+13

sebastian-lerner on Feb 1, 2022

Okay. In my case, I changed LastestSha256 to sha256:6c640d1c0aed49afd1c6a426bfb8f5f40fbb7de439a154b73d53184977b9c1f6 and It works. there seems to be a problem with the latest circleci/picardimage

+10

deokseong-kim-toss on Mar 18, 2022

I’ve tried removing the build agent file already but am still getting the following error:

Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64

Starting container hashicorp/terraform:1.0.0
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as hashicorp/terraform:1.0.0, but refreshing...
hashicorp/terraform:1.0.0:
  using image hashicorp/terraform@sha256:6dafbe34b7c18d189ff18e3dfb7cd5fc4144a2714eb2bc7b6dd9352455e8848a
  pull stats: Image was already available so the image was not pulled
  time to create container: 41ms
Creating Docker containers in parallel
1.0.0: Pulling from hashicorp/terraform
Digest: sha256:6dafbe34b7c18d189ff18e3dfb7cd5fc4144a2714eb2bc7b6dd9352455e8848a
Status: Image is up to date for hashicorp/terraform:1.0.0
Time to upload agent and config: 4.498269919s
Error:
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/b73bc306f5f0b63dbbf7e3ca471cb09d1693082ed24b022c18044c13949b87c5/abe49607cf6cebfc32f940ef1e58d1244b0c4cdc7aecb0a2f62cccfb9b394b56" with domain controllers -- it is in an invalid state: unknown

+10

jmstone617 on Mar 4, 2022

cc: @pete-woods

I’ve been following the steps about for MacOS. It is not working for me, though.

Given the .circleci/config.yml

orbs: # declare what orbs we are going to use
  node: circleci/node@2.0.2 # the node orb provides common node-related configuration

version: 2.1 # using 2.1 provides access to orbs and other features

workflows:
  matrix-tests:
    jobs:
      - node/test:
          version: 16.13.2

Then running the circleci CLI.

$ circleci local execute --job "node/test"
Docker image digest: sha256:156a80a311fbad52d73d2391f5951604f041c687d8b4e0f87624255af8f2ef72
====>> Spin up environment
Build-agent version  ()
process panic
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/circleci/build-agent/cmd/internal/setup.PanicHandler()
        /mnt/ramdisk/circleci/project/cmd/internal/setup/handlers.go:23 +0x85
panic({0x145e780, 0x24a3dd0})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/opencontainers/runc/libcontainer/cgroups/fs2.defaultDirPath(0xc000a13840)
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/defaultpath.go:35 +0x18
github.com/opencontainers/runc/libcontainer/cgroups/fs2.NewManager(0x0, {0x0, 0x0})
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/fs2.go:31 +0x2c
github.com/circleci/build-agent/outer/docker/cgroups.cgroupsV2({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:51 +0x6e
github.com/circleci/build-agent/outer/docker/cgroups.GetSelf({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:25 +0xa5
github.com/circleci/build-agent/outer/docker.NewDockerExecutorWithClient({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:70 +0x13e
github.com/circleci/build-agent/outer/docker.NewDockerExecutor({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:103 +0xdc
github.com/circleci/build-agent/outer.newExecutor({_, _}, {{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/run.go:372 +0x265
github.com/circleci/build-agent/outer.ConfigureAndRun({_, _}, {{0xc000726438, 0x15}, {0x0, 0x0}, {0xc0007265a0, 0x16}, {0x0, 0x0}, ...})
        /mnt/ramdisk/circleci/project/outer/run.go:219 +0x17ad
github.com/circleci/build-agent/cmd/internal/outer/local.runExecuteJob(0xc0000a0380, 0x0, {0x19862b0, 0x0})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:125 +0x2b6
github.com/circleci/build-agent/cmd/internal/outer/local.NewExecuteJobCmd.func1(0xc000349400, {0x165b78d, 0x4, 0x4})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:64 +0x29
github.com/spf13/cobra.(*Command).execute(0xc000349400, {0xc000731140, 0x4, 0x4})
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000349180)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/circleci/build-agent/cmd.Execute()
        /mnt/ramdisk/circleci/project/cmd/root.go:64 +0x2cf
main.main()
        /mnt/ramdisk/circleci/project/main.go:14 +0xca

I can create a new issue if need be, but it seems related because of the cgroups in the stacktrace.

CLI Version: 0.1.16535+5010eb7 (homebrew) Docker Desktop Mac: Docker Desktop 4.4.2 (73305)

jtarchie on Jan 25, 2022

Hitting this today on MacOS Monterey.

$ circleci local execute --job build
Docker image digest: sha256:e025cba3f5236a7e3ed9b97dfadce44c824ea0cc1ae1427823ff4c1dd2f420c3
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

This is using the latest circleci/picard - released ~an hour ago - https://hub.docker.com/layers/circleci/picard/1.0.95690-99ec9ada/images/sha256-e025cba3f5236a7e3ed9b97dfadce44c824ea0cc1ae1427823ff4c1dd2f420c3?context=explore

We can also see that runc is up to 1.0.3:

docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard
/ # cd /opt/circleci
/opt/circleci # grep 'runc@' ./linux/arm64/circleci-agent
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/devices/device_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/userns/userns_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/user/lookup_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/v1_utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/file.go

This is bothersome, since I can no longer test out circleci changes and have to use remote resources to debug.

miketheman on Dec 10, 2021

Hey folks, (I’m from CircleCI btw). The workaround that we described before, still continue to use that. We have raised the issue with Docker but they will take awhile to resolve the cgroupsv2 issue. We will update folks here on the thread once they have resolved the issue and thank you for your patience.

mzarismithnyc on May 6, 2022

@Kayden-lolasery see if the workaround in https://github.com/CircleCI-Public/circleci-cli/issues/676 works for you

matschaffer on Mar 21, 2022

cc: @pete-woods

I’ve been following the steps about for MacOS. It is not working for me, though.

Given the .circleci/config.yml

orbs: # declare what orbs we are going to use
  node: circleci/node@2.0.2 # the node orb provides common node-related configuration

version: 2.1 # using 2.1 provides access to orbs and other features

workflows:
  matrix-tests:
    jobs:
      - node/test:
          version: 16.13.2

Then running the circleci CLI.

$ circleci local execute --job "node/test"
Docker image digest: sha256:156a80a311fbad52d73d2391f5951604f041c687d8b4e0f87624255af8f2ef72
====>> Spin up environment
Build-agent version  ()
process panic
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/circleci/build-agent/cmd/internal/setup.PanicHandler()
        /mnt/ramdisk/circleci/project/cmd/internal/setup/handlers.go:23 +0x85
panic({0x145e780, 0x24a3dd0})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/opencontainers/runc/libcontainer/cgroups/fs2.defaultDirPath(0xc000a13840)
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/defaultpath.go:35 +0x18
github.com/opencontainers/runc/libcontainer/cgroups/fs2.NewManager(0x0, {0x0, 0x0})
        /home/circleci/go/pkg/mod/github.com/opencontainers/runc@v1.1.0/libcontainer/cgroups/fs2/fs2.go:31 +0x2c
github.com/circleci/build-agent/outer/docker/cgroups.cgroupsV2({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:51 +0x6e
github.com/circleci/build-agent/outer/docker/cgroups.GetSelf({0xc00063cb00, 0x8})
        /mnt/ramdisk/circleci/project/outer/docker/cgroups/cgroups_linux.go:25 +0xa5
github.com/circleci/build-agent/outer/docker.NewDockerExecutorWithClient({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:70 +0x13e
github.com/circleci/build-agent/outer/docker.NewDockerExecutor({{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/docker/docker.go:103 +0xdc
github.com/circleci/build-agent/outer.newExecutor({_, _}, {{{{0xc0000403a8, 0x15}, {{...}, {...}, 0x0, 0x1, {...}, {...}, ...}, ...}, ...}, ...}, ...)
        /mnt/ramdisk/circleci/project/outer/run.go:372 +0x265
github.com/circleci/build-agent/outer.ConfigureAndRun({_, _}, {{0xc000726438, 0x15}, {0x0, 0x0}, {0xc0007265a0, 0x16}, {0x0, 0x0}, ...})
        /mnt/ramdisk/circleci/project/outer/run.go:219 +0x17ad
github.com/circleci/build-agent/cmd/internal/outer/local.runExecuteJob(0xc0000a0380, 0x0, {0x19862b0, 0x0})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:125 +0x2b6
github.com/circleci/build-agent/cmd/internal/outer/local.NewExecuteJobCmd.func1(0xc000349400, {0x165b78d, 0x4, 0x4})
        /mnt/ramdisk/circleci/project/cmd/internal/outer/local/execute.go:64 +0x29
github.com/spf13/cobra.(*Command).execute(0xc000349400, {0xc000731140, 0x4, 0x4})
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000349180)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/circleci/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/circleci/build-agent/cmd.Execute()
        /mnt/ramdisk/circleci/project/cmd/root.go:64 +0x2cf
main.main()
        /mnt/ramdisk/circleci/project/main.go:14 +0xca

I can create a new issue if need be, but it seems related because of the cgroups in the stacktrace.

CLI Version: 0.1.16535+5010eb7 (homebrew) Docker Desktop Mac: Docker Desktop 4.4.2 (73305)

Same issue for me.

CLI Version: 0.1.16535+5010eb7 (homebrew)

s4bs on Jan 26, 2022

Can confirm, resolved for me. Thank you @sebastian-lerner.

To make sure everyone is one the same page of instructions:

$ rm ~/.circleci/build_agent_settings.json # this is important
$ circleci local execute --job "node/test"

Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/node:16.13.2, but refreshing...
cimg/node:16.13.2:
  using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
  pull stats: Image was already available so the image was not pulled
  time to create container: 41ms
Creating Docker containers in parallel 
16.13.2: Pulling from cimg/node
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Image is up to date for cimg/node:16.13.2
Time to upload agent and config: 2.026862226s
Time to start containers: 371.585657ms

jtarchie on Feb 1, 2022

@pete-woods Joy! Local CircleCI builds are happy again on MacOS (after rm ~/.circleci/build_agent_settings.json). Thanks for this fix!

bhamail on Jan 11, 2022

FWIW, I think systemd.unified_cgroup_hierarchy needs to be set to 0 (zero) to enable cgroup v1 behaviour. At least it worked for me when setting it to 1 did not.

DavidS-cloud on Aug 11, 2021

Thanks a lot! It’s working now on my Mac with https://github.com/CircleCI-Public/circleci-cli/issues/589#issuecomment-1005865018.

kienstra on Jan 5, 2022

Exactly same error here. Running macOS Monterrey 12.2.1, installed CLI manually (v0.1.16947).

rm ~/.circleci/build_agent_settings.json
circleci local execute --job "node/test"

Trace:

Downloading latest CircleCI build agent...
Docker image digest: sha256:ea206e01751a9d4acc6dc21fb8184bdf67b0c2a524c156e526bdb991b1bd2e23
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/node:16.13.2
16.13.2: Pulling from cimg/node
f3ef4ff62e0d: Pull complete
2c322a662952: Pull complete
fb8baef6d8e7: Pull complete
805b7ba0a256: Pull complete
63833a96b952: Pull complete
5d1ffe5f9ba5: Pull complete
6dbfea31d6de: Pull complete
44f09c08a855: Pull complete
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Downloaded newer image for cimg/node:16.13.2
cimg/node:16.13.2:
  using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
  pull stats: download 358.9MiB in 44.413s (8.08MiB/s), extract 358.7MiB in 44.774s (8.012MiB/s)
  time to create container: 275ms
Creating Docker containers in parallel
Time to upload agent and config: 7.0508151s
Error:
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/f4e94be5ff47d02ebd9b580db37a9af1b9f222c692da90e0c266d6c03bdc3f6e/80d3b87d51ab7db04d27d38021b19568f05769e045c23541dc2fea62fa92838c" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/f4e94be5ff47d02ebd9b580db37a9af1b9f222c692da90e0c266d6c03bdc3f6e/80d3b87d51ab7db04d27d38021b19568f05769e045c23541dc2fea62fa92838c" with domain controllers -- it is in an invalid state: unknown

ucals on Mar 16, 2022

+1 on issue: Running Catalina 10.15.7 and seeing the same error as @jmstone617 . I thought at first it was per my specific config.yml (I was still testing with my original file), but I duplicated the simple config file from @jtarchie, ran the identical commands to remove the build settings file and execute the job locally and still seeing the following error:

rm ~/.circleci/build_agent_settings.json circleci local execute --job "node/test"

Downloading latest CircleCI build agent...
Docker image digest: sha256:6b954f968479e94b5a5a687b467540d9df7359c7e03230052596f0c41c07b3d8
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/node:16.13.2
16.13.2: Pulling from cimg/node
f3ef4ff62e0d: Pull complete 
2c322a662952: Pull complete 
fb8baef6d8e7: Pull complete 
805b7ba0a256: Pull complete 
63833a96b952: Pull complete 
5d1ffe5f9ba5: Pull complete 
6dbfea31d6de: Pull complete 
44f09c08a855: Pull complete 
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Downloaded newer image for cimg/node:16.13.2
cimg/node:16.13.2:
  using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
  pull stats: download 343.3MiB in 43.33s (7.923MiB/s), extract 358.7MiB in 24.413s (14.69MiB/s)
  time to create container: 273ms
Creating Docker containers in parallel 
Time to upload agent and config: 2.0054758s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/3026c5aac7c67fe6af86320edf46e0730deccb4f3e88c2631e3240b976f7b706/e264c13293b89594dc31b307a5b8cc5e183efb2704ce2ac28184e3852a2c7ef0" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/3026c5aac7c67fe6af86320edf46e0730deccb4f3e88c2631e3240b976f7b706/e264c13293b89594dc31b307a5b8cc5e183efb2704ce2ac28184e3852a2c7ef0" with domain controllers -- it is in an invalid state: unknown

steven-dicristofaro on Mar 11, 2022

Can also confirm the workaround from @deokseong-kim-toss re: https://github.com/CircleCI-Public/circleci-cli/issues/676 solves the issue for me as well. Thank you for sharing that!

steven-dicristofaro on Apr 5, 2022

After deleting the build agent file and re-running, I’m now getting a different error on Manjaro:

Error:   error starting container circleci/python:3.7: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"

Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/python:3.7, but refreshing...
3.7: Pulling from circleci/python
Digest: sha256:3de62879e0b899d3de2a836957c736e6711c5a3faabd0ea2ce28bb49cf5e4ae2
Status: Image is up to date for circleci/python:3.7
Error: 
Unexpected environment preparation error: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"
Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"```

mike-hearn on Jan 5, 2022

Running into the same issue with the latest CircleCI CLI (v0.1.16535) after upgrading the local docker desktop to the latest version 4.3.1 (72247) on Mac.

====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed

vingov on Dec 20, 2021

We’re still waiting to see if there is a workaround but at the moment, we’re still investigating with folx.

mzarismithnyc on Sep 12, 2022

FWIW, the issue still persists, even if ~/.circleci/build_agent_settings.json is no longer present in macOS:

cd ~/.circleci/
USER:.circleci USER$ ls
cli.yml                 update_check.yml

Would be nice to see if there’s any workaround for this 🙏

mkotsollaris on Sep 8, 2022

Hey folks, a quick update on this. We believe we’ve identified the root cause with the latest issue that has been reported and we think we’ve attributed it to a bug with Docker Desktop for Mac which we’ve filed here: https://github.com/docker/for-mac/issues/6288

We’re in the process of working with Docker to try and get this fixed. I’ll update when I have more

cc @Scipi @pete-woods

sebastian-lerner on Apr 19, 2022

@matschaffer Thank you! I have been able to get that to work.

mrickard on Apr 7, 2022

@sebastian-lerner I’m having the same problem.

MacOS: Monterey (12.3.1) circleci: 0.1.16947+263902a (homebrew) Docker Desktop: 4.6.1 (76265).

The problem occurs with cimg/14.19.0, as well as cimg/14.19.1 and cimg/14.18.2.

I’ve removed the build agent file, though the error I’m getting is consistently this:

Error: Unhandled prepare executor error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/cab3a2678cf64754071971f9089730323c1f06468efe1002e50e5ec3115345e6/15512e227f7fde93a3b0b5b6300679978c5ad0382b34eadd79fa8131f8260393" with domain controllers -- it is in an invalid state: unknown

mrickard on Apr 6, 2022

I have a very similar error:

Steps to replicate:

rm ~/.circleci/build_agent_settings.json
circleci local execute

Docker image digest: sha256:ea206e01751a9d4acc6dc21fb8184bdf67b0c2a524c156e526bdb991b1bd2e23
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container cimg/python:3.8
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/python:3.8, but refreshing...
cimg/python:3.8:
  using image cimg/python@sha256:db17fcbb1469f1f70bbbd76867c6d371d4bf9596cae84ece3b62de8fe9c20fb2
  pull stats: Image was already available so the image was not pulled
  time to create container: 52ms
Starting container cimg/postgres:12.7
3.8: Pulling from cimg/python
Digest: sha256:db17fcbb1469f1f70bbbd76867c6d371d4bf9596cae84ece3b62de8fe9c20fb2
Status: Image is up to date for cimg/python:3.8
Creating Docker containers in parallel ....
cimg/postgres:12.7:
  using image cimg/postgres@sha256:a488ce128a9e16e98f21a47af8981b0ea797b59cccee907d034d855e63b1a164
  pull stats: Image was already available so the image was not pulled
  time to create container: 48ms
Time to upload agent and config: 1.708347017s
Error: 
Unexpected environment preparation error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/ad8c463b3491878d8cf8a44061b4f0ff3bf147998c4e2a980d30b89e2f57ad8b/dda3240f3a162cc38220cdfb64a84ce60f57a7478277cecdc500d7f5b981d314" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed

Kayden-lolasery on Mar 19, 2022

I was specifically asked for feedback by Pete. Apologies if it’s cluttering your inbox.

mike-hearn on Jan 28, 2022

👍 the comments. That’s what they are there for.

jtarchie on Jan 28, 2022

I’ve just released a new version of the CircleCI build-agent. Could anyone running on Linux with the systemd Cgroup driver please try updating and running again?

pete-woods on Jan 6, 2022

Any update on this?

circleci local execute --job test
Docker image digest: sha256:eb0c4688f843e6987c29f72c9fdf8c910d59b2a72c7e1559d46a3b1837038cdb
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64

Error: 
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy

circleci version
0.1.16535+5010eb7 (homebrew)

podplatnikm on Dec 20, 2021

I’m seeing this also when attempting to test bioconda builds on a MacBook locally.

mjsteinbaugh on Dec 14, 2021