circleci-cli: [circleci local execute] error looking up cgroup: not implemented for cgroup v2 unified hierarchy - 2022 edition

Meta:

CircleCI CLI Version: 0.1.16947+263902a (homebrew)

Operating System: macOS Monterey - 12.3 (21E230)

Model Name: MacBook Pro Model Identifier: MacBookPro16,1 Processor Name: 8-Core Intel Core i9 Processor Speed: 2.4 GHz Number of Processors: 1 Total Number of Cores: 8 L2 Cache (per Core): 256 KB L3 Cache: 16 MB Hyper-Threading Technology: Enabled Memory: 64 GB System Firmware Version: 1731.100.130.0.0 (iBridge: 19.16.14242.0.0,0)

CircleCI CLI Diagnostic:

Debugger mode: false
Config found: /Users/matschaffer/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Mat Schaffer.

---

Current behavior:

When running from https://github.com/LLK/scratch-gui

❯ circleci local execute
Docker image digest: sha256:ea206e01751a9d4acc6dc21fb8184bdf67b0c2a524c156e526bdb991b1bd2e23
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container circleci/node:10-browsers
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/node:10-browsers, but refreshing...
10-browsers: Pulling from circleci/node
Digest: sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
Status: Image is up to date for circleci/node:10-browsers
circleci/node:10-browsers:
  using image circleci/node@sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
  pull stats: Image was already available so the image was not pulled
  time to create container: 62ms
Creating Docker containers in parallel
Time to upload agent and config: 1.6537866s
Error:
Unexpected environment preparation error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/e5e342f52c6bd355332860b6fd6dab41a3f1592032a87675618a92233cd804fc/0f75c7c4b5ddf81be5546a18347ee417666919a5315ac542f279232b3098a1c8" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/e5e342f52c6bd355332860b6fd6dab41a3f1592032a87675618a92233cd804fc/0f75c7c4b5ddf81be5546a18347ee417666919a5315ac542f279232b3098a1c8" with domain controllers -- it is in an invalid state: unknown

Expected behavior:

Build should start.

When did this begin / Was this previously working?:

Not previously working, but I only started using the circleci CLI this week.

Additional Information:

❯ docker version
Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:08:44 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.6.0 (75818)
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:06:05 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Workaround

I was able to get the build started by using the sha found in https://github.com/CircleCI-Public/circleci-cli/issues/589#issuecomment-1027097706

You’ll need to downgrade the CLI first since support for the file was removed in bf781e7d37263bceacc340e9553046d152605c24

• Uninstall with brew uninstall circleci
• Download the old Homebrew formula for v0.1.17087 and install from that file as a local recipe

  curl -LO https://raw.githubusercontent.com/Homebrew/homebrew-core/abe75ec1a5b53b31f8c88eff4b6f402008e1bf46/Formula/circleci.rb
  brew install circleci.rb
  

❯ cat > ~/.circleci/build_agent_settings.json <<JSON
{"LatestSha256":"sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014"}
JSON

❯ circleci local execute
Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container circleci/node:10-browsers
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/node:10-browsers, but refreshing...
10-browsers: Pulling from circleci/node
Digest: sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
Status: Image is up to date for circleci/node:10-browsers
circleci/node:10-browsers:
  using image circleci/node@sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
  pull stats: Image was already available so the image was not pulled
  time to create container: 46ms
Creating Docker containers in parallel
Time to upload agent and config: 1.6189462s
Time to start containers: 337.906ms
====>> Preparing environment variables
Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1647572688
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=
  CIRCLE_SHA1=
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1647572688
  CIRCLE_WORKING_DIRECTORY=~/repo


The redacted variables listed above will be masked in run step output.====>> Restoring cache
Error:
skipping cache - error checking storage: not supported

Step failed
====>> Checkout code
Making checkout directory "/home/circleci/repo"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/repo"

So it seems that the latest circleci/picard image has a regression that effects at least docker for mac.