concourse: iptables error: Could not fetch rule set generation id: Invalid argument in ubuntu jammy

Summary

After bumping concourse’s base image to ubuntu jammy, k8s-topgun jobs in both dev pipeline and release/6.7.x pipeline are red due to a failed test for dns-proxy set up under containerd runtime.

The error log in worker pod is:

{"timestamp":"2022-09-11T03:41:56.523043712Z","level":"error","source":"worker","message":"worker.garden-runner.logging-runner-exited","data":{"error":"Exit trace for group:\ncontainerd-garden-backend exited with error: setup restricted networks failed: create chain or flush if exists failed: running [/usr/sbin/iptables -t filter -N CONCOURSE-OPERATOR --wait]: exit status 4: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument\n\n\ncontainerd exited with nil\n","session":"8"}}

Additional context

Running a local docker (concourse-dev image in ubuntu jammy) with DNS proxy enabled is still working.
Running a local k8s cluster by minikube (Kubernetes v1.24.3 on Docker 20.10.17), it also works.
We have been using iptables v1.8.7 since 6.7.3 in ubuntu bionic.

So it might be a specific problem due to GKE cluster 1.24 with containerd based OS image.

We should make sure this bug doesn’t affect concourse bosh release that running on ubuntu jammy stemcell.

Triaging info

Concourse version:
Browser (if applicable):
Did this used to work?

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 15 (14 by maintainers)

Commits related to this issue

Revert "bump concourse chart" as this caused an issue with starting docker see https://github.com/concourse/concourse/issues/8558 This reverts commit 2d7ca7615c145872f84ad4cb531c49ada08f39e8. — committed to cloudfoundry/concourse-infra-for-fiwg by ramonskie 2 years ago
switch k8s worker to guardian runtime see https://github.com/concourse/concourse/issues/8558 Signed-off-by: Rui Yang <ruiya@vmware.com> — committed to concourse/infrastructure by deleted user 2 years ago
switch to iptables-legacy there are hosts with newer kernel that throwing error like in this issue https://github.com/concourse/concourse/issues/8558 Signed-off-by: Rui Yang <ruiya@vmware.com> — committed to concourse/concourse-docker by deleted user a year ago
switch to iptables-legacy there are hosts with newer kernel that throwing error like in this issue https://github.com/concourse/concourse/issues/8558 Signed-off-by: Rui Yang <ruiya@vmware.com> — committed to concourse/concourse-docker by deleted user a year ago

Most upvoted comments

@cortesl for M1 it is a different issue as Concourse binary doesn’t support M1 yet.

xtremerui on Feb 13, 2023

i just bumped our concourse from 7.8.1 to 7.8.3 and experience this error as well

when fetching a resource i get the following error msg

waiting for docker to come up...
waiting for docker to come up...
time="2022-10-24T09:01:07.368503690Z" level=info msg="Starting up"
time="2022-10-24T09:01:07.369917050Z" level=info msg="libcontainerd: started new containerd process" pid=3624
time="2022-10-24T09:01:07.369953442Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-10-24T09:01:07.369961796Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-10-24T09:01:07.369982011Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-10-24T09:01:07.369994748Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-10-24T09:01:07Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
time="2022-10-24T09:01:07.400505464Z" level=info msg="starting containerd" revision=9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6 version=1.6.8
time="2022-10-24T09:01:07.417742142Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2022-10-24T09:01:07.417794820Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.417912814Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.417963304Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418151822Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /scratch/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418170994Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418184344Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-10-24T09:01:07.418194211Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418219057Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418319137Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418486010Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /scratch/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-10-24T09:01:07.418511290Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2022-10-24T09:01:07.418530393Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-10-24T09:01:07.418540153Z" level=info msg="metadata content store policy set" policy=shared
time="2022-10-24T09:01:07.418630302Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2022-10-24T09:01:07.418652936Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2022-10-24T09:01:07.418678144Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2022-10-24T09:01:07.418718984Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418745857Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418760152Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418774184Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418799561Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418820955Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418834399Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418846988Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.418859694Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2022-10-24T09:01:07.418927544Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2022-10-24T09:01:07.418972910Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2022-10-24T09:01:07.419994711Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2022-10-24T09:01:07.420066822Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420096579Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2022-10-24T09:01:07.420170704Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420197473Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420223047Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420250472Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420276530Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420301470Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420324480Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420344518Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420374253Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2022-10-24T09:01:07.420440008Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420515491Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420640747Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2022-10-24T09:01:07.420839908Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2022-10-24T09:01:07.420869130Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2022-10-24T09:01:07.420889990Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-10-24T09:01:07.420921239Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2022-10-24T09:01:07.421270017Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2022-10-24T09:01:07.421385431Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2022-10-24T09:01:07.421500870Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2022-10-24T09:01:07.421537770Z" level=info msg="containerd successfully booted in 0.021973s"
time="2022-10-24T09:01:07.426587157Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-10-24T09:01:07.426611738Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-10-24T09:01:07.426630805Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-10-24T09:01:07.426641113Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-10-24T09:01:07.428407767Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-10-24T09:01:07.428434700Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-10-24T09:01:07.428522534Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-10-24T09:01:07.428540830Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-10-24T09:01:07.444757387Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
time="2022-10-24T09:01:07.479638452Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2022-10-24T09:01:07.479663504Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2022-10-24T09:01:07.479841152Z" level=info msg="Loading containers: start."
time="2022-10-24T09:01:07.482411995Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument`, error: exit status 4"
time="2022-10-24T09:01:07.527194341Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2022-10-24T09:01:07.527691475Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2022-10-24T09:01:07.527690456Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2022-10-24T09:01:08.528683230Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument

 (exit status 4)
Docker failed to start within 120 seconds.
/opt/resource/common.sh: line 113: kill: (3616) - No such process

seems to me that the worker host as has a newer iptables cli and causes this problem. when searching for this issue it seems that iptables-legacy is suggested for now

ramonskie on Oct 24, 2022