zarf: Zarf-Seed-Registry Installation Fails on Init with Deployment is not ready: zarf/zarf-docker-registry error
Environment
Device and OS: Azure AKS Linux Ubuntu 20.04 App version: 0.19.6 Kubernetes distro being used: AKS Kubernetes V 1.22.6 Other:
Steps to reproduce
- Create an AKS Cluster
- Run
zarf init --components git-server.
Expected result
Command succeeds and Zarf is initialized in the cluster.
Actual Result
The following message repeats until the init run timesout
Deployment is not ready: zarf/zarf-docker-registry. 0 out of 1 expected pods are ready
output of kubectl -n zarf get events
LAST SEEN TYPE REASON OBJECT MESSAGE
15m Normal Scheduled pod/injector Successfully assigned zarf/injector to aks-agentpool-40722291-vmss000001
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-018" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-023" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-013" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-008" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-009" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-019" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "stage1" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-027" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector MountVolume.SetUp failed for volume "zarf-payload-007" : failed to sync configmap cache: timed out waiting for the condition
15m Warning FailedMount pod/injector (combined from similar events): MountVolume.SetUp failed for volume "zarf-payload-027" : failed to sync configmap cache: timed out waiting for the condition
15m Normal Scheduled pod/zarf-docker-registry-789d8ddfb8-4pfgj Successfully assigned zarf/zarf-docker-registry-789d8ddfb8-4pfgj to aks-agentpool-40722291-vmss000000
14m Normal Pulling pod/zarf-docker-registry-789d8ddfb8-4pfgj Pulling image "127.0.0.1:32178/library/registry:2.7.1"
14m Warning Failed pod/zarf-docker-registry-789d8ddfb8-4pfgj Failed to pull image "127.0.0.1:32178/library/registry:2.7.1": rpc error: code = Unknown desc = failed to pull and unpack image "127.0.0.1:32178/library/registry:2.7.1": failed to resolve reference "127.0.0.1:32178/library/registry:2.7.1": failed to do request: Head "https://127.0.0.1:32178/v2/library/registry/manifests/2.7.1": http: server gave HTTP response to HTTPS client
14m Warning Failed pod/zarf-docker-registry-789d8ddfb8-4pfgj Error: ErrImagePull
34s Normal BackOff pod/zarf-docker-registry-789d8ddfb8-4pfgj Back-off pulling image "127.0.0.1:32178/library/registry:2.7.1"
13m Warning Failed pod/zarf-docker-registry-789d8ddfb8-4pfgj Error: ImagePullBackOff
15m Normal SuccessfulCreate replicaset/zarf-docker-registry-789d8ddfb8 Created pod: zarf-docker-registry-789d8ddfb8-4pfgj
15m Normal ScalingReplicaSet deployment/zarf-docker-registry Scaled up replica set zarf-docker-registry-789d8ddfb8 to 1
output of kubectl -n zarf get all
NAME READY STATUS RESTARTS AGE
pod/injector 1/1 Running 0 40m
pod/zarf-docker-registry-789d8ddfb8-4pfgj 0/1 ImagePullBackOff 0 40m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/zarf-docker-registry NodePort 10.0.16.59 <none> 5000:31999/TCP 40m
service/zarf-injector NodePort 10.0.144.122 <none> 5000:32178/TCP 40m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/zarf-docker-registry 0/1 1 0 40m
NAME DESIRED CURRENT READY AGE
replicaset.apps/zarf-docker-registry-789d8ddfb8 1 1 0 40m
Severity/Priority
😕 Blocked on deploying zarf packages to Azure AKS
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 4
- Comments: 33 (24 by maintainers)
Commits related to this issue
- feat: update injector away from rouille to axum (#2457) Fixes #592 --------- Signed-off-by: schristoff <28318173+schristoff@users.noreply.github.com> — committed to zarf-dev/zarf by schristoff 3 years ago
- feat: update injector away from rouille to axum (#2457) Fixes #592 --------- Signed-off-by: schristoff <28318173+schristoff@users.noreply.github.com> Signed-off-by: Austin Abro <AustinAbro321@gmail... — committed to zarf-dev/zarf by schristoff 3 years ago
Hi guys. Any plans here? Still having a problem with k8s 1.27+ versions with containerd
1.7.11.Any recommendations from the community on how we can “tune” a containerd config to avoid this issue?
A new issue has been opened against containerd to address this: https://github.com/containerd/containerd/issues/10014
Also ran into this issue on newer RKE2 versions. It seems linked back to this commit which introduced the
config_pathline. The containerd update to 1.7.7+ actually did not immediately cause the issue (RKE2 1.29.0 works and is on containerd 1.7.11), it was specifically the introduction of that containerd config line 👀.Just for reference affected versions of k3s/rke2 appear to be 1.29.1+, 1.28.6+, and 1.27.10+. Definitely curious if there is anything to address this on the zarf side or if this should make its way into the docs as a recommended pre-req/setup for the cluster?
hi folks, as a workaround (that includes patch by iceberg you can update containerd in your cluster.
Just apt update and upgrade the node. I do it through a DaemonSet (example):
I was able to initialize zarf:
I did also deploy the Big Bang into AKS, but had a “bump”
control-plane: controller-managerin their NS… preventing the hook to change the image … after removing it, everything went smoothly. -> in rke2 didn’t get the problem …All things coming around, we may need to look to a way to avoid the localhost/http behavior since containerd has introduced bugs multiple times for this in the past year or so. https://github.com/containerd/containerd/pull/9188
I’ve run into this with containerd > 1.6.25. It seems the default behavior changed here https://github.com/containerd/containerd/pull/9300
I’ve been commenting out the following lines in the containerd config.toml to work around it:
quick update, containerd was updated and I could deploy bigbang 1.48.0.
Commenting out the containerd config lines mentioned in this post got things working again for me.
Upstream issue has been closed and @brianrexrode has successfully tested zarf v0.25.2 with EKS V1.26