zarf: Zarf-Seed-Registry Installation Fails on Init with Deployment is not ready: zarf/zarf-docker-registry error

Environment

Device and OS: Azure AKS Linux Ubuntu 20.04 App version: 0.19.6 Kubernetes distro being used: AKS Kubernetes V 1.22.6 Other:

Steps to reproduce

Create an AKS Cluster
Run zarf init --components git-server.

Expected result

Command succeeds and Zarf is initialized in the cluster.

Actual Result

The following message repeats until the init run timesout
Deployment is not ready: zarf/zarf-docker-registry. 0 out of 1 expected pods are ready

output of kubectl -n zarf get events

LAST SEEN   TYPE      REASON              OBJECT                                       MESSAGE
15m         Normal    Scheduled           pod/injector                                 Successfully assigned zarf/injector to aks-agentpool-40722291-vmss000001
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-018" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-023" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-013" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-008" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-009" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-019" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "stage1" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-027" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 MountVolume.SetUp failed for volume "zarf-payload-007" : failed to sync configmap cache: timed out waiting for the condition
15m         Warning   FailedMount         pod/injector                                 (combined from similar events): MountVolume.SetUp failed for volume "zarf-payload-027" : failed to sync configmap cache: timed out waiting for the condition
15m         Normal    Scheduled           pod/zarf-docker-registry-789d8ddfb8-4pfgj    Successfully assigned zarf/zarf-docker-registry-789d8ddfb8-4pfgj to aks-agentpool-40722291-vmss000000
14m         Normal    Pulling             pod/zarf-docker-registry-789d8ddfb8-4pfgj    Pulling image "127.0.0.1:32178/library/registry:2.7.1"
14m         Warning   Failed              pod/zarf-docker-registry-789d8ddfb8-4pfgj    Failed to pull image "127.0.0.1:32178/library/registry:2.7.1": rpc error: code = Unknown desc = failed to pull and unpack image "127.0.0.1:32178/library/registry:2.7.1": failed to resolve reference "127.0.0.1:32178/library/registry:2.7.1": failed to do request: Head "https://127.0.0.1:32178/v2/library/registry/manifests/2.7.1": http: server gave HTTP response to HTTPS client
14m         Warning   Failed              pod/zarf-docker-registry-789d8ddfb8-4pfgj    Error: ErrImagePull
34s         Normal    BackOff             pod/zarf-docker-registry-789d8ddfb8-4pfgj    Back-off pulling image "127.0.0.1:32178/library/registry:2.7.1"
13m         Warning   Failed              pod/zarf-docker-registry-789d8ddfb8-4pfgj    Error: ImagePullBackOff
15m         Normal    SuccessfulCreate    replicaset/zarf-docker-registry-789d8ddfb8   Created pod: zarf-docker-registry-789d8ddfb8-4pfgj
15m         Normal    ScalingReplicaSet   deployment/zarf-docker-registry              Scaled up replica set zarf-docker-registry-789d8ddfb8 to 1

output of kubectl -n zarf get all

NAME                                        READY   STATUS             RESTARTS   AGE
pod/injector                                1/1     Running            0          40m
pod/zarf-docker-registry-789d8ddfb8-4pfgj   0/1     ImagePullBackOff   0          40m

NAME                           TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/zarf-docker-registry   NodePort   10.0.16.59     <none>        5000:31999/TCP   40m
service/zarf-injector          NodePort   10.0.144.122   <none>        5000:32178/TCP   40m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/zarf-docker-registry   0/1     1            0           40m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/zarf-docker-registry-789d8ddfb8   1         1         0       40m

Severity/Priority

😕 Blocked on deploying zarf packages to Azure AKS

About this issue

Original URL
State: open
Created 2 years ago
Comments: 21 (17 by maintainers)

Most upvoted comments

hi folks, as a workaround (that includes patch by iceberg you can update containerd in your cluster.
Just apt update and upgrade the node. I do it through a DaemonSet (example):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: update-cluster
  labels:
    app: update-cluster
spec:
  selector:
    matchLabels:
      app: update-cluster
  template:
    metadata:
      labels:
        app: update-cluster
    spec:
      containers:
      - name: update-cluster
        image: alpine
        imagePullPolicy: IfNotPresent
        command:
          - nsenter
          - --target
          - "1"
          - --mount
          - --uts
          - --ipc
          - --net
          - --pid
          - --
          - sh
          - -c
          - |
            # apt update and upgrade 
            export DEBIAN_FRONTEND=noninteractive apt update && apt upgrade -y
            sleep infinity
        securityContext:
          privileged: true
      dnsPolicy: ClusterFirst
      hostPID: true

I was able to initialize zarf:

I did also deploy the Big Bang into AKS, but had a “bump”

gatekeeper has the label control-plane: controller-manager in their NS… preventing the hook to change the image … after removing it, everything went smoothly. -> in rke2 didn’t get the problem …

jsburckhardt on Nov 30, 2022

quick update, containerd was updated and I could deploy bigbang 1.48.0.

jsburckhardt on Feb 24, 2023

Tracking EKS AMI containerd update: awslabs/amazon-eks-ami#1162

Upstream issue has been closed and @brianrexrode has successfully tested zarf v0.25.2 with EKS V1.26

ntwkninja on Apr 14, 2023