zarf: Intermittent Hangs at crane.Push() on Registry Push

Environment

Device and OS: Rocky 8 EC2 App version: 0.29.2 Kubernetes distro being used: RKE2 v1.26.9+rke2r1 Other: Bigbang v2.11.1

Steps to reproduce

  1. zarf package deploy zarf-package-mvp-cluster-amd64-v5.0.0-alpha.7.tar.zst --confirm -l=debug
  2. About 80% of the time or so, the above command gets stuck at crane.Push(). A retry usually works.

Expected result

That the zarf package deploy... command wouldn’t get hung up, and continue along.

Actual Result

The zarf package deploy... command gets hung up

Visual Proof (screenshots, videos, text, etc)

  DEBUG   2023-10-23T18:37:19Z  -  Pushing ...1.dso.mil/ironbank/neuvector/neuvector/manager:5.1.3
  DEBUG   2023-10-23T18:37:19Z  -  crane.Push() /tmp/zarf-3272389118/images:registry1.dso.mil/ironbank/neuvector/neuvector/manager:5.1.3 -> 127.0.0.1:39357/ironbank/neuvector/neuvector/manager:5.1.3-zarf-487612511)
section_end:1698087620:step_script
ERROR: Job failed: execution took longer than 35m0s seconds

Severity/Priority

There is a workaround, by keeping retrying until the process succeeds.

Additional Context

This looks exactly like https://github.com/defenseunicorns/zarf/issues/1568, which was closed.

We have a multi-node cluster on AWS EC2, our package size is about 2.9G. Here are a few things that we noticed after some extensive testing:

  • this issue is not seen on a single EC2 node RKE2 cluster, it seems to only occur on multi-node clusters.
  • our zarf docker registry is backed by S3. The issue is always seen in this case, but only if a multi-node cluster.
  • if we back the registry with the default PVC (instead of S3), the issue is not seen at all. Since data transfer to S3 is slower than to the EBS backed PVC, maybe this extra time causes the problem to appear?
  • disabling or enabling the zarf docker registry HPA doesn’t seem to matter either ways.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 25 (20 by maintainers)

Commits related to this issue

Most upvoted comments

Performance has significantly improved for me using zarf package mirror-resources... as of Zarf 0.40.1. I was seeing hang-ups every time and after upgrading to that Zarf version I haven’t seen a hang up in a couple attempts. Very promising so far 🤞

This should be mitigated now in https://github.com/defenseunicorns/zarf/releases/tag/v0.32.4 - leaving this open until we get more community feedback though (and again this is a mitigation not a true fix, that will have to happen upstream).

Sadly, I do not think this solves the issue. I am still experiencing timeouts when publishing images. I am noticing that Zarf is now explicitly timing out instead of just hanging forever though.

Screenshot 2024-03-22 at 4 33 58 PM

I can confirm that adding a nodeSelector and taint/toleration to schedule the zarf registry pod(s) on the RKE2 control plane node(s) does ~resolve~ work around this issue:

kubectl patch deployment -n zarf zarf-docker-registry --patch-file=/dev/stdin <<-EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zarf-docker-registry
  namespace: zarf
spec:
  template:
    spec:
      nodeSelector:
        node-role.kubernetes.io/master: "true"
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
EOF

@philiversen is correct, using zarf package mirror-resources has been working for me, but I do want to add it to the lifecycle of the zarf init package, when deploying rook-ceph, so below is part of my yaml to help things along:

variables:
- name: CONTROL_PLANE_ONE_ADDRESS
  prompt: true
  description: This is the IP/Hostname of a control-plane
components:
...
    actions:
      onDeploy:
        before:
          - cmd: |-
              ./zarf package mirror-resources zarf-init-amd64-*.tar.zst \
                --registry-url ${ZARF_VAR_CONTROL_PLANE_ONE_ADDRESS}:${ZARF_NODEPORT} \
                --registry-push-username zarf-push \
                --registry-push-password ${ZARF_REGISTRY_AUTH_PUSH}

A workaround that has seemed to work for me consistently to get past this particular issue is to use zarf package mirror-resources concurrently with a zarf connect git tunnel open and mirror the package’s internal resources to the specified image registry and git repository. I use the IP address of the node that the zarf-docker-registry is running on and NodePort service the zarf-docker-registry is using for the --registry-url. Authentication is also required with --registry-push-username/password and --git-push-username/password. Gotten from running a zarf tools get-creds. For example:

zarf package mirror-resources zarf-package-structsure-enterprise-amd64-v5.9.0.tar.zst --registry-url <IP address of node zarf-docker-registry is running on>:31999 --registry-push-username zarf-push --registry-push-password <zarf-push-password> --git-url http://127.0.0.1:<tunnel port from zarf connect git> --git-push-username zarf-git-user --git-push-password <git-user-password>

proof: running the zarf package deploy zarf-package-structsure-enterprise-amd64-v5.0.0.tar.zst --confirm has error’d everytime when it gets stuck on a specific blob unable to push the image: Screenshot 2024-05-02 at 4 16 35 PM

The zarf package mirror-resource command working to push the same image to the zarf-docker-registry with a zarf connect git tunnel open that previously always gets stuck during a zarf package deploy command: Screenshot 2024-05-02 at 4 27 45 PM

After the zarf package’s internal resources are mirrored to the specified registry and git repository a zarf package deploy zarf-package-structsure-enterprise-amd64-v5.0.0.tar.zst --confirm is successful.

Also what is the node role layout for your clusters - I have heard reports that if all nodes are control plane nodes that the issue is also not seen.

We’ve always had agent nodes when we saw this issue, whether with 1 or 3 control plane nodes. We’ve never seen this issue on single node clusters. Haven’t tried a cluster with only 3 control plane nodes and no agent nodes.