zarf: Intermittent Hangs at crane.Push() on Registry Push
Environment
Device and OS: Rocky 8 EC2 App version: 0.29.2 Kubernetes distro being used: RKE2 v1.26.9+rke2r1 Other: Bigbang v2.11.1
Steps to reproduce
zarf package deploy zarf-package-mvp-cluster-amd64-v5.0.0-alpha.7.tar.zst --confirm -l=debug- About 80% of the time or so, the above command gets stuck at
crane.Push(). A retry usually works.
Expected result
That the zarf package deploy... command wouldn’t get hung up, and continue along.
Actual Result
The zarf package deploy... command gets hung up
Visual Proof (screenshots, videos, text, etc)
[30;100m[30;100m DEBUG [0m[0m [90m[90m2023-10-23T18:37:19Z - Pushing ...1.dso.mil/ironbank/neuvector/neuvector/manager:5.1.3[0m[0m
[30;100m[30;100m DEBUG [0m[0m [90m[90m2023-10-23T18:37:19Z - crane.Push() /tmp/zarf-3272389118/images:registry1.dso.mil/ironbank/neuvector/neuvector/manager:5.1.3 -> 127.0.0.1:39357/ironbank/neuvector/neuvector/manager:5.1.3-zarf-487612511)[0m[0m
section_end:1698087620:step_script
[0K[31;1mERROR: Job failed: execution took longer than 35m0s seconds
Severity/Priority
There is a workaround, by keeping retrying until the process succeeds.
Additional Context
This looks exactly like https://github.com/defenseunicorns/zarf/issues/1568, which was closed.
We have a multi-node cluster on AWS EC2, our package size is about 2.9G. Here are a few things that we noticed after some extensive testing:
- this issue is not seen on a single EC2 node RKE2 cluster, it seems to only occur on multi-node clusters.
- our zarf docker registry is backed by S3. The issue is always seen in this case, but only if a multi-node cluster.
- if we back the registry with the default PVC (instead of S3), the issue is not seen at all. Since data transfer to S3 is slower than to the EBS backed PVC, maybe this extra time causes the problem to appear?
- disabling or enabling the zarf docker registry HPA doesn’t seem to matter either ways.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 25 (20 by maintainers)
Commits related to this issue
- fix: properly handle the tunnel error channel to retry image pushing (#2190) ## Description This PR fixes error channel handling for Zarf tunnels so lost pod connections don't result in infinite ... — committed to zarf-dev/zarf by Racer159 3 years ago
- feat: add configurable backoff and retries for Zarf operations (#2345) ## Description This adds `--backoff` and `--retries` to package operations to allow those to be configured. ## Related Is... — committed to zarf-dev/zarf by Racer159 3 years ago
Performance has significantly improved for me using
zarf package mirror-resources...as of Zarf0.40.1. I was seeing hang-ups every time and after upgrading to that Zarf version I haven’t seen a hang up in a couple attempts. Very promising so far 🤞Sadly, I do not think this solves the issue. I am still experiencing timeouts when publishing images. I am noticing that Zarf is now explicitly timing out instead of just hanging forever though.
I can confirm that adding a nodeSelector and taint/toleration to schedule the zarf registry pod(s) on the RKE2 control plane node(s) does ~resolve~ work around this issue:
@philiversen is correct, using
zarf package mirror-resourceshas been working for me, but I do want to add it to the lifecycle of the zarf init package, when deploying rook-ceph, so below is part of my yaml to help things along:A workaround that has seemed to work for me consistently to get past this particular issue is to use
zarf package mirror-resourcesconcurrently with azarf connect gittunnel open and mirror the package’s internal resources to the specified image registry and git repository. I use the IP address of the node that the zarf-docker-registry is running on and NodePort service the zarf-docker-registry is using for the--registry-url. Authentication is also required with--registry-push-username/passwordand--git-push-username/password. Gotten from running azarf tools get-creds. For example:zarf package mirror-resources zarf-package-structsure-enterprise-amd64-v5.9.0.tar.zst --registry-url <IP address of node zarf-docker-registry is running on>:31999 --registry-push-username zarf-push --registry-push-password <zarf-push-password> --git-url http://127.0.0.1:<tunnel port from zarf connect git> --git-push-username zarf-git-user --git-push-password <git-user-password>proof: running the
zarf package deploy zarf-package-structsure-enterprise-amd64-v5.0.0.tar.zst --confirmhas error’d everytime when it gets stuck on a specific blob unable to push the image:The
zarf package mirror-resourcecommand working to push the same image to the zarf-docker-registry with azarf connect gittunnel open that previously always gets stuck during azarf package deploycommand:After the zarf package’s internal resources are mirrored to the specified registry and git repository a
zarf package deploy zarf-package-structsure-enterprise-amd64-v5.0.0.tar.zst --confirmis successful.We’ve always had agent nodes when we saw this issue, whether with 1 or 3 control plane nodes. We’ve never seen this issue on single node clusters. Haven’t tried a cluster with only 3 control plane nodes and no agent nodes.