argo-cd: argocd-applicationset-controller doesn't start on arm64

No worries about the ping, sorry this hasn’t been addressed yet.

@jgwest can we cherry-pick https://github.com/argoproj/applicationset/commit/a2b34681feb1d0a00b5a09f615e9716856c3c97b back and release 0.4.2?

I have upgraded to the latest RC https://github.com/argoproj/argo-cd/releases/tag/v2.4.0-rc1 and things seems to be working fine now.

I patched @jr64’s image in my argocd-autopilot and it’s working now on my Pi 4 https://github.com/narze/home-cluster/commit/0f370548c4a0e3d09f4ca31c5ad233864ae97aba

The problem here is that argocd does not yet provide arm64 builds of the applicationset image. We can easily verify that with docker manifest inspect -v quay.io/argoproj/argocd-applicationset:latest:

{
        "Ref": "quay.io/argoproj/argocd-applicationset:latest",
        "Descriptor": {
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "digest": "sha256:ea791980b0e2925e97492c4185a6319a28e7790a58392b3006e1961018ec0e02",
                "size": 2402,
                "platform": {
                        "architecture": "amd64",
                        "os": "linux"
                }
        },

I have investigated a bit and I think the problem is the different build system they use for that particular image. The other images use buildx for building which has integrated support for qemu making it easy to build cross-platform images for arm64. Argocd-applicationset however seems to use a custom build chain based on “make” which is fine for building a native image but probably not so easy to adapt for cross-compilation.

While we wait for official support, I have forked their repository and natively compiled it for arm64 on my Raspberry Pi 4. If you want you can directly use ghcr.io/jr64/argocd-applicationset:v0.4.0 for testing (v0.3.0 is also provided, my preliminary tests show that both should work) however I would not recommend that for production use.

Alternatively, you can also build it yourself. Clone the applicationset repo, checkout the branch you want and run CONTAINER_REGISTRY=<your registry> IMAGE_NAMESPACE=<your namespace> IMAGE_TAG=<the tag you want> make image. I had to install docker, go, make and gcc as dependencies first.

Just updated to latest 2.3 today and experiencing the same issue on Raspberry Pi 4

I know why applicationset don’t support arm64 yet. It’s because the arm64 support commit is not yet in release builds: So, in the meantime, we could wait it. We can watch its releases for notifications on https://github.com/argoproj/applicationset

Doh! I need to reflash. My fault!!!

When I attempt to install 2.4.0-rc5 or rc4 on a Raspberry PI k3s cluster. My understanding is that ARM support is in this release. Here’s the errors I get from a kubectl describe pod

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  3m2s                 default-scheduler  Successfully assigned argocd/argocd-application-controller-0 to k3s-master
  Normal   Pulling    99s (x4 over 3m)     kubelet            Pulling image "quay.io/argoproj/argocd:v2.4.0-rc5"
  Warning  Failed     98s (x4 over 2m59s)  kubelet            Failed to pull image "quay.io/argoproj/argocd:v2.4.0-rc5": rpc error: code = NotFound desc = failed to pull and unpack image "quay.io/argoproj/argocd:v2.4.0-rc5": no match for platform in manifest: not found
  Warning  Failed     98s (x4 over 2m59s)  kubelet            Error: ErrImagePull
  Warning  Failed     74s (x6 over 2m59s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    59s (x7 over 2m59s)  kubelet            Back-off pulling image "quay.io/argoproj/argocd:v2.4.0-rc5"

The strange thing I did this to work with RC4 a few days ago.

I can confirm that, as of at least ArgoCD 2.4.3 if not sooner, and the latest helm chart 4.9 or greater, the image used is “argocd” instead of “argocd-application-controller” and it works out of the box on my Pi 4s and CM4s both.

I would presume this was a natural win after consolidated the outer applicationset repository into the main argocd repository proper, thus combining the build processes of each release that build arm images because now the applicationset controller is in the main argocd image.

I believe this can be closed with the advisement with upgrade to ArgoCD 2.4.x series because that’s when this seems to have occurred.

I could go git blaming and linking commits etc but this is being typed on a mobile phone sooo 😃

I was being dumb - as soon as I saw your uname . <hangs head in shame>

Fix #8400 is supposed to address the problem. What I was attempting to do was track whether the updated images included the fix or not. Maybe there’s a better way to do that, but I don’t see a good way to correlate the image tag/sha with the git tag/sha. I’ll try your container later, thanks.