argo-cd: `argocd app sync/diff --local` doesn't account for sidecar CMPs

i tried

• hard refresh in the ui
• adding --refresh to the cli command
• narrowing down the files sent with --local-include helmfile.yaml same result. the command also fails against a different app

argocd app diff istio-system-preprod --server-side-generate --local "$PWD" --refresh --local-include helmfile.yaml
WARN[0001] spec.plugin.name is set, which means this Application uses a plugin installed in the argocd-cm ConfigMap. Installing plugins via that ConfigMap is deprecated in Argo CD v2.5. Starting in Argo CD v2.6, this Application will fail to sync. Contact your Argo CD admin to make sure an upgrade plan is in place. More info: https://argo-cd.readthedocs.io/en/latest/operator-manual/upgrading/2.4-2.5/
FATA[0005] rpc error: code = Unknown desc = error receiving manifest file stream: error receiving tgz file: file checksum validation error: calc e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sent c897b354287cb29d7886a775131f3514113e7e1a35e48c87b1ff6b5f09a3ae0e

argocd log logs

argocd-server-86c9cf5874-8p8sw server time="2022-11-04T19:47:13Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-11-04T19:47:13Z" span.kind=server system=grpc
argocd-server-86c9cf5874-8p8sw server time="2022-11-04T19:47:13Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-11-04T19:47:13Z" grpc.time_ms=8.785 span.kind=server system=grpc
argocd-server-86c9cf5874-8p8sw server time="2022-11-04T19:47:13Z" level=info msg="received streaming call /application.ApplicationService/GetManifestsWithFiles" grpc.method=GetManifestsWithFiles grpc.request.content="query:<name:\"istio-system-preprod\" checksum:\"c897b354287cb29d7886a775131f3514113e7e1a35e48c87b1ff6b5f09a3ae0e\" appNamespace:\"\" > " grpc.service=application.ApplicationService grpc.start_time="2022-11-04T19:47:13Z" span.kind=server system=grpc
argocd-server-86c9cf5874-8p8sw server I1104 19:47:14.884336       1 request.go:601] Waited for 1.015823353s due to client-side throttling, not priority and fairness, request: GET:https://172.31.232.1:443/apis/security.istio.io/v1beta1?timeout=32s
argocd-server-86c9cf5874-8p8sw server time="2022-11-04T19:47:15Z" level=error msg="Partial success when performing preferred resource discovery" error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
argocd-repo-server-545b5785b4-jp2ld repo-server time="2022-11-04T19:47:15Z" level=error msg="finished streaming call with code Unknown" error="error receiving manifest file stream: error receiving tgz file: file checksum validation error: calc e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sent c897b354287cb29d7886a775131f3514113e7e1a35e48c87b1ff6b5f09a3ae0e" grpc.code=Unknown grpc.method=GenerateManifestWithFiles grpc.service=repository.RepoServerService grpc.start_time="2022-11-04T19:47:15Z" grpc.time_ms=1.423 span.kind=server system=grpc
argocd-server-86c9cf5874-8p8sw server time="2022-11-04T19:47:15Z" level=error msg="finished streaming call with code Unknown" error="rpc error: code = Unknown desc = error receiving manifest file stream: error receiving tgz file: file checksum validation error: calc e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sent c897b354287cb29d7886a775131f3514113e7e1a35e48c87b1ff6b5f09a3ae0e" grpc.code=Unknown grpc.method=GetManifestsWithFiles grpc.service=application.ApplicationService grpc.start_time="2022-11-04T19:47:13Z" grpc.time_ms=1540.895 span.kind=server system=grpc

is there an example of the directory context and files to include when using --server-side-generate?

Hi!

I also got into the error of checksums not matching today when trying to figure out the diff from local path. I see that the issue referenced in this PR was removed from the v2.6 milestone but the warning about not using --server-side-generate is still present and it clearly says that it will become the default in 2.6.

Does it mean that this issue will be addressed before v2.6 is released? Since otherwise once v2.6 would be released argocd app diff --local would be broken.

I hit the same issue in a scenario where we are using --port-forward and --plaintext on the argocd CLI - with the same exact checksum for the empty array of bytes.

For now, we’ll skirt around this issue in our use case (which is to show the diffs on pull requests pre-merge) by using the --revision flag, but it’d be nice to be able to run it locally, obviously.

@cha7ri I’m on k3d/k3s, so it’s a Traefik-based ingress controller. It seem that the Ingresses provided by the Helm chart have been incomplete, so I’ve bit the bullet and implemented Traefik Ingress as described in the docs.

could this be getting filtered at the ingress level? ie: the payload’s not reaching the server?

we tried the same exercise using kubectl port-forward to bypass nginx ingress, and the results were still the same

Yep, e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is the same “calculated” checksum as what @joshuasimon-taulia was getting - zero bytes.

Should I be running this with a debug log level or similar?

Wouldn’t hurt, but I don’t think there will be much more info. I think the issue is pretty low-level.

could this be getting filtered at the ingress level? ie: the payload’s not reaching the server?

Quite possibly. I doubt that the server is receiving the bytes and then just ignoring them. I bet they’re not making it over the network.

Following the original motivation, what do you think of app manifests also accepting --server-side-generate?