argo-cd: Argo CD causing deprecated resource warnings on EKS/GKE

We did it like this and it works:

        resource.exclusions: |
          - apiGroups:
            - policy
            kinds:
            - PodSecurityPolicy
            clusters:
            - "*"

I have the same problem on a GKE cluster and it is becoming pretty serious:

This cluster will not be scheduled for an automatic upgrade to v1.25, the next minor version, because your API clients have used APIs in the last 30 days that are removed in this version. Once the cluster’s version 1.24.14-gke.2700 reaches end of life, the cluster will be scheduled for automatic upgrade to v1.25. But upgrading the cluster before it’s migrated to updated APIs could cause the API calls to break.

So my cluster cannot upgrade to v1.25 because of this:

The version used is: v2.9.0+389cf75

{
    "Version": "v2.9.0+389cf75",
    "BuildDate": "2023-08-08T15:42:51Z",
    "GitCommit": "389cf7521de83bdd70d18ea33d1d4e233d6a5382",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.6",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.1.0 2023-06-19T16:58:18Z",
    "HelmVersion": "v3.12.1+gf32a527",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Is there a plan to solve this once and for all and stop querying the api for deprecated resources if none of the manifests use these?

Just ignoring all resources that have a deprecated API is not really a workable solution

Ahhh. Yep, that’s what’s happening. Argo CD by default watches everything. I think you could just exclude that kind and avoid the API calls.

Hi, does anyone know if there is a way to verify if ArgoCD exclusion works? I have added resource.exclusions section to argocd-cm configmap and have restartedargocd-server and argocd-application-controller deployments but a few days later GKE still reports that ArgoCd is scanning /apis/policy/v1beta1/podsecuritypolicies and this prevents GKE from upgrading the cluster from v1.24 to v1.25

@alexsunins You can use Logs Explorer on GCP to search for hits to that API.

A basic query would look like:

"PodSecurityPolicy"
"v1beta1"

Note: you will require the privateLogViewer role to see these events.

See: Locate API clients making write calls to deprecated APIs - GKE for guidance.

@alexsunins In Argo CD’s case we don’t define PodSecurityPolicy in the codebase. The controller will watch all resources available in the cluster. If a cluster is upgraded to a newer k8s version where a specific API version isn’t available it won’t be watched by Argo CD. This means that this GKE validation is probably a false positive. Do you know how exactly GKE is verifying that a given application uses a certain API version?

Hi @florianmutter did you do anything to your existing gce.gke-metrics-agent psp before upgrading to v1.25? Or we can leave it and just upgrade the GKE to v1.25? Thanks.

Since it is managed by Google and part of GKE I did not do anything. We updated to v1.25 and everything worked for us.

In our GKE cluster we only have one PodSecurityPolicy resource:

$ kubectl get PodSecurityPolicy --all-namespaces                              
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME                    PRIV    CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
gce.gke-metrics-agent   false          RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            hostPath,secret,configMap

$ kubectl describe PodSecurityPolicy gce.gke-metrics-agent --all-namespaces
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Name:  gce.gke-metrics-agent

Settings:
  Allow Privileged:                       false
  Allow Privilege Escalation:             false
  Default Add Capabilities:               <none>
  Required Drop Capabilities:             <none>
  Allowed Capabilities:                   <none>
  Allowed Volume Types:                   hostPath,secret,configMap
  Allow Host Network:                     true
  Allow Host Ports:                       <none>
  Allow Host PID:                         false
  Allow Host IPC:                         false
  Read Only Root Filesystem:              false
  SELinux Context Strategy: RunAsAny      
    User:                                 <none>
    Role:                                 <none>
    Type:                                 <none>
    Level:                                <none>
  Run As User Strategy: RunAsAny          
    Ranges:                               <none>
  FSGroup Strategy: RunAsAny              
    Ranges:                               <none>
  Supplemental Groups Strategy: RunAsAny  
    Ranges:                               <none>

Is argo cd maybe only querying the API to see if there are resources with some argecd label? We updated to ArgoCD 2.8.0 today maybe this changes something. Right now this is the recommendation from google:

@crenshaw-dev by prevents I meant that GKE is refusing to upgrade the versions automatically unless deprecated APIs are removd. I don’t recall the exact warning message but the message implies that manual intervention is required before GKE can continue with automatic version upgrade.

Hi, does anyone know if there is a way to verify if ArgoCD exclusion works? I have added resource.exclusions section to argocd-cm configmap and have restartedargocd-server and argocd-application-controller deployments but a few days later GKE still reports that ArgoCd is scanning /apis/policy/v1beta1/podsecuritypolicies and this prevents GKE from upgrading the cluster from v1.24 to v1.25