rancher: [BUG] After Rancher 2.6.11 upgrade, rancher-v2.6.11-secret-migrator userAgent is constantly updating cluster resources

Rancher Server Setup

  • Rancher version: 2.6.11
  • Installation option (Docker install/Helm Chart): Helm Chart on RKE1 K8s v1.21.10
  • Proxy/Cert Details: TCP load balancer; Rancher terminates TLS with company certificate

Information about the Cluster

  • Kubernetes version: 1.21.10
  • Cluster Type (Local/Downstream): local

User Information

  • What is the role of the user logged in? n/a

Describe the bug After upgrading Rancher from 2.6.9 to 2.6.11, we saw a huge increase in kube-apiserver calls in the local cluster. A substantial contributor of these API calls are coming from the previously-unseen rancher-2.6.11-secrets-migrator userAgent, at the rate of ~2000-3000 PUT calls per hour:

I0417 18:14:01.727838       1 httplog.go:94] "HTTP" verb="PUT" URI="/apis/management.cattle.io/v3/clusters/c-bq829" latency="24.283621ms" userAgent="rancher-v2.6.11-secret-migrator" srcIP="172.19.2.95:49578" resp=200

To Reproduce

  • Install Rancher 2.6.11
  • Add some downstream clusters

Result Incessant PUTs to the clusters.management.cattle.io resources from the rancher-v2.6.11-secret-migrator userAgent

Expected Result Perhaps a one-time spike as “secrets” were “migrated”, but little to no calls after that. I’m not sure what this particular userAgent is actually doing – is it syncing Project-scoped Secrets? Cattle-cluster-agent secrets? Tokens? There’s vanishingly little documentation about what this thing does.

Screenshots image

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 20 (4 by maintainers)

Most upvoted comments

Test Environment:

Rancher version: v2.6-head 9c4e37f Rancher cluster type: HA Docker version: 20.10

Downstream cluster type: RKE2


Testing:

Tested this issue with the following steps:

  1. Create Rancher server on 2.6.9
  2. Provision rke2 aws node driver cluster
  3. Upgrade Rancher server to 2.6-head
  4. Monitor pods in local cluster
  5. Verify there are no pods with spike in resources being used

Result I am not seeing the spike in the secrets migrator resource anymore. This issue seems to be not reproducible anymore.