rancher: Bitnami Chart fails to refresh, causing Rancher Backup Restore timeouts and loops

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible): On the Rancher Management Cluster, in the Cluster Explorer, navigate to Rancher Backups. Create a Restore attempt.

Result: UI becomes unavalible. After a few minutes navigated to one of the Rancher nodes and ran kubectl logs -n cattle-resources-system -l app.kubernetes.io/name=rancher-backup -f to view the logs. The following error occurs, and then it trys again, never ending.

...
INFO[2021/04/05 21:45:34] Getting new UID for grb-7blfg
INFO[2021/04/05 21:45:34] Successfully restored cattle-globalrolebinding-grb-7blfg
INFO[2021/04/05 21:45:34] Processing controllerRef apps/v1/deployments/rancher
INFO[2021/04/05 21:45:34] Scaling up controllerRef apps/v1/deployments/rancher to 3
ERRO[2021/04/05 21:45:34] Error restoring cluster-scoped resources [error restoring bitnami-v3 of type management.cattle.io/v3, Resource=catalogs: restoreResource: err updating resource Timeout: request did not complete within requested timeout 34s]
ERRO[2021/04/05 21:45:34] error syncing 'restore-kj2pp': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing
...

Other details that may be helpful: After 15+ minutes of this constant looping, I decided to restore the cluster using rke. Which I successfully did. I setup daily reoccurring RKE etcd snapshots previously.

Environment information

  • Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.5.7
  • Installation option (single install/HA): HA

Cluster information

  • Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Imported
  • Machine type (cloud/VM/metal) and specifications (CPU/memory): VMs, 5 vCPUS, 8G Ram
  • Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:02:01Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
  • Docker version (use docker version):
$ docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:46:54 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:45:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 25 (7 by maintainers)

Most upvoted comments

My mistake. I had recreated the catalog in the wrong scope. When I recreate the ‘bitnami-test’ catalog in the global scope the app is no longer ‘orphaned’

image

Ok, this will work. Thank you @nickgerace, I greatly appreciate the assistance!

I am nervous to do this to all the apps lol. Will have to go though and backup all the clusters just in case, though it should be fine.