rancher: Bitnami Chart fails to refresh, causing Rancher Backup Restore timeouts and loops
What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible): On the Rancher Management Cluster, in the Cluster Explorer, navigate to Rancher Backups. Create a Restore attempt.
Result:
UI becomes unavalible.
After a few minutes navigated to one of the Rancher nodes and ran kubectl logs -n cattle-resources-system -l app.kubernetes.io/name=rancher-backup -f
to view the logs.
The following error occurs, and then it trys again, never ending.
...
INFO[2021/04/05 21:45:34] Getting new UID for grb-7blfg
INFO[2021/04/05 21:45:34] Successfully restored cattle-globalrolebinding-grb-7blfg
INFO[2021/04/05 21:45:34] Processing controllerRef apps/v1/deployments/rancher
INFO[2021/04/05 21:45:34] Scaling up controllerRef apps/v1/deployments/rancher to 3
ERRO[2021/04/05 21:45:34] Error restoring cluster-scoped resources [error restoring bitnami-v3 of type management.cattle.io/v3, Resource=catalogs: restoreResource: err updating resource Timeout: request did not complete within requested timeout 34s]
ERRO[2021/04/05 21:45:34] error syncing 'restore-kj2pp': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing
...
Other details that may be helpful: After 15+ minutes of this constant looping, I decided to restore the cluster using rke. Which I successfully did. I setup daily reoccurring RKE etcd snapshots previously.
Environment information
- Rancher version (
rancher/rancher
/rancher/server
image tag or shown bottom left in the UI): v2.5.7 - Installation option (single install/HA): HA
Cluster information
- Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Imported
- Machine type (cloud/VM/metal) and specifications (CPU/memory): VMs, 5 vCPUS, 8G Ram
- Kubernetes version (use
kubectl version
):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:02:01Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
- Docker version (use
docker version
):
$ docker version
Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:46:54 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:28 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 25 (7 by maintainers)
My mistake. I had recreated the catalog in the wrong scope. When I recreate the ‘bitnami-test’ catalog in the global scope the app is no longer ‘orphaned’
Ok, this will work. Thank you @nickgerace, I greatly appreciate the assistance!
I am nervous to do this to all the apps lol. Will have to go though and backup all the clusters just in case, though it should be fine.