harvester: [BUG] upgrade from v.1.1.0 to v1.1.1 stuck on upgrade upgrade system service

Describe the bug Upgrade from v1.1.0 to v1.1.1. it stuck on upgrade system service(Upgrading monitoring)

To Reproduce Steps to reproduce the behavior:

  1. create v1.1.0 environment
  2. upgrade to v1.1.1

Expected behavior Upgrade could complete

Environment

  • Harvester ISO version: v1.1.0

Additional context Stuck on upgrade managed chart

Fri, Dec 2 2022 1:40:16 pm | Upgrading Monitoring
Fri, Dec 2 2022 1:40:16 pm | managedchart.management.cattle.io/rancher-monitoring-crd patched (no change)
Fri, Dec 2 2022 1:40:18 pm | managedchart.management.cattle.io/rancher-monitoring unchanged
Fri, Dec 2 2022 1:40:18 pm | managedchart.management.cattle.io/rancher-monitoring patched
Fri, Dec 2 2022 1:40:19 pm | managedchart.management.cattle.io/rancher-monitoring-crd patched
Fri, Dec 2 2022 1:40:19 pm | Waiting for ManagedChart fleet-local/rancher-monitoring from generation 6
Fri, Dec 2 2022 1:40:19 pm | Target version: 100.1.0+up19.0.3, Target state: ready
Fri, Dec 2 2022 1:40:20 pm | Current version: 100.1.0+up19.0.3, Current state: Modified, Current generation: 6
Fri, Dec 2 2022 1:40:25 pm | Sleep for 5 seconds to retry
Fri, Dec 2 2022 1:40:26 pm | Current version: 100.1.0+up19.0.3, Current state: Modified, Current generation: 6
Fri, Dec 2 2022 1:40:31 pm | Sleep for 5 seconds to retry
Fri, Dec 2 2022 1:40:31 pm | Current version: 100.1.0+up19.0.3, Current state: Modified, Current generation: 7

Stuck on upgrading monitoring

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 21 (6 by maintainers)

Most upvoted comments

FYI - in my case I have seen this:

NAME                                       	NAMESPACE                	REVISION	UPDATED                                	STATUS         	CHART                                                                                            	APP VERSION
fleet                                      	cattle-fleet-system      	47      	2022-12-06 08:26:35.3791045 +0000 UTC  	deployed       	fleet-100.1.0+up0.4.0                                                                            	0.4.0      
fleet-agent-harvester                      	cattle-fleet-system      	5       	2022-11-21 11:55:09.527963143 +0000 UTC	deployed       	fleet-agent-harvester-v0.0.0+s-cddece6760276de8c4a096f66e80ebb1d77304bc2c94ea220a314bcae734d     	           
fleet-agent-local                          	cattle-fleet-local-system	492     	2022-12-06 08:36:25.998286779 +0000 UTC	deployed       	fleet-agent-local-v0.0.0+s-d15ee758ef713110666b888ca0c3b5f819b730f340f139a6a01c7b0ad01de         	           
fleet-crd                                  	cattle-fleet-system      	3       	2022-10-28 13:35:02.956377898 +0000 UTC	deployed       	fleet-crd-100.1.0+up0.4.0                                                                        	0.4.0      
harvester                                  	harvester-system         	234     	2022-10-28 15:15:22.706752623 +0000 UTC	pending-upgrade	harvester-1.1.0                                                                                  	v1.1.0     
harvester-crd                              	harvester-system         	403     	2022-12-06 09:47:38.03219906 +0000 UTC 	deployed       	harvester-crd-1.1.1                                                                              	v1.1.1     
local-managed-system-agent                 	cattle-system            	435     	2022-12-06 08:36:25.599757196 +0000 UTC	deployed       	local-managed-system-agent-v0.0.0+s-c6ea391a626943550690f7a2d5a6d3a31674d735a22ab77106c98f2b221d3	           
mcc-local-managed-system-upgrade-controller	cattle-system            	441     	2022-12-06 08:36:26.141908423 +0000 UTC	deployed       	system-upgrade-controller-100.0.3+up0.3.3                                                        	v0.9.1     
rancher                                    	cattle-system            	3       	2022-10-28 13:32:51.015678293 +0000 UTC	deployed       	rancher-2.6.9                                                                                    	v2.6.9     
rancher-logging                            	cattle-logging-system    	75      	2022-12-06 08:36:26.939880303 +0000 UTC	deployed       	rancher-logging-100.1.3+up3.17.7                                                                 	3.17.7     
rancher-logging-crd                        	cattle-logging-system    	69      	2022-12-06 08:36:28.443721853 +0000 UTC	deployed       	rancher-logging-crd-100.1.3+up3.17.7                                                             	           
rancher-monitoring                         	cattle-monitoring-system 	152     	2022-10-28 15:15:27.486916015 +0000 UTC	pending-upgrade	rancher-monitoring-100.1.0+up19.0.3                                                              	0.50.0     
rancher-monitoring-crd                     	cattle-monitoring-system 	206     	2022-10-28 15:15:20.449373287 +0000 UTC	pending-upgrade	rancher-monitoring-crd-100.1.0+up19.0.3                                                          	           
rancher-webhook                            	cattle-system            	3       	2022-10-28 13:35:10.792063079 +0000 UTC	deployed       	rancher-webhook-1.0.6+up0.2.7                                                                    	0.2.7      
rke2-canal                                 	kube-system              	11      	2022-10-28 14:28:10.535980612 +0000 UTC	deployed       	rke2-canal-v3.24.1-build2022101103                                                               	v3.24.1    
rke2-coredns                               	kube-system              	9       	2022-10-28 14:34:49.257204173 +0000 UTC	deployed       	rke2-coredns-1.19.401                                                                            	1.9.3      
rke2-ingress-nginx                         	kube-system              	13      	2022-10-28 13:54:53.344449206 +0000 UTC	deployed       	rke2-ingress-nginx-4.1.005                                                                       	1.2.0      
rke2-metrics-server                        	kube-system              	9       	2022-10-28 13:55:09.543552208 +0000 UTC	deployed       	rke2-metrics-server-2.11.100-build2022101106                                                     	0.6.1      
rke2-multus                                	kube-system              	9       	2022-10-28 14:12:07.584420983 +0000 UTC	deployed       	rke2-multus-v3.8-build2022101103                                                                 	3.8

And my workaround has been

helm rollback -n harvester-system harvester
helm rollback -n cattle-monitoring-system rancher-monitoring-crd
helm rollback -n cattle-monitoring-system rancher-monitoring

Not sure if this has any side-effects but at least my upgrade went through after this…

+1
Martin-Weiss on Dec 6, 2022
Read more comments on GitHub
← harvester: [BUG] Upgrade from 1.1.2 to 1.2.0 stuck 50% through the "Upgrading System Service"
harvester: [BUG] Upgrade: rancher-monitoring charts can't be upgraded →