istio: istio-operator/istioctl > 1.19 unexpectedly remove MutatingWebhookConfiguration::istio-revision-tag

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

We are following Istio Canary Upgrades and found istioctl uninstall will unexpected remove the revision-tag created for canary upgrades.

Version

$ istioctl version
client version: 1.18.1
control plane version: 1.18.1
data plane version: 1.18.1 (1 proxies)

$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.6
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1

Additional Information

How to repro

Install Istio 1-18-1

$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.18.1 sh -

$ istioctl version
no ready Istio pods in "istio-system"
1.18.1

$ istioctl operator init --revision 1-18-1 --tag "1.18.1-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.18.1-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

$ cat istio-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: minimal
  tag: 1.18.1-distroless
  revision: 1-18-1
  components:
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        name: istio-ingressgateway

$ k apply -f istio-operator.yaml
istiooperator.install.istio.io/istiocontrolplane created

$ k label namespace default istio.io/rev=prod-stable --overwrite
namespace/default labeled

$ istioctl tag set prod-canary --revision 1-18-1
Revision tag "prod-canary" created, referencing control plane revision "1-18-1". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'

$ istioctl tag set prod-stable --revision 1-18-1
Revision tag "prod-stable" created, referencing control plane revision "1-18-1". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-18-1
prod-stable 1-18-1   default

$ k get MutatingWebhookConfigurations
NAME                               WEBHOOKS   AGE
istio-revision-tag-prod-canary     2          27s
istio-revision-tag-prod-stable     2          18s
istio-sidecar-injector-1-18-1      2          13m

Upgrade to istio 1-19-3

$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.3 sh -

$ istioctl version
client version: 1.19.3
control plane version: 1.18.1
data plane version: 1.18.1 (1 proxies)

$ istioctl operator init --revision 1-19-3 --tag "1.19.3-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

$ cat istio-operator-1.19.3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: minimal
  tag: 1.19.3-distroless
  revision: 1-19-3
  components:
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        name: istio-ingressgateway

$ k apply -f istio-operator-1.19.3.yaml
istiooperator.install.istio.io/istiocontrolplane configured

$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag set prod-canary --revision 1-19-3 --overwrite
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-19-3
prod-stable 1-19-3   default

Uninstall Istio 1-18-1

$ istioctl uninstall --revision 1-18-1
  Removed Deployment:istio-operator:istio-operator-1-18-1.
  Removed Deployment:istio-system:istiod-1-18-1.
  Removed Service:istio-operator:istio-operator-1-18-1.
  Removed Service:istio-system:istiod-1-18-1.
  Removed ConfigMap:istio-system:istio-1-18-1.
  Removed ConfigMap:istio-system:istio-sidecar-injector-1-18-1.
object: Pod:istio-system:istiod-1-18-1-d4b874889-mbdqx is not being deleted because it no longer exists
  Removed ServiceAccount:istio-operator:istio-operator-1-18-1.
  Removed ServiceAccount:istio-system:istiod-1-18-1.
  Removed RoleBinding:istio-system:istiod-1-18-1.
  Removed Role:istio-system:istiod-1-18-1.
  Removed HorizontalPodAutoscaler:istio-system:istiod-1-18-1.
  Removed PodDisruptionBudget:istio-system:istiod-1-18-1.
  Removed MutatingWebhookConfiguration::istio-sidecar-injector-1-18-1.
  Removed ValidatingWebhookConfiguration::istio-validator-1-18-1-istio-system.
  Removed ClusterRole::istio-operator-1-18-1.
  Removed ClusterRole::istio-reader-clusterrole-1-18-1-istio-system.
  Removed ClusterRole::istiod-clusterrole-1-18-1-istio-system.
  Removed ClusterRole::istiod-gateway-controller-1-18-1-istio-system.
  Removed ClusterRoleBinding::istio-operator-1-18-1.
  Removed ClusterRoleBinding::istio-reader-clusterrole-1-18-1-istio-system.
  Removed ClusterRoleBinding::istiod-clusterrole-1-18-1-istio-system.
  Removed ClusterRoleBinding::istiod-gateway-controller-1-18-1-istio-system.

After above command run, istio-operator log shows istio-operator deleted the revision-tag

Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary. Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.

✔ Ingress gateways installed
- Pruning removed resources[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
	>  goroutine 411 [running]:
	>  runtime/debug.Stack()
	>  	runtime/debug/stack.go:24 +0x5e
	>  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
	>  	sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/log.go:60 +0xcd
	>  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc001531780, 0x0?)
	>  	sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/deleg.go:111 +0x32
	>  github.com/go-logr/logr.Logger.Enabled(...)
	>  	github.com/go-logr/logr@v1.2.4/logr.go:261
	>  github.com/go-logr/logr.Logger.Info({{0x3449f28?, 0xc001531780?}, 0xc000eb5ad0?}, {0x2eaaa40, 0x14}, {0x0, 0x0, 0x0})
	>  	github.com/go-logr/logr@v1.2.4/logr.go:274 +0x72
	>  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c16be0, {0x3443ec8, 0xc000ae5ef0}, {0x2a90f80?, 0xc0000b7bc0?})
	>  	sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:344 +0x4f7
	>  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c16be0, {0x3443ec8, 0xc000ae5ef0})
	>  	sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:266 +0x1c9
	>  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	>  	sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:227 +0x79
	>  created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 73
	>  	sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:223 +0x565
2023-10-18T05:59:43.118237Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:43.118344Z	info	installer	Reconciling IstioOperator
2023-10-18T05:59:43.172465Z	info	installer	Updating IstioOperator
2023-10-18T05:59:43.172841Z	info	installer	Detecting third-party JWT support
2023-10-18T05:59:43.663698Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:43.964206Z	info	installer	Watching a change for istio resource: istio-system/istio-1-18-1
2023-10-18T05:59:44.276378Z	info	installer	Watching a change for istio resource: istio-system/istio-sidecar-injector-1-18-1
2023-10-18T05:59:44.784728Z	info	kube	Starting Pilot K8S CRD controller	controller=analysis-controller
2023-10-18T05:59:44.784764Z	info	kube	Pilot K8S CRD controller synced in 43.103µs	controller=analysis-controller
2023-10-18T05:59:44.784780Z	info	kube	Starting Pilot K8S CRD controller	controller=analysis-controller
2023-10-18T05:59:44.784789Z	info	kube	Pilot K8S CRD controller synced in 11.401µs	controller=analysis-controller
2023-10-18T05:59:44.787734Z	info	installer	Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T05:59:44.787781Z	info	installer	Pilot is waiting on dependency...
2023-10-18T05:59:44.787797Z	info	installer	Cni is waiting on dependency...
2023-10-18T05:59:44.787807Z	info	installer	IngressGateways is waiting on dependency...
2023-10-18T05:59:44.787815Z	info	installer	Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T05:59:44.787868Z	info	installer	Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T05:59:44.787879Z	info	installer	EgressGateways is waiting on dependency...
2023-10-18T05:59:44.787744Z	info	installer	Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T05:59:44.788452Z	info	installer	Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T05:59:45.064198Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.368543Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.524126Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.864460Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:46.055691Z	info	installer	Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:46.074125Z	info	installer	Generated manifest objects are the same as cached for component Base.
2023-10-18T05:59:46.074185Z	info	installer	Unblocking dependency Pilot.
2023-10-18T05:59:46.074224Z	info	installer	Dependency for Pilot has completed, proceeding.
2023-10-18T05:59:46.074234Z	info	installer	Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
2023-10-18T05:59:46.269846Z	info	installer	Generated manifest objects are the same as cached for component Pilot.
2023-10-18T05:59:46.270075Z	info	installer	Unblocking dependency Cni.
2023-10-18T05:59:46.270149Z	info	installer	Unblocking dependency IngressGateways.
2023-10-18T05:59:46.270190Z	info	installer	Unblocking dependency EgressGateways.
2023-10-18T05:59:46.363699Z	info	installer	Dependency for EgressGateways has completed, proceeding.
2023-10-18T05:59:46.363748Z	info	installer	Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T05:59:46.363867Z	info	installer	Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T05:59:46.363883Z	info	installer	Dependency for Cni has completed, proceeding.
2023-10-18T05:59:46.363900Z	info	installer	Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T05:59:46.363942Z	info	installer	Generated manifest objects are the same as cached for component Cni.
2023-10-18T05:59:46.363948Z	info	installer	Dependency for IngressGateways has completed, proceeding.
2023-10-18T05:59:46.363957Z	info	installer	Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T05:59:46.367348Z	info	installer	Watching a change for istio resource: /istio-sidecar-injector-1-18-1
2023-10-18T05:59:46.375135Z	info	installer	Generated manifest objects are the same as cached for component IngressGateways.

2023-10-18T05:59:46.764964Z	info	installer	Watching a change for istio resource: /istio-validator-1-18-1-istio-system
2023-10-18T05:59:47.166836Z	info	installer	Watching a change for istio resource: /istio-reader-clusterrole-1-18-1-istio-system
2023-10-18T05:59:47.383171Z	info	installer	Watching a change for istio resource: /istiod-clusterrole-1-18-1-istio-system
2023-10-18T05:59:47.672115Z	info	installer	Watching a change for istio resource: /istiod-gateway-controller-1-18-1-istio-system
2023-10-18T05:59:48.178424Z	info	installer	Watching a change for istio resource: /istio-reader-clusterrole-1-18-1-istio-system
2023-10-18T05:59:48.568199Z	info	installer	Watching a change for istio resource: /istiod-clusterrole-1-18-1-istio-system
2023-10-18T05:59:48.709472Z	info	installer	Watching a change for istio resource: /istiod-gateway-controller-1-18-1-istio-system
2023-10-18T06:00:01.971533Z	info	installer	Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache.
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:00:01.971583Z	info	  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:00:01.973419Z	info	installer	Watching a change for istio resource: /istio-revision-tag-prod-canary
2023-10-18T06:00:01.978272Z	info	installer	Watching a change for istio resource: /istio-revision-tag-prod-stable
2023-10-18T06:00:01.978330Z	info	installer	Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-stable from Cache.
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
2023-10-18T06:00:01.978365Z	info	  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
2023-10-18T06:00:04.586509Z	info	installer	Reconciling IstioOperator
2023-10-18T06:00:04.680744Z	info	installer	Updating IstioOperator
2023-10-18T06:00:04.680811Z	info	installer	Detecting third-party JWT support
2023-10-18T06:00:06.371940Z	info	kube	Starting Pilot K8S CRD controller	controller=analysis-controller
2023-10-18T06:00:06.371984Z	info	kube	Pilot K8S CRD controller synced in 51.203µs	controller=analysis-controller
2023-10-18T06:00:06.372005Z	info	kube	Starting Pilot K8S CRD controller	controller=analysis-controller
2023-10-18T06:00:06.372015Z	info	kube	Pilot K8S CRD controller synced in 14.2µs	controller=analysis-controller
2023-10-18T06:00:06.374930Z	info	installer	Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T06:00:06.375007Z	info	installer	Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T06:00:06.375030Z	info	installer	IngressGateways is waiting on dependency...
2023-10-18T06:00:06.375042Z	info	installer	EgressGateways is waiting on dependency...
2023-10-18T06:00:06.375050Z	info	installer	Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T06:00:06.467573Z	info	installer	Cni is waiting on dependency...
2023-10-18T06:00:06.467621Z	info	installer	Pilot is waiting on dependency...
2023-10-18T06:00:06.467644Z	info	installer	Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T06:00:06.467706Z	info	installer	Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T06:00:07.473118Z	info	installer	Generated manifest objects are the same as cached for component Base.
2023-10-18T06:00:07.473176Z	info	installer	Unblocking dependency Pilot.
2023-10-18T06:00:07.473196Z	info	installer	Dependency for Pilot has completed, proceeding.
2023-10-18T06:00:07.473206Z	info	installer	Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
2023-10-18T06:00:07.765838Z	info	installer	Generated manifest objects are the same as cached for component Pilot.
2023-10-18T06:00:07.765881Z	info	installer	Unblocking dependency Cni.
2023-10-18T06:00:07.765887Z	info	installer	Unblocking dependency IngressGateways.
2023-10-18T06:00:07.765891Z	info	installer	Unblocking dependency EgressGateways.
2023-10-18T06:00:07.765898Z	info	installer	Dependency for EgressGateways has completed, proceeding.
2023-10-18T06:00:07.765908Z	info	installer	Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T06:00:07.765953Z	info	installer	Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T06:00:07.765959Z	info	installer	Dependency for Cni has completed, proceeding.
2023-10-18T06:00:07.765964Z	info	installer	Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T06:00:07.765989Z	info	installer	Generated manifest objects are the same as cached for component Cni.
2023-10-18T06:00:07.765994Z	info	installer	Dependency for IngressGateways has completed, proceeding.
2023-10-18T06:00:07.765998Z	info	installer	Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T06:00:07.772990Z	info	installer	Generated manifest objects are the same as cached for component IngressGateways.
- Pruning removed resources


$ istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

Also observed istioctl install will prune user set revision tag.

2023-10-24 01:55:41.664295: ++ istioctl install -y --verify --revision 1-19-3 -f setup/istio-operator.yaml
2023-10-24 01:55:42.422813: WARNING: Istio is being upgraded from 1.19.0 to 1.19.3.
2023-10-24 01:55:42.423134: Before upgrading, you may wish to use 'istioctl x precheck' to check for upgrade warnings.
2023-10-24 01:55:43.364625:
2023-10-24 01:55:45.723057: - Processing resources for Istio core.
2023-10-24 01:55:45.820562: \N{HEAVY CHECK MARK} Istio core installed
2023-10-24 01:55:47.009512: - Processing resources for Istiod.
2023-10-24 01:55:47.069855: \N{HEAVY CHECK MARK} Istiod installed
2023-10-24 01:55:47.467949: - Processing resources for Ingress gateways.
2023-10-24 01:55:47.469677: \N{HEAVY CHECK MARK} Ingress gateways installed
2023-10-24 01:55:52.979047: - Pruning removed resources Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-24 01:55:53.026674: Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.

This is blocking us upgrading to istio 1.19.3, which is a security fix. @hanxiaop do you know any other workaround? considering it will take time to release a new istio version.

@sonnyhcl I’m going to fix it up soon. Before having the fix, maybe you can try to set the annotation install.operator.istio.io/owning-resource-not-pruned="true" to keep the resource? If it’s not working, deleting the operator related labels (starting with install.operator.istio.io) of the tag webhook you’ve created with istioctl.

+2
hanxiaop on Oct 24, 2023

BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.

+2
hanxiaop on Oct 18, 2023
Read more comments on GitHub
← istio: Istio Operator Error Loops
istio: Istio-pilot crash due to high memory leak  →