istio: Upgrade from 1.1.8 to 1.2.0 breaks SDS
Bug description
After upgrading to 1.2.0 using helm template installation I’ve got:
TLS error: Secret is not supplied by SDS
messages all over the ingress gateway instance. I have recreated the certificates restarted the ingress instance, the logs said it found the secret, but it still wouldn’t work.
Downgrading back to 1.1.8 github tag worked.
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure [ ] Docs [X] Installation [X] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [X] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure
Steps to reproduce the bug Upgrade to 1.2.0 from 1.1.8 using helm template with SDS set up in your gateway.
How was Istio installed? Helm template (github tag)
Environment where bug was observed (cloud vendor, OS, etc) GKE
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 49 (26 by maintainers)
@MrBlaise ideally follow https://istio.io/docs/setup/kubernetes/upgrade/steps/ should be good, requiring sidecar restart issue maybe due to https://github.com/istio/istio/issues/14853 as mentioned in a previous comment.
@wattli @quanjielin @duderino Is there a recommended way or guide on how to upgrade istio on a cluster whith sds enabled without downtime? Currently even with this fix the old sidecars will reject any connection and they need to be restarted to get the new version which works.
Thanks for the finding, it’s indeed an issue, we will send out the fix soon. @quanjielin
This is probably the root cause:
Seems like they won’t do rolling update, I need to manually delete them.