istio: Sidecar injector seems to have invalid configs
Bug description
It seems that after changing values related to security, the sidecar injector lags behind and does things that causes the sidecar to come up with invalid configuration. We’ve seen this happen when:
- disabling/enabling telemetry, policy, citadel, nodeagent
We see logs like this from the sidecar:
[Envoy (Epoch 0)] [2020-03-30 15:05:50.914][24][critical][main] [external/envoy/source/server/server.cc:96] error initializing configuration ‘/etc/istio/proxy/envoy-rev0.json’: Invalid path: ./var/run/secrets/istio/root-cert.pem
Expected behavior
When making a change to istio, I expect the sidecar to ensure configurations match those changes
Steps to reproduce the bug
Install istio upgrade istio and remove citadel or another component restart pod with istio enabled see error
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.5.1
How was Istio installed? Standalone operator
Environment where bug was observed (cloud vendor, OS, etc)
Local (docker-desktop, Mac) AWS (EKS 1.15)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 21 (7 by maintainers)
Haven’t seen this issue since we implemented the solution above.
To summarize the problem: Essentially having both istiod and the sidecarinjector enabled causes a race condition between the two and the mutatingwebhookconfiguration associated with them.
The solution is to disable the sidecar injector webhook and allow istiod to manage injection.
Closing this out now. Thanks @howardjohn for all the help!