istio: Istio blocking initContainer traffic (using Istio CNI)
Hoping someone can help with this issue. I have a few initContainers which need to reach out to download various bits and pieces before my application containers spin up.
Currently I’m getting errors trying to deal with any HTTP(S) traffic but IP layer traffic seems fine.
I have the CNI plugin running (rather than using Istio’s traditional initContainer approach to configure IPTABLES).
E.g., in my initContainer -
- If I try to run
curl www.google.comI get back0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 2404:6800:4006:806::2004: Cannot assign requested address - If I just run a simple
ping www.google.comeverything is fine and it finds the address.
I see similar behaviour on both Busybox and Alpine images, using both curl and wget.
My versions are below -
client version: 1.4.0
citadel version: 1.4-alpha.c4ec4b8c102da86d4d02f4b268f9af60a35dee1c
galley version: 1.4-alpha.ee4df7fb5e7a9069a6fc0c9de85b501a6c791c38
ingressgateway version: 1.4-alpha.c4ec4b8c102da86d4d02f4b268f9af60a35dee1c
pilot version: 1.4-alpha.400f9e3780af62b1742372085b45317d8fbe9810
policy version: 1.4-alpha.c4ec4b8c102da86d4d02f4b268f9af60a35dee1c
sidecar-injector version: 1.4-alpha.ee4df7fb5e7a9069a6fc0c9de85b501a6c791c38
data plane version: 1.4-alpha.1c963ed31062b56c866ac8a601df0e91edefc12d (4 proxies), 1.4-alpha.8e7ab73399db64fa37f3af190c6caa1a36e66586 (27 proxies), 1.4-alpha.c4ec4b8c102da86d4d02f4b268f9af60a35dee1c (12 proxies), 1.3.3 (4 proxies), 1.4-alpha.1df6a42f1f10b472fde05a13e588bdbf67927869 (17 proxies)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (7 by maintainers)
Commits related to this issue
- Use UID 1337 in storage initializer container see here: https://github.com/istio/istio/issues/23802 — committed to mesosphere/kfserving by alembiewski 2 years ago
- Use UID 1337 in storage initializer container see here: https://github.com/istio/istio/issues/23802 — committed to mesosphere/kfserving by alembiewski 2 years ago
- Use UID 1337 in storage initializer container see here: https://github.com/istio/istio/issues/23802 — committed to mesosphere/kfserving by alembiewski 2 years ago
So, now that the sidecar KEP is pretty much dead in the water, is there any hope of solving this without a monkey patch?
No, there will be no network until Envoy is running. Iptables rules will direct traffic to Envoy, but envoy is not there, so you get connection refused. Envoy only runs in normal container so its not around for init
This is a limitation of CNI. Because it sets up iptables rules before init containers, the traffic will be redirected to Envoy which is not yet running. There are a few workarounds that you can use. You can run as UID 1337 in the init container, or disable iptables for specific port/ip ranges (https://istio.io/docs/reference/config/annotations/).
We had an init container with rclone setup to retrieve some s3 artifacts that had this issue. I was able to get around by adding the following annotation to the pod spec:
source: https://istio.io/latest/docs/reference/config/annotations/
I need to see what the implications of this is though. Haven’t investigated beyond this.
I thought that problem is that init container starts before network is set That is why I added long timeout and retry, but it is retrying 2-3 minutes and there is no point when network becomes available
Shouldn’t making init container wait for network be enough? I understand it can be without network until istio container will finish its thing, but when istio will be done shouldn’t it get network (even if it started before istio sidecar)?
@michalzxc we are not because its impossible/very hard without https://github.com/kubernetes/enhancements/issues/753 which we are actively working with k8s to get solved.
Thanks all, this is the info I needed. On reflection, this is probably a weakness of K8s, as it should probably support something like runlevels rather than a binary init/application container distinction.
Would it be worth someone (potentially me) updating the CNI docs to reflect that using it will break initContainer’s usual behaviour? Running as 1337 would probably cause issues with PSPs, but it is a good workaround to have front and centre in the docs.
That sounds accurate to me.