istio: Istio CNI blocks traffic in application init containers
After installing istio cni for using ambient mesh, i faced such problem that Istio CNI blocks traffic in application init containers. I am familiar with the workaround to the problem that is proposed in the documentation (https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers) but these instructions did not help me, the plugin ignores them.
I`ve set next configurations of istiod helm values file and istio cni values.
“istiod”
pilot:
traceSampling: 100.0
env:
PILOT_HTTP10: "1"
istio_cni:
enabled: true
chained: true
“istio cni”
cni:
ambient:
enabled: true
privileged: true
With deployed istio helms I have tried to run redis helm chart with init container using such values file.
helm install redis bitnami/redis --version 18.2.0 --namespace test --values <(cat redis-values.yaml)
<values.yaml>
image:
registry: quay.io
repository: opstree/redis
tag: latest
master:
livenessProbe:
enabled: true
initialDelaySeconds: 40
periodSeconds: 15
timeoutSeconds: 15
successThreshold: 1
failureThreshold: 5
podSecurityContext:
enabled: true
fsGroup: 1337
podAnnotations:
proxy.istio.io/config: |
proxyMetadata:
ISTIO_META_DNS_CAPTURE: "true"
ISTIO_META_DNS_AUTO_ALLOCATE: "true"
traffic.sidecar.istio.io/includeOutboundIPRanges: "*"
traffic.sidecar.istio.io/includeInboundPorts: "*"
traffic.sidecar.istio.io/excludeOutboundPorts: "443"
traffic.sidecar.istio.io/excludeInboundPorts: "443"
traffic.sidecar.istio.io/excludeOutboundIPRanges: "172.20.0.1/32"
initContainers:
- name: init-celery-workers-restart
image: bitnami/kubectl
command: ['sh', '-c', 'sleep 30; kubectl delete pods -n test -l app.kubernetes.io/component=celery-worker']
securityContext:
runAsUser: 1337
resources:
requests:
cpu: 80m
memory: 60Mi
limits:
cpu: 80m
memory: 60Mi
As pod starts i get such logs in init container`s output. At the same time istio cni, ztunnel & istiod have next logs.
“init container”
E0118 09:57:46.188460 8 memcache.go:265] couldn't get current server API group list: Get "https://172.20.0.1:443/api?timeout=32s": EOF
E0118 09:57:56.195553 8 memcache.go:265] couldn't get current server API group list: Get "https://172.20.0.1:443/api?timeout=32s": EOF
E0118 09:58:06.202729 8 memcache.go:265] couldn't get current server API group list: Get "https://172.20.0.1:443/api?timeout=32s": EOF
Unable to connect to the server: EOF
“istio cni”
info cni istio-cni ambient cmdAdd podName: redis-master-0 podIPs: [{IP:10.0.95.14 Mask:ffffffff}]
info cni Adding pod 'redis-master-0/test' (a9573a59-6adc-48e8-9d41-a9bd08fdc40c) to ipset
info cni Adding route for redis-master-0/test: [table 100 10.0.95.14/32 via 192.168.126.2 dev istioin src 10.0.81.204]
“ztunnel”
INFO xds{id=2}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Address" size=1
INFO xds{id=2}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Address" size=1
WARN outbound{id=b395fe07254d28f5ec84ffc0850994eb}: ztunnel::proxy::outbound: failed dur=96.464µs err=unknown source: 10.0.82.219
WARN outbound{id=a1c3ba7180c313fa148f1899baa136e4}: ztunnel::proxy::outbound: failed dur=87.892µs err=unknown source: 10.0.82.219
“istiod”
info ads Push debounce stable[44] 3 for config ServiceEntry/pslabs/redis-master.pslabs.svc.cluster.local and 1 more configs: 100.244777ms since last change, 103.240829ms since last push, full=true
info ads XDS: Pushing Services:95 ConnectedEndpoints:6 Version:2024-01-18T10:05:13Z/37
info validationController Not ready to switch validation to fail-closed: dummy invalid config not rejected
info validationController validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore, resourceVersion=14938048) is up-to-date. No change required.
error controllers error handling istio-validator-istio-system, retrying (retry count: 1004): webhook is not ready, retry controller=validation
Also i tried to use ServiceEntry and DestinationRule to allow traffic to k8s service bypassing istio mesh.
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: k8s-api
namespace: test
spec:
hosts:
- kubernetes.default.svc.cluster.local
addresses:
- 172.20.0.1
endpoints:
- address: 172.20.0.1
exportTo:
- "*"
location: MESH_EXTERNAL
resolution: STATIC
ports:
- number: 443
name: https-k8s
protocol: HTTPS
___
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: k8s-destrule
namespace: test
spec:
host: kubernetes.default.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 40 (26 by maintainers)
Commits related to this issue
- Fix podWorkloadBuilder (#48854) — committed to zengyuxing007/istio by zengyuxing007 3 years ago
- Fix podWorkloadBuilder (#48854) (#49767) * Fix podWorkloadBuilder (#48854) * fix TestPodLifecycleWorkloadGates * add some comments — committed to istio/istio by zengyuxing007 3 years ago
I reproduced this issue from AtoZ, and tried every tries to solve this issue. But I think this is not implemented in ambient mesh and Istio-cni yet. Those
exclude*annotations only works in service mesh, not ambient mesh@howardjohn If this feature about init-container’s outbound traffic is not implemented yet, can you assign me to this issue? I wanna implement this.
Thanks for your reply, I’ll work on fixing it
@howardjohn
Thanks for your reply, I will take a look at the coesbase.
I apologize for my mistake in selecting two outdated images out of the three for testing. However, I have tried a newer image named
1.22-alpha.fefbe293c1a0d04cfc79aae2683b2d402251edb1which was built on April 9, 2024, and it appears to be working perfectly fine. Thank you very much for your assistance. It is my understanding that this issue will be resolved in the upcoming 1.22 release.@Str33tWalk3r it works in my testing, can you give reproduction steps?
@zengyuxing007 @bleggett The development build incorporating this fix has been tested and appears to be functional. I’m looking forward to the release. I sincerely appreciate your dedication and efforts.
@bleggett Thank you a lot for your answer.
@zengyuxing007 fixed it, not me, and they fixed it for the
ambient meshcase, which is (as far as I know, given this issue is only related to ambient) the only place that needed fixing.Sorry, just got confirmation from Eric that the change is not in beta 0 yet, but should be in -beta.1 since it was merged after beta.0 was released. I expect that the new build will be available in a day or 2.