istio: Istio CNI blocks traffic in application init containers

After installing istio cni for using ambient mesh, i faced such problem that Istio CNI blocks traffic in application init containers. I am familiar with the workaround to the problem that is proposed in the documentation (https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers) but these instructions did not help me, the plugin ignores them.

I`ve set next configurations of istiod helm values file and istio cni values.

“istiod”

pilot:
 traceSampling: 100.0
 env:
   PILOT_HTTP10: "1"

istio_cni:
  enabled: true
  chained: true

“istio cni”

cni:
 ambient:
   enabled: true
 privileged: true

With deployed istio helms I have tried to run redis helm chart with init container using such values file.

helm install redis bitnami/redis --version 18.2.0 --namespace test --values <(cat redis-values.yaml)

<values.yaml>

image:
  registry: quay.io
  repository: opstree/redis
  tag: latest

master:
  livenessProbe:
    enabled: true
    initialDelaySeconds: 40
    periodSeconds: 15
    timeoutSeconds: 15
    successThreshold: 1
    failureThreshold: 5

  podSecurityContext:
    enabled: true
    fsGroup: 1337

  podAnnotations:
    proxy.istio.io/config: |
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: "true"
        ISTIO_META_DNS_AUTO_ALLOCATE: "true"
    traffic.sidecar.istio.io/includeOutboundIPRanges: "*"
    traffic.sidecar.istio.io/includeInboundPorts: "*"
    traffic.sidecar.istio.io/excludeOutboundPorts: "443"
    traffic.sidecar.istio.io/excludeInboundPorts: "443"
    traffic.sidecar.istio.io/excludeOutboundIPRanges: "172.20.0.1/32"

  initContainers:
    - name: init-celery-workers-restart
      image: bitnami/kubectl
      command: ['sh', '-c', 'sleep 30; kubectl delete pods -n test -l app.kubernetes.io/component=celery-worker']
      securityContext:
        runAsUser: 1337

  resources:
    requests:
      cpu: 80m
      memory: 60Mi
    limits:
      cpu: 80m
      memory: 60Mi

As pod starts i get such logs in init container`s output. At the same time istio cni, ztunnel & istiod have next logs.

“init container”

E0118 09:57:46.188460       8 memcache.go:265] couldn't get current server API group list: Get "https://172.20.0.1:443/api?timeout=32s": EOF
E0118 09:57:56.195553       8 memcache.go:265] couldn't get current server API group list: Get "https://172.20.0.1:443/api?timeout=32s": EOF
E0118 09:58:06.202729       8 memcache.go:265] couldn't get current server API group list: Get "https://172.20.0.1:443/api?timeout=32s": EOF
Unable to connect to the server: EOF

“istio cni”

info    cni istio-cni ambient cmdAdd podName: redis-master-0 podIPs: [{IP:10.0.95.14 Mask:ffffffff}]
info    cni Adding pod 'redis-master-0/test' (a9573a59-6adc-48e8-9d41-a9bd08fdc40c) to ipset
info    cni Adding route for redis-master-0/test: [table 100 10.0.95.14/32 via 192.168.126.2 dev istioin src 10.0.81.204]

“ztunnel”

INFO xds{id=2}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Address" size=1
INFO xds{id=2}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Address" size=1
WARN outbound{id=b395fe07254d28f5ec84ffc0850994eb}: ztunnel::proxy::outbound: failed dur=96.464µs err=unknown source: 10.0.82.219
WARN outbound{id=a1c3ba7180c313fa148f1899baa136e4}: ztunnel::proxy::outbound: failed dur=87.892µs err=unknown source: 10.0.82.219

“istiod”

info    ads Push debounce stable[44] 3 for config ServiceEntry/pslabs/redis-master.pslabs.svc.cluster.local and 1 more configs: 100.244777ms since last change, 103.240829ms since last push, full=true
info    ads XDS: Pushing Services:95 ConnectedEndpoints:6 Version:2024-01-18T10:05:13Z/37
info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
info    validationController    validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore, resourceVersion=14938048) is up-to-date. No change required.
error   controllers error handling istio-validator-istio-system, retrying (retry count: 1004): webhook is not ready, retry  controller=validation

Also i tried to use ServiceEntry and DestinationRule to allow traffic to k8s service bypassing istio mesh.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: k8s-api
  namespace: test
spec:
  hosts:
    - kubernetes.default.svc.cluster.local
  addresses:
    - 172.20.0.1
  endpoints:
    - address: 172.20.0.1
  exportTo:
    - "*"
  location: MESH_EXTERNAL
  resolution: STATIC
  ports:
    - number: 443
      name: https-k8s
      protocol: HTTPS
___

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: k8s-destrule
  namespace: test
spec:
  host: kubernetes.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 40 (26 by maintainers)

Commits related to this issue

Most upvoted comments

I reproduced this issue from AtoZ, and tried every tries to solve this issue. But I think this is not implemented in ambient mesh and Istio-cni yet. Those exclude* annotations only works in service mesh, not ambient mesh

@howardjohn If this feature about init-container’s outbound traffic is not implemented yet, can you assign me to this issue? I wanna implement this.

@zengyuxing007 Unfortunately this problem still exists, logs are the same as shown by @tomahkvt, and root problem is init-container cant be added in istio mesh

Thanks for your reply, I’ll work on fixing it

@howardjohn

Thanks for your reply, I will take a look at the coesbase.

I apologize for my mistake in selecting two outdated images out of the three for testing. However, I have tried a newer image named 1.22-alpha.fefbe293c1a0d04cfc79aae2683b2d402251edb1 which was built on April 9, 2024, and it appears to be working perfectly fine. Thank you very much for your assistance. It is my understanding that this issue will be resolved in the upcoming 1.22 release.

@Str33tWalk3r it works in my testing, can you give reproduction steps?

@zengyuxing007 @bleggett The development build incorporating this fix has been tested and appears to be functional. I’m looking forward to the release. I sincerely appreciate your dedication and efforts.

@bleggett Thank you a lot for your answer.

@bleggett I want to clarify. Did you fix init container network problem for ambient mesh or side car case?

@zengyuxing007 fixed it, not me, and they fixed it for the ambient mesh case, which is (as far as I know, given this issue is only related to ambient) the only place that needed fixing.

Sorry, just got confirmation from Eric that the change is not in beta 0 yet, but should be in -beta.1 since it was merged after beta.0 was released. I expect that the new build will be available in a day or 2.