istio: ipv6 doesn't work anymore after updating from 1.11.5 to 1.12.2

Bug Description

When the pod only has an ipv6 address, the sidecar fails to start. It seems to try to create iptables rules in the ipv4 table which doesn’t exist in this case. This is a regression that was introduced in 1.12 as 1.11 worked fine so far

Before updating on version 1.11.5 the sidecards got created just fine

% kubectl get pod -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP                         NODE       NOMINATED NODE   READINESS GATES
nginx-85b98978db-hprkr   2/2     Running   0          2m10s   2001:19f0:6c01:1f10:2::7   worker-1   <none>           <none>

 % kubectl logs -l app=nginx --all-containers                
2022-02-17T14:38:21.334472Z     info    ads     XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2022-02-17T14:38:21.334569Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.665433537s
2022-02-17T14:38:21.334598Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.665404565s
2022-02-17T14:38:21.334851Z     info    cache   returned workload certificate from cache        ttl=23h59m59.665157079s
2022-02-17T14:38:21.335061Z     info    ads     SDS: PUSH request for node:nginx-85b98978db-hprkr.default resources:1 size:1.1kB resource:ROOTCA
2022-02-17T14:38:21.335124Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.664879103s
2022-02-17T14:38:21.335153Z     info    ads     SDS: PUSH for node:nginx-85b98978db-hprkr.default resources:1 size:1.1kB resource:ROOTCA
2022-02-17T14:38:21.335062Z     info    ads     SDS: PUSH request for node:nginx-85b98978db-hprkr.default resources:1 size:4.0kB resource:default
2022-02-17T14:38:22.594234Z     info    Initialization took 1.542331611s
2022-02-17T14:38:22.594259Z     info    Envoy proxy is ready
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Thu Feb 17 14:38:15 2022

2022-02-17T14:38:15.082858Z     info    Running command: ip6tables-save 
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/02/17 14:38:20 [notice] 1#1: using the "epoll" event method
2022/02/17 14:38:20 [notice] 1#1: nginx/1.21.6
2022/02/17 14:38:20 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/02/17 14:38:20 [notice] 1#1: OS: Linux 5.10.84-flatcar
2022/02/17 14:38:20 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/02/17 14:38:20 [notice] 1#1: start worker processes
2022/02/17 14:38:20 [notice] 1#1: start worker process 31
2022/02/17 14:38:20 [notice] 1#1: start worker process 32
arian@Arians-MBP vultr-kubernetes % kubectl logs -l app=nginx --all-containers
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/02/17 14:38:20 [notice] 1#1: using the "epoll" event method
2022/02/17 14:38:20 [notice] 1#1: nginx/1.21.6
2022/02/17 14:38:20 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/02/17 14:38:20 [notice] 1#1: OS: Linux 5.10.84-flatcar
2022/02/17 14:38:20 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/02/17 14:38:20 [notice] 1#1: start worker processes
2022/02/17 14:38:20 [notice] 1#1: start worker process 31
2022/02/17 14:38:20 [notice] 1#1: start worker process 32
2022-02-17T14:38:21.334472Z     info    ads     XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2022-02-17T14:38:21.334569Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.665433537s
2022-02-17T14:38:21.334598Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.665404565s
2022-02-17T14:38:21.334851Z     info    cache   returned workload certificate from cache        ttl=23h59m59.665157079s
2022-02-17T14:38:21.335061Z     info    ads     SDS: PUSH request for node:nginx-85b98978db-hprkr.default resources:1 size:1.1kB resource:ROOTCA
2022-02-17T14:38:21.335124Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.664879103s
2022-02-17T14:38:21.335153Z     info    ads     SDS: PUSH for node:nginx-85b98978db-hprkr.default resources:1 size:1.1kB resource:ROOTCA
2022-02-17T14:38:21.335062Z     info    ads     SDS: PUSH request for node:nginx-85b98978db-hprkr.default resources:1 size:4.0kB resource:default
2022-02-17T14:38:22.594234Z     info    Initialization took 1.542331611s
2022-02-17T14:38:22.594259Z     info    Envoy proxy is ready
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Thu Feb 17 14:38:15 2022

After upgrading to 1.12.2 the sidecar container doesn’t start anymore and crashloopbackoffs

% istioctl upgrade
% kubectl rollout restart deployment nginx
% kubectl get pod -owide
NAME                     READY   STATUS                  RESTARTS     AGE    IP                         NODE       NOMINATED NODE   READINESS GATES
nginx-556bc7cbd8-gq2pg   2/2     Running                 0            114s   2001:19f0:6c01:1f10:2::c   worker-1   <none>           <none>
nginx-6ddd56bf6-w68zx    0/2     Init:CrashLoopBackOff   1 (4s ago)   7s     2001:19f0:6c01:1f10:1::c   worker-0   <none>           <none>

% kubectl logs -l app=nginx --all-containers
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2022-02-17T14:27:32.342611Z     info    Running command: iptables-restore --noflush /tmp/iptables-rules-1645108052342433684.txt1047550074
2022-02-17T14:27:32.343854Z     error   Command error output: xtables other problem: line 2 failed
2022-02-17T14:27:32.343869Z     error   Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1645108052342433684.txt1047550074, exit status 1

Version

% kubectl version --short
Client Version: v1.22.3
Server Version: v1.23.3

Before:

% istioctl version             
client version: 1.11.5
control plane version: 1.11.5
data plane version: 1.11.5 (2 proxies)

After:

% istioctl version
client version: 1.12.2
control plane version: 1.12.2
data plane version: 1.12.2 (1 proxies)

Additional Information

Before update:

bug-report-1.11.5.tar.gz

 % istioctl bug-report

Target cluster context: kubernetes-admin@kubernetes

Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage } AND { Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage }
end-time: 2022-02-17 15:46:34.09607 +0100 CET


The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Info:      version.BuildInfo{Version:"1.11.5", GitRevision:"4959f6f447280f5fd702f1af92ad26630fb00bd6", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.11.5"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.11.5}


Fetching proxy logs for the following containers:

default/nginx/nginx-85b98978db-ns278/istio-proxy
default/nginx/nginx-85b98978db-ns278/nginx
istio-system/istio-ingressgateway/istio-ingressgateway-86b75f74c9-j8kvq/istio-proxy
istio-system/istiod/istiod-7c8c747bd-sg8sq/discovery

Fetching Istio control plane information from cluster.

Running istio analyze on all namespaces and report as below:
Analysis Report:
Info [IST0102] (Namespace istio-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-system istio-injection=enabled' to enable it, or 'kubectl label namespace istio-system istio-injection=disabled' to explicitly mark it as not needing injection.
Creating an archive at /Users/arian/Projects/vultr-kubernetes/bug-report.tar.gz.
Cleaning up temporary files in /var/folders/_t/r_qtl_3d1g7dm2vz75h1vx880000gn/T/bug-report.
Done.

After update to 1.12.2

bug-report-1.12.2.tar.gz

% istioctl bug-report

Target cluster context: kubernetes-admin@kubernetes

Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-system,kube-public,kube-node-lease,local-path-storage }
end-time: 2022-02-17 15:54:39.382633 +0100 CET



Cluster endpoint: https://[2a05:f480:1800:263:5400:3ff:fedc:d9e9]:6443
CLI version:
version.BuildInfo{Version:"1.12.2", GitRevision:"unknown", GolangVersion:"go1.17.7", BuildStatus:"Nix", GitTag:"1.12.2"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Info:      version.BuildInfo{Version:"1.12.2", GitRevision:"af0d66fd0aa363e9a7b0164f3a94ba36252fe60f", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.12.2"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.12.2, 1.11.5}


Fetching proxy logs for the following containers:

default/nginx/nginx-556bc7cbd8-gq2pg/istio-proxy
default/nginx/nginx-556bc7cbd8-gq2pg/nginx
default/nginx/nginx-6ddd56bf6-w68zx/istio-proxy
default/nginx/nginx-6ddd56bf6-w68zx/nginx
istio-system/istio-ingressgateway/istio-ingressgateway-c6d9f449-n8kzk/istio-proxy
istio-system/istiod/istiod-5ffcccb477-v5pvd/discovery

Fetching Istio control plane information from cluster.

Running istio analyze on all namespaces and report as below:
Analysis Report:
Warning [IST0105] (Pod default/nginx-556bc7cbd8-gq2pg) The image of the Istio proxy running on the pod does not match the image defined in the injection configuration (pod image: docker.io/istio/proxyv2:1.11.5; injection configuration image: docker.io/istio/proxyv2:1.12.2). This often happens after upgrading the Istio control-plane and can be fixed by redeploying the pod.
Creating an archive at /Users/arian/Projects/vultr-kubernetes/bug-report.tar.gz.
Cleaning up temporary files in /var/folders/_t/r_qtl_3d1g7dm2vz75h1vx880000gn/T/bug-report.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

There were a LOT of ipv6 PRs in this time frame. Mostly by me…

Do you have the full logs of the init contianer? seems the bug report does not/did not capture it

+1
howardjohn on Feb 17, 2022
Read more comments on GitHub
← istio: Exposing jager via ingress gateway gives strange errors with http2 - Envoy breaks http1.1 <-> http2 ALPN
istio: 0.8 Not being able to use hosts different of "*" on VirtualService for (Ingress) Gateways →