istio: istioctl proxy-config does not work on EKS IPv6 clusters

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

The important detail here I believe is that I’m using AWS EKS 1.29 IPv6 with their weird IPv4 link-local NAT egress.

Related analysis from @howardjohn (thanks!) https://github.com/istio/istio/issues/49476#issuecomment-1959864574.

I am able to shell into a pod and run curl http://localhost:15000/config_dump however all istioctl pc commands fail.

❯ istioctl pc l myapp-web-5f5b6c695f-n97jz
2024-02-22T17:20:23.339406Z     error   klog    an error occurred forwarding 60619 -> 15000: error forwarding port 15000 to pod 23c62f702462c62a8fd4ee632038302b4bc7e4a235264a3033d806bb51d26a41, uid : failed to execute portforward in network namespace "/var/run/netns/cni-b8d29c04-fabf-ba84-4e72-5bb792cbd579": failed to connect to localhost:15000 inside namespace "23c62f702462c62a8fd4ee632038302b4bc7e4a235264a3033d806bb51d26a41", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable
address found
2024-02-22T17:20:23.341278Z     error   port forward failed: lost connection to pod
Error: failed to execute command on myapp-web-5f5b6c695f-n97jz.myapp sidecar: failure running port forward process: Get "http://localhost:60619/config_dump": EOF

With logging enabled:

❯ istioctl pc l myapp-web-6dc987cfb8-vw5nx --vklog 9
2024-02-22T17:11:18.019274Z     info    klog    curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: istioctl/1.20.3" 'https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx'
2024-02-22T17:11:18.515508Z     info    klog    HTTP Trace: DNS Lookup for REDACTED.gr7.us-east-2.eks.amazonaws.com resolved to [{10.34.7.111 } {10.34.105.167 }]
2024-02-22T17:11:18.618212Z     info    klog    HTTP Trace: Dial to tcp:10.34.7.111:443 succeed
2024-02-22T17:11:18.870643Z     info    klog    GET https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx 200 OK in 851 milliseconds
2024-02-22T17:11:18.870997Z     info    klog    HTTP Statistics: DNSLookup 75 ms Dial 101 ms TLSHandshake 115 ms ServerProcessing 132 ms Duration 851 ms
2024-02-22T17:11:18.871016Z     info    klog    Response Headers:
2024-02-22T17:11:18.871034Z     info    klog        Content-Type: application/vnd.kubernetes.protobuf
2024-02-22T17:11:18.871070Z     info    klog        X-Kubernetes-Pf-Flowschema-Uid: 48115a1f-2c46-4eb8-8350-bab2930df0cf
2024-02-22T17:11:18.871084Z     info    klog        X-Kubernetes-Pf-Prioritylevel-Uid: 329063b9-b080-4896-962f-c405f977c3d0
2024-02-22T17:11:18.871095Z     info    klog        Date: Thu, 22 Feb 2024 17:11:18 GMT
2024-02-22T17:11:18.871120Z     info    klog        Audit-Id: ddfce54e-83f4-4e55-a896-eb900d4df924
2024-02-22T17:11:18.871153Z     info    klog        Cache-Control: no-cache, private
2024-02-22T17:11:18.941864Z     debug   klog    Response Body:
00000000  6b 38 73 00 0a 09 0a 02  76 31 12 03 50 6f 64 12  |k8s.....v1..Pod.|
00000010  93 63 0a 90 23 0a 1a 6d  79 61 70 70 2d 77 65 62  |.c..#..myapp-web|
00000020  2d 36 64 63 39 38 37 63  66 62 38 2d 76 77 35 6e  |-6dc987cfb8-vw5n|
00000030  78 12 15 6d 79 61 70 70  2d 77 65 62 2d 36 64 63  |x..myapp-web-6dc|
00000040  39 38 37 63 66 62 38 2d  1a 05 6d 79 61 70 70 22  |987cfb8-..myapp"|
00000050  00 2a 24 63 36 34 63 33  39 62 61 2d 39 30 65 32  |.*$c64c39ba-90e2|
00000060  2d 34 31 64 37 2d 62 30  33 35 2d 66 66 34 64 61  |-41d7-b035-ff4da|
00000070  34 33 32 64 39 64 63 32  08 38 30 37 30 31 38 37  |432d9dc2.8070187|
00000080  30 38 00 42 08 08 8f fe  dd ae 06 10 00 5a 10 0a  |08.B.........Z..|
00000090  03 61 70 70 12 09 6d 79  61 70 70 2d 77 65 62 5a  |.app..myapp-webZ|
000000a0  22 0a 1b 61 70 70 2e 6b  75 62 65 72 6e 65 74 65  |"..app.kubernete|
000000b0  73 2e 69 6f 2f 63 6f 6d  70 6f 6e 65 6e 74 12 03  |s.io/component..|
000000c0  77 65 62 5a 23 0a 1a 61  70 70 2e 6b 75 62 65 72  |webZ#..app.kuber|
000000d0  6e 65 74 65 73 2e 69 6f  2f 69 6e 73 74 61 6e 63  |netes.io/instanc|
000000e0  65 12 05 6d 79 61 70 70  5a 24 0a 1c 61 70 70 2e  |e..myappZ$..app.|
000000f0  6b 75 62 65 72 6e 65 74  65 73 2e 69 6f 2f 6d 61  |kubernetes.io/ma|
00000100  6e 61 67 65 64 2d 62 79  12 04 48 65 6c 6d 5a 23  |naged-by..HelmZ#|
00000110  0a 16 61 70 70 2e 6b 75  62 65 72 6e 65 74 65 73  |..app.kubernetes|
00000120  2e 69 6f 2f 6e 61 6d 65  12 09 6d 79 61 70 70 2d  |.io/name..myapp-|
00000130  77 65 62 5a 2a 0a 19 61  70 70 2e 6b 75 62 65 72  |webZ*..app.kuber|
00000140  6e 65 74 65 73 2e 69 6f  2f 70 61 72 74 2d 6f 66  |netes.io/part-of|
00000150  12 0d 62 72 61 69 6e 62  61 73 65 6c 69 6e 65 5a  |..brainbaselineZ|
00000160  45 0a 19 61 70 70 2e 6b  75 62 65 72 6e 65 74 65  |E..app.kubernete|
00000170  73 2e 69 6f 2f 76 65 72  73 69 6f 6e 12 28 64 30  |s.io/version.(d0|
00000180  34 61 34 34 62 36 37 63  64 66 30 61 37 64 61 30  |4a44b67cdf0a7da0|
00000190  61 65 65 63 64 64 38 63  36 38 30 61 31 65 35 62  |aeecdd8c680a1e5b|
000001a0  38 34 39 35 32 32 5a 25  0a 0d 68 65 6c 6d 2e 73  |849522Z%..helm.s|
000001b0  68 2f 63 68 61 72 74 12  14 62 72 61 69 6e 62 61  |h/chart..brainba|
000001c0  73 65 6c 69 6e 65 2d 30  2e 30 2e 35 39 5a 1f 0a  |seline-0.0.59Z..|
000001d0  11 70 6f 64 2d 74 65 6d  70 6c 61 74 65 2d 68 61  |.pod-template-ha|
000001e0  73 68 12 0a 36 64 63 39  38 37 63 66 62 38 5a 22  |sh..6dc987cfb8Z"|
000001f0  0a 19 73 65 63 75 72 69  74 79 2e 69 73 74 69 6f  |..security.istio|
00000200  2e 69 6f 2f 74 6c 73 4d  6f 64 65 12 05 69 73 74  |.io/tlsMode..ist|
00000210  69 6f 5a 2c 0a 1f 73 65  72 76 69 63 65 2e 69 73  |ioZ,..service.is|
00000220  74 69 6f 2e 69 6f 2f 63  61 6e 6f 6e 69 63 61 6c  |tio.io/canonical|
00000230  2d 6e 61 6d 65 12 09 6d  79 61 70 70 2d 77 65 62  |-name..myapp-web|
00000240  5a 4f 0a 23 73 65 72 76  69 63 65 2e 69 73 74 69  |ZO.#service.isti|
00000250  6f 2e 69 6f 2f 63 61 6e  6f 6e 69 63 61 6c 2d 72  |o.io/canonical-r|
00000260  65 76 69 73 69 6f 6e 12  28 64 30 34 61 34 34 62  |evision.(d04a44b|
00000270  36 37 63 64 66 30 61 37  64 61 30 61 65 65 63 64  |67cdf0a7da0aeecd|
00000280  64 38 63 36 38 30 61 31  65 35 62 38 34 39 35 32  |d8c680a1e5b84952|
00000290  32 5a 33 0a 07 76 65 72  73 69 6f 6e 12 28 64 30  |2Z3..version.(d0|
000002a0  34 61 34 34 62 36 37 63  64 66 30 61 37 64 61 30  |4a44b67cdf0a7da0|
000002b0  61 65 65 63 64 64 38 63  36 38 30 61 31 65 35 62  |aeecdd8c680a1e5b|
000002c0  38 34 39 35 32 32 62 17  0a 0c 69 73 74 69 6f 2e  |849522b...istio.|
000002d0  69 6f 2f 72 65 76 12 07  64 65 66 61 75 6c 74 62  |io/rev..defaultb|
000002e0  2e 0a 27 6b 75 62 65 63  74 6c 2e 6b 75 62 65 72  |..'kubectl.kuber|
000002f0  6e 65 74 65 73 2e 69 6f  2f 64 65 66 61 75 6c 74  |netes.io/default|
00000300  2d 63 6f 6e 74 61 69 6e  65 72 12 03 77 65 62 62  |-container..webb|
00000310  33 0a 2c 6b 75 62 65 63  74 6c 2e 6b 75 62 65 72  |3.,kubectl.kuber|
00000320  6e 65 74 65 73 2e 69 6f  2f 64 65 66 61 75 6c 74  |netes.io/default|
00000330  2d 6c 6f 67 73 2d 63 6f  6e 74 61 69 6e 65 72 12  |-logs-container.|
00000340  03 77 65 62 62 27 0a 12  70 72 6f 6d 65 74 68 65  |.webb'..promethe|
00000350  75 73 2e 69 6f 2f 70 61  74 68 12 11 2f 73 74 61  |us.io/path../sta|
00000360  74 73 2f 70 72 6f 6d 65  74 68 65 75 73 62 1b 0a  |ts/prometheusb..|
00000370  12 70 72 6f 6d 65 74 68  65 75 73 2e 69 6f 2f 70  |.prometheus.io/p|
00000380  6f 72 74 12 05 31 35 30  32 30 62 1c 0a 14 70 72  |ort..15020b...pr|
00000390  6f 6d 65 74 68 65 75 73  2e 69 6f 2f 73 63 72 61  |ometheus.io/scra|
000003a0  70 65 12 04 74 72 75 65  62 5e 0a 15 70 72 6f 78  |pe..trueb^..prox|
000003b0  79 2e 69 73 74 69 6f 2e  69 6f 2f 63 6f 6e 66 69  |y.istio.io/confi|
000003c0  67 12 45 74 65 72 6d 69  6e 61 74 69 6f 6e 44 72  |g.EterminationDr|
000003d0  61 69 6e 44 75 72 61 74  69 6f 6e 3a 20 33 30 73  |ainDuration: 30s|
000003e0  0a 68 6f 6c 64 41 70 70  6c 69 63 61 74 69 6f 6e  |.holdApplication|
000003f0  55 6e 74 69 6c 50 72 6f  78 79 53 74 61 72 74 73  |UntilProxyStarts|
00000400  3a 20 66 61 6c 73 65 0a  62 97 02 0a 17 73 69 64  |: false.b....sid|
00000410  65 63 61 72 2e 69 73 74  69 6f 2e 69 6f 2f 73 74  |ecar.istio.io/st|
00000420  61 74 75 73 12 fb 01 7b  22 69 6e 69 74 43 6f 6e  |atus...{"initCon|
00000430  74 61 69 6e 65 72 73 22  3a 5b 22 69 73 74 69 6f  |tainers":["istio|
00000440  2d 69 6e 69 74 22 2c 22  69 73 74 69 6f 2d 70 72  |-init","istio-pr|
00000450  6f 78 79 22 5d 2c 22 63  6f 6e 74 61 69 6e 65 72  |oxy"],"container|
00000460  73 22 3a 6e 75 6c 6c 2c  22 76 6f 6c 75 6d 65 73  |s":null,"volumes|
00000470  22 3a 5b 22 77 6f 72 6b  6c 6f 61 64 2d 73 6f 63  |":["workload-soc|
00000480  6b 65 74 22 2c 22 63 72  65 64 65 6e 74 69 61 6c  |ket","credential|
00000490  2d 73 6f 63 6b 65 74 22  2c 22 77 6f 72 6b 6c 6f  |-socket","worklo|
000004a0  61 64 2d 63 65 72 74 73  22 2c 22 69 73 74 69 6f  |ad-certs","istio|
000004b0  2d 65 6e 76 6f 79 22 2c  22 69 73 74 69 6f 2d 64  |-envoy","istio-d|
000004c0  61 74 61 22 2c 22 69 73  74 69 6f 2d 70 6f 64 69  |ata","istio-podi|
000004d0  6e 66 6f 22 2c 22 69 73  74 69 6f 2d 74 6f 6b 65  |nfo","istio-toke|
000004e0  6e 22 2c 22 69 73 74 69  6f 64 2d 63 61 2d 63 65  |n","istiod-ca-ce|
000004f0  72 74 22 5d 2c 22 69 6d  61 67 65 50 75 6c 6c 53  |rt"],"imagePullS|
00000500  65 63 72 65 74 73 22 3a  6e 75 6c 6c 2c 22 72 65  |ecrets":null,"re|
00000510  76 69 73 69 6f 6e 22 3a  22 64 65 66 61 75 6c 74  |vision":"default|
00000520  22 7d 6a 55 0a 0a 52 65  70 6c 69 63 61 53 65 74  |"}jU..ReplicaSet|
00000530  1a 14 6d 79 61 70 70 2d  77 65 62 2d 36 64 63 39  |..myapp-web-6dc9|
00000540  38 37 63 66 62 38 22 24  61 38 61 64 35 64 34 33  |87cfb8"$a8ad5d43|
00000550  2d 65 38 37 39 2d 34 63  39 32 2d 62 31 38 61 2d  |-e879-4c92-b18a-|
00000560  64 64 32 30 38 64 33 33  30 34 64 61 2a 07 61 70  |dd208d3304da*.ap|
00000570  70 73 2f 76 31 30 01 38  01 8a 01 ae 12 0a 17 6b  |ps/v10.8.......k|
00000580  75 62 65 2d 63 6f 6e 74  72 6f 6c 6c 65 72 2d 6d  |ube-controller-m|
00000590  61 6e 61 67 65 72 12 06  55 70 64 61 74 65 1a 02  |anager..Update..|
000005a0  76 31 22 08 08 8f fe dd  ae 06 10 00 32 08 46 69  |v1".........2.Fi|
000005b0  65 6c 64 73 56 31 3a f0  11 0a ed 11 7b 22 66 3a  |eldsV1:.....{"f:|
000005c0  6d 65 74 61 64 61 74 61  22 3a 7b 22 66 3a 61 6e  |metadata":{"f:an|
000005d0  6e 6f 74 61 74 69 6f 6e  73 22 3a 7b 22 2e 22 3a  |notations":{".":|
000005e0  7b 7d 2c 22 66 3a 70 72  6f 78 79 2e 69 73 74 69  |{},"f:proxy.isti|
000005f0  6f 2e 69 6f 2f 63 6f 6e  66 69 67 22 3a 7b 7d 7d  |o.io/config":{}}|
00000600  2c 22 66 3a 67 65 6e 65  72 61 74 65 4e 61 6d 65  |,"f:generateName|
00000610  22 3a 7b 7d 2c 22 66 3a  6c 61 62 65 6c 73 22 3a  |":{},"f:labels":|
00000620  7b 22 2e 22 3a 7b 7d 2c  22 66 3a 61 70 70 22 3a  |{".":{},"f:app":|
00000630  7b 7d 2c 22 66 3a 61 70  70 2e 6b 75 62 65 72 6e  |{},"f:app.kubern|
00000640  65 74 65 73 2e 69 6f 2f  63 6f 6d 70 6f 6e 65 6e  |etes.io/componen|
00000650  74 22 3a 7b 7d 2c 22 66  3a 61 70 70 2e 6b 75 62  |t":{},"f:app.kub|
00000660  65 72 6e 65 74 65 73 2e  69 6f 2f 69 6e 73 74 61  |ernetes.io/insta|
00000670  6e 63 65 22 3a 7b 7d 2c  22 66 3a 61 70 70 2e 6b  |nce":{},"f:app.k|
00000680  75 62 65 72 6e 65 74 65  73 2e 69 6f 2f 6d 61 6e  |ubernetes.io/man|
00000690  61 67 65 64 2d 62 79 22  3a 7b 7d 2c 22 66 3a 61  |aged-by":{},"f:a|
000006a0  70 70 2e 6b 75 62 65 72  6e 65 74 65 73 2e 69 6f  |pp.kubernetes.io|
000006b0  2f 6e 61 6d 65 22 3a 7b  7d 2c 22 66 3a 61 70 70  |/name":{},"f:app|
000006c0  2e 6b 75 62 65 72 6e 65  74 65 73 2e 69 6f 2f 70  |.kubernetes.io/p|
000006d0  61 72 74 2d 6f 66 22 3a  7b 7d 2c 22 66 3a 61 70  |art-of":{},"f:ap|
000006e0  70 2e 6b 75 62 65 72 6e  65 74 65 73 2e 69 6f 2f  |p.kubernetes.io/|
000006f0  76 65 72 73 69 6f 6e 22  3a 7b 7d 2c 22 66 3a 68  |version":{},"f:h|
00000700  65 6c 6d 2e 73 68 2f 63  68 61 72 74 22 3a 7b 7d  |elm.sh/chart":{}|
00000710  2c 22 66 3a 70 6f 64 2d  74 65 6d 70 6c 61 74 65  |,"f:pod-template|
00000720  2d 68 61 73 68 22 3a 7b  7d 2c 22 66 3a 76 65 72  |-hash":{},"f:ver|
00000730  73 69 6f 6e 22 3a 7b 7d  7d 2c 22 66 3a 6f 77 6e  |sion":{}},"f:own|
00000740  65 72 52 65 66 65 72 65  6e 63 65 73 22 3a 7b 22  |erReferences":{"|
00000750  2e 22 3a 7b 7d 2c 22 6b  3a 7b 5c 22 75 69 64 5c  |.":{},"k:{\"uid\|
00000760  22 3a 5c 22 61 38 61 64  35 64 34 33 2d 65 38 37  |":\"a8ad5d43-e87|
00000770  39 2d 34 63 39 32 2d 62  31 38 61 2d 64 64 32 30  |9-4c92-b18a-dd20|
00000780  38 64 33 33 30 34 64 61  5c 22 7d 22 3a 7b 7d 7d  |8d3304da\"}":{}}|
00000790  7d 2c 22 66 3a 73 70 65  63 22 3a 7b 22 66 3a 63  |},"f:spec":{"f:c|
000007a0  6f 6e 74 61 69 6e 65 72  73 22 3a 7b 22 6b 3a 7b  |ontainers":{"k:{|
000007b0  5c 22 6e 61 6d 65 5c 22  3a 5c 22 77 65 62 5c 22  |\"name\":\"web\"|
000007c0  7d 22 3a 7b 22 2e 22 3a  7b 7d 2c 22 66 3a 61 72  |}":{".":{},"f:ar|
000007d0  67 73 22 3a 7b 7d 2c 22  66 3a 65 6e 76 46 72 6f  |gs":{},"f:envFro|
000007e0  6d 22 3a 7b 7d 2c 22 66  3a 69 6d 61 67 65 22 3a  |m":{},"f:image":|
000007f0  7b 7d 2c 22 66 3a 69 6d  61 67 65 50 75 6c 6c 50  |{},"f:imagePullP|
00000800  6f 6c 69 63 79 22 3a 7b  7d 2c 22 66 3a 6c 69 76  |olicy":{},"f:liv|
00000810  65 6e 65 73 73 50 72 6f  62 65 22 3a 7b [truncated 52558 chars]
2024-02-22T17:11:18.947794Z     info    klog    curl -v -XPOST  -H "User-Agent: istioctl/1.20.3" -H "X-Stream-Protocol-Version: portforward.k8s.io" 'https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx/portforward'
2024-02-22T17:11:18.948992Z     info    klog    HTTP Trace: DNS Lookup for REDACTED.gr7.us-east-2.eks.amazonaws.com resolved to [{10.34.7.111 } {10.34.105.167 }]
2024-02-22T17:11:19.048088Z     info    klog    HTTP Trace: Dial to tcp:10.34.7.111:443 succeed
2024-02-22T17:11:19.300318Z     info    klog    POST https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx/portforward 101 Switching Protocols in 352 milliseconds
2024-02-22T17:11:19.300425Z     info    klog    HTTP Statistics: DNSLookup 1 ms Dial 99 ms TLSHandshake 0 ms Duration 352 ms
2024-02-22T17:11:19.300454Z     info    klog    Response Headers:
2024-02-22T17:11:19.300490Z     info    klog        Upgrade: SPDY/3.1
2024-02-22T17:11:19.300516Z     info    klog        X-Stream-Protocol-Version: portforward.k8s.io
2024-02-22T17:11:19.300539Z     info    klog        Connection: Upgrade
2024-02-22T17:11:19.563738Z     error   klog    an error occurred forwarding 60301 -> 15000: error forwarding port 15000 to pod 37e6dae5a6d46df325420609ffd9cfddef0ef20377ce69082961f34578d4bf5f, uid : failed to execute portforward in network namespace "/var/run/netns/cni-ef0013a0-a3fd-d141-8058-905f556ca236": failed to connect to localhost:15000 inside namespace "37e6dae5a6d46df325420609ffd9cfddef0ef20377ce69082961f34578d4bf5f", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
2024-02-22T17:11:19.564620Z     error   port forward failed: lost connection to pod
Error: failed to execute command on myapp-web-6dc987cfb8-vw5nx.myapp sidecar: failure running port forward process: Get "http://localhost:60301/config_dump": EOF

/etc/hosts:

app@myapp-web-5f5b6c695f-n97jz:~$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
2600:1f16:40:5306:1f57::6       myapp-web-5f5b6c695f-n97jz

cc @codeverifier

Version

Istio 1.20.3
AWS EKS 1.29 IPv6-enabled

Additional Information

I did find https://github.com/istio/istio/issues/34358 but EKS is currently running containerd 1.7.11 and I added the /etc/hosts above which appears to already have localhost properly mapped.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 23 (11 by maintainers)

Most upvoted comments

I know that port fowarding is not generally broken. It works for things like istioctl dashboard kiali as well as generic kubectl port-forward commands. The issue is something specific to what istioctl proxy-config is doing.

FWIW dashbord and proxy-config share the same port-forward logic in this case, may need to investigate why only proxy-config command is broken.