istio: istioctl proxy-config does not work on EKS IPv6 clusters
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
The important detail here I believe is that I’m using AWS EKS 1.29 IPv6 with their weird IPv4 link-local NAT egress.
Related analysis from @howardjohn (thanks!) https://github.com/istio/istio/issues/49476#issuecomment-1959864574.
I am able to shell into a pod and run curl http://localhost:15000/config_dump however all istioctl pc commands fail.
❯ istioctl pc l myapp-web-5f5b6c695f-n97jz
2024-02-22T17:20:23.339406Z error klog an error occurred forwarding 60619 -> 15000: error forwarding port 15000 to pod 23c62f702462c62a8fd4ee632038302b4bc7e4a235264a3033d806bb51d26a41, uid : failed to execute portforward in network namespace "/var/run/netns/cni-b8d29c04-fabf-ba84-4e72-5bb792cbd579": failed to connect to localhost:15000 inside namespace "23c62f702462c62a8fd4ee632038302b4bc7e4a235264a3033d806bb51d26a41", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable
address found
2024-02-22T17:20:23.341278Z error port forward failed: lost connection to pod
Error: failed to execute command on myapp-web-5f5b6c695f-n97jz.myapp sidecar: failure running port forward process: Get "http://localhost:60619/config_dump": EOF
With logging enabled:
❯ istioctl pc l myapp-web-6dc987cfb8-vw5nx --vklog 9
2024-02-22T17:11:18.019274Z info klog curl -v -XGET -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: istioctl/1.20.3" 'https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx'
2024-02-22T17:11:18.515508Z info klog HTTP Trace: DNS Lookup for REDACTED.gr7.us-east-2.eks.amazonaws.com resolved to [{10.34.7.111 } {10.34.105.167 }]
2024-02-22T17:11:18.618212Z info klog HTTP Trace: Dial to tcp:10.34.7.111:443 succeed
2024-02-22T17:11:18.870643Z info klog GET https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx 200 OK in 851 milliseconds
2024-02-22T17:11:18.870997Z info klog HTTP Statistics: DNSLookup 75 ms Dial 101 ms TLSHandshake 115 ms ServerProcessing 132 ms Duration 851 ms
2024-02-22T17:11:18.871016Z info klog Response Headers:
2024-02-22T17:11:18.871034Z info klog Content-Type: application/vnd.kubernetes.protobuf
2024-02-22T17:11:18.871070Z info klog X-Kubernetes-Pf-Flowschema-Uid: 48115a1f-2c46-4eb8-8350-bab2930df0cf
2024-02-22T17:11:18.871084Z info klog X-Kubernetes-Pf-Prioritylevel-Uid: 329063b9-b080-4896-962f-c405f977c3d0
2024-02-22T17:11:18.871095Z info klog Date: Thu, 22 Feb 2024 17:11:18 GMT
2024-02-22T17:11:18.871120Z info klog Audit-Id: ddfce54e-83f4-4e55-a896-eb900d4df924
2024-02-22T17:11:18.871153Z info klog Cache-Control: no-cache, private
2024-02-22T17:11:18.941864Z debug klog Response Body:
00000000 6b 38 73 00 0a 09 0a 02 76 31 12 03 50 6f 64 12 |k8s.....v1..Pod.|
00000010 93 63 0a 90 23 0a 1a 6d 79 61 70 70 2d 77 65 62 |.c..#..myapp-web|
00000020 2d 36 64 63 39 38 37 63 66 62 38 2d 76 77 35 6e |-6dc987cfb8-vw5n|
00000030 78 12 15 6d 79 61 70 70 2d 77 65 62 2d 36 64 63 |x..myapp-web-6dc|
00000040 39 38 37 63 66 62 38 2d 1a 05 6d 79 61 70 70 22 |987cfb8-..myapp"|
00000050 00 2a 24 63 36 34 63 33 39 62 61 2d 39 30 65 32 |.*$c64c39ba-90e2|
00000060 2d 34 31 64 37 2d 62 30 33 35 2d 66 66 34 64 61 |-41d7-b035-ff4da|
00000070 34 33 32 64 39 64 63 32 08 38 30 37 30 31 38 37 |432d9dc2.8070187|
00000080 30 38 00 42 08 08 8f fe dd ae 06 10 00 5a 10 0a |08.B.........Z..|
00000090 03 61 70 70 12 09 6d 79 61 70 70 2d 77 65 62 5a |.app..myapp-webZ|
000000a0 22 0a 1b 61 70 70 2e 6b 75 62 65 72 6e 65 74 65 |"..app.kubernete|
000000b0 73 2e 69 6f 2f 63 6f 6d 70 6f 6e 65 6e 74 12 03 |s.io/component..|
000000c0 77 65 62 5a 23 0a 1a 61 70 70 2e 6b 75 62 65 72 |webZ#..app.kuber|
000000d0 6e 65 74 65 73 2e 69 6f 2f 69 6e 73 74 61 6e 63 |netes.io/instanc|
000000e0 65 12 05 6d 79 61 70 70 5a 24 0a 1c 61 70 70 2e |e..myappZ$..app.|
000000f0 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f 6d 61 |kubernetes.io/ma|
00000100 6e 61 67 65 64 2d 62 79 12 04 48 65 6c 6d 5a 23 |naged-by..HelmZ#|
00000110 0a 16 61 70 70 2e 6b 75 62 65 72 6e 65 74 65 73 |..app.kubernetes|
00000120 2e 69 6f 2f 6e 61 6d 65 12 09 6d 79 61 70 70 2d |.io/name..myapp-|
00000130 77 65 62 5a 2a 0a 19 61 70 70 2e 6b 75 62 65 72 |webZ*..app.kuber|
00000140 6e 65 74 65 73 2e 69 6f 2f 70 61 72 74 2d 6f 66 |netes.io/part-of|
00000150 12 0d 62 72 61 69 6e 62 61 73 65 6c 69 6e 65 5a |..brainbaselineZ|
00000160 45 0a 19 61 70 70 2e 6b 75 62 65 72 6e 65 74 65 |E..app.kubernete|
00000170 73 2e 69 6f 2f 76 65 72 73 69 6f 6e 12 28 64 30 |s.io/version.(d0|
00000180 34 61 34 34 62 36 37 63 64 66 30 61 37 64 61 30 |4a44b67cdf0a7da0|
00000190 61 65 65 63 64 64 38 63 36 38 30 61 31 65 35 62 |aeecdd8c680a1e5b|
000001a0 38 34 39 35 32 32 5a 25 0a 0d 68 65 6c 6d 2e 73 |849522Z%..helm.s|
000001b0 68 2f 63 68 61 72 74 12 14 62 72 61 69 6e 62 61 |h/chart..brainba|
000001c0 73 65 6c 69 6e 65 2d 30 2e 30 2e 35 39 5a 1f 0a |seline-0.0.59Z..|
000001d0 11 70 6f 64 2d 74 65 6d 70 6c 61 74 65 2d 68 61 |.pod-template-ha|
000001e0 73 68 12 0a 36 64 63 39 38 37 63 66 62 38 5a 22 |sh..6dc987cfb8Z"|
000001f0 0a 19 73 65 63 75 72 69 74 79 2e 69 73 74 69 6f |..security.istio|
00000200 2e 69 6f 2f 74 6c 73 4d 6f 64 65 12 05 69 73 74 |.io/tlsMode..ist|
00000210 69 6f 5a 2c 0a 1f 73 65 72 76 69 63 65 2e 69 73 |ioZ,..service.is|
00000220 74 69 6f 2e 69 6f 2f 63 61 6e 6f 6e 69 63 61 6c |tio.io/canonical|
00000230 2d 6e 61 6d 65 12 09 6d 79 61 70 70 2d 77 65 62 |-name..myapp-web|
00000240 5a 4f 0a 23 73 65 72 76 69 63 65 2e 69 73 74 69 |ZO.#service.isti|
00000250 6f 2e 69 6f 2f 63 61 6e 6f 6e 69 63 61 6c 2d 72 |o.io/canonical-r|
00000260 65 76 69 73 69 6f 6e 12 28 64 30 34 61 34 34 62 |evision.(d04a44b|
00000270 36 37 63 64 66 30 61 37 64 61 30 61 65 65 63 64 |67cdf0a7da0aeecd|
00000280 64 38 63 36 38 30 61 31 65 35 62 38 34 39 35 32 |d8c680a1e5b84952|
00000290 32 5a 33 0a 07 76 65 72 73 69 6f 6e 12 28 64 30 |2Z3..version.(d0|
000002a0 34 61 34 34 62 36 37 63 64 66 30 61 37 64 61 30 |4a44b67cdf0a7da0|
000002b0 61 65 65 63 64 64 38 63 36 38 30 61 31 65 35 62 |aeecdd8c680a1e5b|
000002c0 38 34 39 35 32 32 62 17 0a 0c 69 73 74 69 6f 2e |849522b...istio.|
000002d0 69 6f 2f 72 65 76 12 07 64 65 66 61 75 6c 74 62 |io/rev..defaultb|
000002e0 2e 0a 27 6b 75 62 65 63 74 6c 2e 6b 75 62 65 72 |..'kubectl.kuber|
000002f0 6e 65 74 65 73 2e 69 6f 2f 64 65 66 61 75 6c 74 |netes.io/default|
00000300 2d 63 6f 6e 74 61 69 6e 65 72 12 03 77 65 62 62 |-container..webb|
00000310 33 0a 2c 6b 75 62 65 63 74 6c 2e 6b 75 62 65 72 |3.,kubectl.kuber|
00000320 6e 65 74 65 73 2e 69 6f 2f 64 65 66 61 75 6c 74 |netes.io/default|
00000330 2d 6c 6f 67 73 2d 63 6f 6e 74 61 69 6e 65 72 12 |-logs-container.|
00000340 03 77 65 62 62 27 0a 12 70 72 6f 6d 65 74 68 65 |.webb'..promethe|
00000350 75 73 2e 69 6f 2f 70 61 74 68 12 11 2f 73 74 61 |us.io/path../sta|
00000360 74 73 2f 70 72 6f 6d 65 74 68 65 75 73 62 1b 0a |ts/prometheusb..|
00000370 12 70 72 6f 6d 65 74 68 65 75 73 2e 69 6f 2f 70 |.prometheus.io/p|
00000380 6f 72 74 12 05 31 35 30 32 30 62 1c 0a 14 70 72 |ort..15020b...pr|
00000390 6f 6d 65 74 68 65 75 73 2e 69 6f 2f 73 63 72 61 |ometheus.io/scra|
000003a0 70 65 12 04 74 72 75 65 62 5e 0a 15 70 72 6f 78 |pe..trueb^..prox|
000003b0 79 2e 69 73 74 69 6f 2e 69 6f 2f 63 6f 6e 66 69 |y.istio.io/confi|
000003c0 67 12 45 74 65 72 6d 69 6e 61 74 69 6f 6e 44 72 |g.EterminationDr|
000003d0 61 69 6e 44 75 72 61 74 69 6f 6e 3a 20 33 30 73 |ainDuration: 30s|
000003e0 0a 68 6f 6c 64 41 70 70 6c 69 63 61 74 69 6f 6e |.holdApplication|
000003f0 55 6e 74 69 6c 50 72 6f 78 79 53 74 61 72 74 73 |UntilProxyStarts|
00000400 3a 20 66 61 6c 73 65 0a 62 97 02 0a 17 73 69 64 |: false.b....sid|
00000410 65 63 61 72 2e 69 73 74 69 6f 2e 69 6f 2f 73 74 |ecar.istio.io/st|
00000420 61 74 75 73 12 fb 01 7b 22 69 6e 69 74 43 6f 6e |atus...{"initCon|
00000430 74 61 69 6e 65 72 73 22 3a 5b 22 69 73 74 69 6f |tainers":["istio|
00000440 2d 69 6e 69 74 22 2c 22 69 73 74 69 6f 2d 70 72 |-init","istio-pr|
00000450 6f 78 79 22 5d 2c 22 63 6f 6e 74 61 69 6e 65 72 |oxy"],"container|
00000460 73 22 3a 6e 75 6c 6c 2c 22 76 6f 6c 75 6d 65 73 |s":null,"volumes|
00000470 22 3a 5b 22 77 6f 72 6b 6c 6f 61 64 2d 73 6f 63 |":["workload-soc|
00000480 6b 65 74 22 2c 22 63 72 65 64 65 6e 74 69 61 6c |ket","credential|
00000490 2d 73 6f 63 6b 65 74 22 2c 22 77 6f 72 6b 6c 6f |-socket","worklo|
000004a0 61 64 2d 63 65 72 74 73 22 2c 22 69 73 74 69 6f |ad-certs","istio|
000004b0 2d 65 6e 76 6f 79 22 2c 22 69 73 74 69 6f 2d 64 |-envoy","istio-d|
000004c0 61 74 61 22 2c 22 69 73 74 69 6f 2d 70 6f 64 69 |ata","istio-podi|
000004d0 6e 66 6f 22 2c 22 69 73 74 69 6f 2d 74 6f 6b 65 |nfo","istio-toke|
000004e0 6e 22 2c 22 69 73 74 69 6f 64 2d 63 61 2d 63 65 |n","istiod-ca-ce|
000004f0 72 74 22 5d 2c 22 69 6d 61 67 65 50 75 6c 6c 53 |rt"],"imagePullS|
00000500 65 63 72 65 74 73 22 3a 6e 75 6c 6c 2c 22 72 65 |ecrets":null,"re|
00000510 76 69 73 69 6f 6e 22 3a 22 64 65 66 61 75 6c 74 |vision":"default|
00000520 22 7d 6a 55 0a 0a 52 65 70 6c 69 63 61 53 65 74 |"}jU..ReplicaSet|
00000530 1a 14 6d 79 61 70 70 2d 77 65 62 2d 36 64 63 39 |..myapp-web-6dc9|
00000540 38 37 63 66 62 38 22 24 61 38 61 64 35 64 34 33 |87cfb8"$a8ad5d43|
00000550 2d 65 38 37 39 2d 34 63 39 32 2d 62 31 38 61 2d |-e879-4c92-b18a-|
00000560 64 64 32 30 38 64 33 33 30 34 64 61 2a 07 61 70 |dd208d3304da*.ap|
00000570 70 73 2f 76 31 30 01 38 01 8a 01 ae 12 0a 17 6b |ps/v10.8.......k|
00000580 75 62 65 2d 63 6f 6e 74 72 6f 6c 6c 65 72 2d 6d |ube-controller-m|
00000590 61 6e 61 67 65 72 12 06 55 70 64 61 74 65 1a 02 |anager..Update..|
000005a0 76 31 22 08 08 8f fe dd ae 06 10 00 32 08 46 69 |v1".........2.Fi|
000005b0 65 6c 64 73 56 31 3a f0 11 0a ed 11 7b 22 66 3a |eldsV1:.....{"f:|
000005c0 6d 65 74 61 64 61 74 61 22 3a 7b 22 66 3a 61 6e |metadata":{"f:an|
000005d0 6e 6f 74 61 74 69 6f 6e 73 22 3a 7b 22 2e 22 3a |notations":{".":|
000005e0 7b 7d 2c 22 66 3a 70 72 6f 78 79 2e 69 73 74 69 |{},"f:proxy.isti|
000005f0 6f 2e 69 6f 2f 63 6f 6e 66 69 67 22 3a 7b 7d 7d |o.io/config":{}}|
00000600 2c 22 66 3a 67 65 6e 65 72 61 74 65 4e 61 6d 65 |,"f:generateName|
00000610 22 3a 7b 7d 2c 22 66 3a 6c 61 62 65 6c 73 22 3a |":{},"f:labels":|
00000620 7b 22 2e 22 3a 7b 7d 2c 22 66 3a 61 70 70 22 3a |{".":{},"f:app":|
00000630 7b 7d 2c 22 66 3a 61 70 70 2e 6b 75 62 65 72 6e |{},"f:app.kubern|
00000640 65 74 65 73 2e 69 6f 2f 63 6f 6d 70 6f 6e 65 6e |etes.io/componen|
00000650 74 22 3a 7b 7d 2c 22 66 3a 61 70 70 2e 6b 75 62 |t":{},"f:app.kub|
00000660 65 72 6e 65 74 65 73 2e 69 6f 2f 69 6e 73 74 61 |ernetes.io/insta|
00000670 6e 63 65 22 3a 7b 7d 2c 22 66 3a 61 70 70 2e 6b |nce":{},"f:app.k|
00000680 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f 6d 61 6e |ubernetes.io/man|
00000690 61 67 65 64 2d 62 79 22 3a 7b 7d 2c 22 66 3a 61 |aged-by":{},"f:a|
000006a0 70 70 2e 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f |pp.kubernetes.io|
000006b0 2f 6e 61 6d 65 22 3a 7b 7d 2c 22 66 3a 61 70 70 |/name":{},"f:app|
000006c0 2e 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f 70 |.kubernetes.io/p|
000006d0 61 72 74 2d 6f 66 22 3a 7b 7d 2c 22 66 3a 61 70 |art-of":{},"f:ap|
000006e0 70 2e 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f |p.kubernetes.io/|
000006f0 76 65 72 73 69 6f 6e 22 3a 7b 7d 2c 22 66 3a 68 |version":{},"f:h|
00000700 65 6c 6d 2e 73 68 2f 63 68 61 72 74 22 3a 7b 7d |elm.sh/chart":{}|
00000710 2c 22 66 3a 70 6f 64 2d 74 65 6d 70 6c 61 74 65 |,"f:pod-template|
00000720 2d 68 61 73 68 22 3a 7b 7d 2c 22 66 3a 76 65 72 |-hash":{},"f:ver|
00000730 73 69 6f 6e 22 3a 7b 7d 7d 2c 22 66 3a 6f 77 6e |sion":{}},"f:own|
00000740 65 72 52 65 66 65 72 65 6e 63 65 73 22 3a 7b 22 |erReferences":{"|
00000750 2e 22 3a 7b 7d 2c 22 6b 3a 7b 5c 22 75 69 64 5c |.":{},"k:{\"uid\|
00000760 22 3a 5c 22 61 38 61 64 35 64 34 33 2d 65 38 37 |":\"a8ad5d43-e87|
00000770 39 2d 34 63 39 32 2d 62 31 38 61 2d 64 64 32 30 |9-4c92-b18a-dd20|
00000780 38 64 33 33 30 34 64 61 5c 22 7d 22 3a 7b 7d 7d |8d3304da\"}":{}}|
00000790 7d 2c 22 66 3a 73 70 65 63 22 3a 7b 22 66 3a 63 |},"f:spec":{"f:c|
000007a0 6f 6e 74 61 69 6e 65 72 73 22 3a 7b 22 6b 3a 7b |ontainers":{"k:{|
000007b0 5c 22 6e 61 6d 65 5c 22 3a 5c 22 77 65 62 5c 22 |\"name\":\"web\"|
000007c0 7d 22 3a 7b 22 2e 22 3a 7b 7d 2c 22 66 3a 61 72 |}":{".":{},"f:ar|
000007d0 67 73 22 3a 7b 7d 2c 22 66 3a 65 6e 76 46 72 6f |gs":{},"f:envFro|
000007e0 6d 22 3a 7b 7d 2c 22 66 3a 69 6d 61 67 65 22 3a |m":{},"f:image":|
000007f0 7b 7d 2c 22 66 3a 69 6d 61 67 65 50 75 6c 6c 50 |{},"f:imagePullP|
00000800 6f 6c 69 63 79 22 3a 7b 7d 2c 22 66 3a 6c 69 76 |olicy":{},"f:liv|
00000810 65 6e 65 73 73 50 72 6f 62 65 22 3a 7b [truncated 52558 chars]
2024-02-22T17:11:18.947794Z info klog curl -v -XPOST -H "User-Agent: istioctl/1.20.3" -H "X-Stream-Protocol-Version: portforward.k8s.io" 'https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx/portforward'
2024-02-22T17:11:18.948992Z info klog HTTP Trace: DNS Lookup for REDACTED.gr7.us-east-2.eks.amazonaws.com resolved to [{10.34.7.111 } {10.34.105.167 }]
2024-02-22T17:11:19.048088Z info klog HTTP Trace: Dial to tcp:10.34.7.111:443 succeed
2024-02-22T17:11:19.300318Z info klog POST https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx/portforward 101 Switching Protocols in 352 milliseconds
2024-02-22T17:11:19.300425Z info klog HTTP Statistics: DNSLookup 1 ms Dial 99 ms TLSHandshake 0 ms Duration 352 ms
2024-02-22T17:11:19.300454Z info klog Response Headers:
2024-02-22T17:11:19.300490Z info klog Upgrade: SPDY/3.1
2024-02-22T17:11:19.300516Z info klog X-Stream-Protocol-Version: portforward.k8s.io
2024-02-22T17:11:19.300539Z info klog Connection: Upgrade
2024-02-22T17:11:19.563738Z error klog an error occurred forwarding 60301 -> 15000: error forwarding port 15000 to pod 37e6dae5a6d46df325420609ffd9cfddef0ef20377ce69082961f34578d4bf5f, uid : failed to execute portforward in network namespace "/var/run/netns/cni-ef0013a0-a3fd-d141-8058-905f556ca236": failed to connect to localhost:15000 inside namespace "37e6dae5a6d46df325420609ffd9cfddef0ef20377ce69082961f34578d4bf5f", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
2024-02-22T17:11:19.564620Z error port forward failed: lost connection to pod
Error: failed to execute command on myapp-web-6dc987cfb8-vw5nx.myapp sidecar: failure running port forward process: Get "http://localhost:60301/config_dump": EOF
/etc/hosts:
app@myapp-web-5f5b6c695f-n97jz:~$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
2600:1f16:40:5306:1f57::6 myapp-web-5f5b6c695f-n97jz
Version
Istio 1.20.3
AWS EKS 1.29 IPv6-enabled
Additional Information
I did find https://github.com/istio/istio/issues/34358 but EKS is currently running containerd 1.7.11 and I added the /etc/hosts above which appears to already have localhost properly mapped.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 23 (11 by maintainers)
FWIW
dashbordandproxy-configshare the same port-forward logic in this case, may need to investigate why onlyproxy-configcommand is broken.