istio: Failed to get secret for proxy

Hi,

I have an Istio 1.5 installation on AWS EKS (Kubernetes 1.15) Istio was installed with istioctl and Demo profile, and a modified manifest to enable Istio to work with AWS NLB. Everything worked fine up until last week Friday I tried to manually scale the istio-ingressgateway, which basically did not work. Raising the replicaset did not actually raise the number of pods. However, since then, in k8s dashboard, I can see two ingress-gateways, one, the original replica, up and running since numerous days. The second one constantly recreating. With some efforts, I was finally able to extract container logs (see below) from the node the terminating pods were assigned to

Bug description From the container logs it seems the Istio proxy is terminating the connection , but I don’t know what the underlying reason may be and how to get it fixed.

[ ] Docs [ ] Installation [X ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

How was Istio installed?

Environment where bug was observed (cloud vendor, OS, etc)

{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:35.656][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream\n","stream":"stderr","time":"2020-08-12T07:14:35.656390386Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:35.656][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream\n","stream":"stderr","time":"2020-08-12T07:14:35.656419936Z"}
{"log":"2020-08-12T07:14:35.664305Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-1 resource:default new connection\n","stream":"stdout","time":"2020-08-12T07:14:35.675832022Z"}
{"log":"2020-08-12T07:14:35.742597Z\u0009error\u0009citadelclient\u0009Failed to create certificate: rpc error: code = Unauthenticated desc= request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.74271502Z"}
{"log":"2020-08-12T07:14:35.742636Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-1 resource:default request:b40e778b-2f66-44c6-b8ee-60e52d86e8d9 CSR hit non-retryable error (HTTP code: 0). Error: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.742772223Z"}
{"log":"2020-08-12T07:14:35.742658Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-1 resource:default failed to generate secret for proxy: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.742779142Z"}
{"log":"2020-08-12T07:14:35.742718Z\u0009error\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-1 resource:default Close connection. Failed to get secret for proxy \"router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local\" from secret cache: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.742788059Z"}
{"log":"2020-08-12T07:14:35.742830Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-1 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled\n","stream":"stdout","time":"2020-08-12T07:14:35.742905078Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:35.742][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 16, request authenticate failure\n","stream":"stderr","time":"2020-08-12T07:14:35.743027671Z"}
{"log":"2020-08-12T07:14:35.874955Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-2 resource:default new connection\n","stream":"stdout","time":"2020-08-12T07:14:35.883571973Z"}
{"log":"2020-08-12T07:14:35.989052Z\u0009error\u0009citadelclient\u0009Failed to create certificate: rpc error: code = Unauthenticated desc= request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.989188639Z"}
{"log":"2020-08-12T07:14:35.989084Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-2 resource:default request:a7f2dbb1-2389-4b18-a74a-bff5b8885daa CSR hit non-retryable error (HTTP code: 0). Error: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.989214107Z"}
{"log":"2020-08-12T07:14:35.989247Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-2 resource:default failed to generate secret for proxy: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.989293983Z"}
{"log":"2020-08-12T07:14:35.989323Z\u0009error\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-2 resource:default Close connection. Failed to get secret for proxy \"router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local\" from secret cache: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:35.989356058Z"}
{"log":"2020-08-12T07:14:35.989396Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-2 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled\n","stream":"stdout","time":"2020-08-12T07:14:35.989453882Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:35.989][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 16, request authenticate failure\n","stream":"stderr","time":"2020-08-12T07:14:35.989593061Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:36.363][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream\n","stream":"stderr","time":"2020-08-12T07:14:36.363698818Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:36.363][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream\n","stream":"stderr","time":"2020-08-12T07:14:36.363757681Z"}
{"log":"2020-08-12T07:14:37.020048Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-3 resource:default new connection\n","stream":"stdout","time":"2020-08-12T07:14:37.020227224Z"}
{"log":"2020-08-12T07:14:37.170853Z\u0009error\u0009citadelclient\u0009Failed to create certificate: rpc error: code = Unauthenticated desc= request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:37.170952319Z"}
{"log":"2020-08-12T07:14:37.170877Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-3 resource:default request:a34ba318-50ca-4567-81cc-c640c24a4ea6 CSR hit non-retryable error (HTTP code: 0). Error: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:37.170983465Z"}
{"log":"2020-08-12T07:14:37.170906Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-3 resource:default failed to generate secret for proxy: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:37.171023334Z"}
{"log":"2020-08-12T07:14:37.170916Z\u0009error\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-3 resource:default Close connection. Failed to get secret for proxy \"router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local\" from secret cache: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:37.171029035Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:37.171][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 16, request authenticate failure\n","stream":"stderr","time":"2020-08-12T07:14:37.171127026Z"}
{"log":"2020-08-12T07:14:37.171144Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-3 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled\n","stream":"stdout","time":"2020-08-12T07:14:37.171189359Z"}
{"log":"2020-08-12T07:14:38.592463Z\u0009info\u0009watchFileEvents has successfully terminated\n","stream":"stdout","time":"2020-08-12T07:14:38.592731506Z"}
{"log":"2020-08-12T07:14:38.592477Z\u0009info\u0009Watcher has successfully terminated\n","stream":"stdout","time":"2020-08-12T07:14:38.592772821Z"}
{"log":"2020-08-12T07:14:38.592464Z\u0009info\u0009Agent draining Proxy\n","stream":"stdout","time":"2020-08-12T07:14:38.592779464Z"}
{"log":"2020-08-12T07:14:38.592488Z\u0009info\u0009Status server has successfully terminated\n","stream":"stdout","time":"2020-08-12T07:14:38.592783903Z"}
{"log":"2020-08-12T07:14:38.592535Z\u0009error\u0009accept tcp [::]:15020: use of closed network connection\n","stream":"stdout","time":"2020-08-12T07:14:38.592787804Z"}
{"log":"2020-08-12T07:14:38.593243Z\u0009info\u0009Graceful termination period is 5s, starting...\n","stream":"stdout","time":"2020-08-12T07:14:38.593341783Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:39.180][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream\n","stream":"stderr","time":"2020-08-12T07:14:39.1809113Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:39.180][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream\n","stream":"stderr","time":"2020-08-12T07:14:39.180940339Z"}
{"log":"2020-08-12T07:14:40.520407Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-4 resource:default new connection\n","stream":"stdout","time":"2020-08-12T07:14:40.520615876Z"}
{"log":"2020-08-12T07:14:40.750383Z\u0009error\u0009citadelclient\u0009Failed to create certificate: rpc error: code = Unauthenticated desc= request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:40.750523315Z"}
{"log":"2020-08-12T07:14:40.750412Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-4 resource:default request:10a279c7-ef5e-4b38-90c0-9f968203d7e0 CSR hit non-retryable error (HTTP code: 0). Error: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:40.750567464Z"}
{"log":"2020-08-12T07:14:40.750450Z\u0009error\u0009cache\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-4 resource:default failed to generate secret for proxy: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:40.750574796Z"}
{"log":"2020-08-12T07:14:40.750480Z\u0009error\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-4 resource:default Close connection. Failed to get secret for proxy \"router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local\" from secret cache: rpc error: code = Unauthenticated desc = request authenticate failure\n","stream":"stdout","time":"2020-08-12T07:14:40.750579737Z"}
{"log":"2020-08-12T07:14:40.750551Z\u0009info\u0009sds\u0009node:router~10.225.201.102~istio-ingressgateway-bcd79ccb-cc8tw.istio-system~istio-system.svc.cluster.local-4 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled\n","stream":"stdout","time":"2020-08-12T07:14:40.750597752Z"}
{"log":"[Envoy (Epoch 0)] [2020-08-12 07:14:40.750][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 16, request authenticate failure\n","stream":"stderr","time":"2020-08-12T07:14:40.750805653Z"}

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (6 by maintainers)

Most upvoted comments

Update @djablonski-moia @amoskyler @hniehus

We got to make this work on Kops 1.19 by set only the following:

    apiAudiences:
    - api
    - istio-ca
    serviceAccountIssuer: kubernetes.default.svc

Let the rest (serviceAccountKeyFile + serviceAccountSigningKeyFile) for the Kops to handle ^^

+5
dntosas on Feb 2, 2021

I run into the same situation, but my configuration is slightly different, I’m using istio-csr+ cert-manager+ vault + istio, and I’m also run it on EKS with cert-manager-istio-csr v0.4.2 istio 1.14.1 istiod 2022-06-24T20:41:04.790540Z error klog grpc-server “msg”=“failed to authenticate request” “error”=“could not get cluster aws-eu-central1’s kube client” “serving-addr”=“0.0.0.0:6443”

istio-proxy 2022-06-24T20:41:11.075081Z warning envoy config StreamSecrets gRPC config stream closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unauthenticated desc = request authenticate failure

+2
jasonhao518 on Jun 24, 2022

For posterity, this seems to have been a kOps issue and recently changed in kOps 1.19 (https://github.com/kubernetes/kops/issues/8451#issuecomment-822131914):

one should not override either serviceAccountKeyFile or serviceAccountSigningKeyFile in kops 1.19+.

The root cause of that change (and why it was changed) is mentioned in this kOps PR: https://github.com/kubernetes/kops/pull/9534.

@dntosas, who made the last comment, had also made a PR (https://github.com/kubernetes/kops/pull/10712) to update the relevant kOps docs on this as well as a PR to the Istio docs (https://github.com/istio/istio.io/pull/8893) to update the kOps config section (on the same day as that comment no less!). The latter made it into the docs for Istio 1.9+, but Istio 1.6-1.8’s docs have the old config with serviceAccountKeyFile and serviceAccountSigningKeyFile, so if you’re wondering where you even got that config from since it’s not in the docs currently, it’s because it used to be in both the kOps and Istio docs.

+2
agilgur5 on Aug 19, 2021

@remidebette yes, I fix it by using a different solution, just extended istio samples certs a little bit using helm chart, in certs folder, call make to create intermediate certificate for each cluster make $CLUSTER-certs, then deploy the chart to each cluster under istio-system namespace, and providing helm parameter of the cluster, here I’m using “istiod.global.meshID”, and I automated this process using argocd. This is only a temporary solution, will come back to cert-manager-istio-csr when it works. helm chart

ca-cert.yaml {{- $certPath := printf “certs/%s/**” .Values.istiod.global.meshID -}} apiVersion: v1 kind: Secret metadata: name: cacerts labels: app: cacerts stringData: {{- (.Files.Glob $certPath ).AsConfig | nindent 2 }}

+1
jasonhao518 on Apr 13, 2023

@amoskyler We have the same problem with Kops 1.19.0-alpha.5 and Istio 1.7.3, but your proposed change leads to master nodes not starting up anymore in our setting ☹️

+1
djablonski-moia on Nov 10, 2020
Read more comments on GitHub
← istio: failed calling webhook "sidecar-injector.istio.io
istio: File path based Gateway TLS setting is not working when using SPIRE as workload identity issuer. →