istio: Istio proxy fais with JWT token missing JWT token

Bug description We are upgrading from istio 1.4.5 to 1.5.5 and the istio-proxy container keeps restarting with error:

2020-06-29T21:45:03.269986Z	info	JWT policy is third-party-jwt
2020-06-29T21:45:03.270042Z	warn	Missing JWT token, can't use in process SDS ./var/run/secrets/tokens/istio-tokenstat ./var/run/secrets/tokens/istio-token: no such file or directory
2020-06-29T21:45:03.270052Z	fatal	Missing JWT, can't authenticate with control plane. Try using plain text (15010)

We have set controlPlaneSecurityEnabled: false but seems this config is eliminated in istio 1.5. So there is no way to disable proxy to control plane secure communication? Tried to install with --set values.global.jwtPolicy=first-party-jwt but this doesn’t seem work, it still use third party token, And our cluster seems support third party jwt token since when I run kubectl get --raw /api/v1 it returns result contains: {"name":"serviceaccounts/token","singularName":"","namespaced":true,"group":"authentication.k8s.io","version":"v1","kind":"TokenRequest","verbs":["create"]} I do managed to set ‘first-party-jwt’ once and I saw from istio-proxy logs that it is using first party token and it works. but next time with same code, it will still use third party jwt, I’m confused here.

we have 1.4.5 running fine for a long time. I’m guessing that’s because we have controlPlaneSecurityEnabled: false and it works in 1.4.5

[x ] Configuration Infrastructure [ ] Docs [ ] Installation [x] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [x ] User Experience [ ] Developer Infrastructure

Expected behavior

Steps to reproduce the bug I can reproduce the bug by deploying istio 1.5.5 using istioctl manifest apply to the k8s cluster

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) istioctl version 1.5.5 kubectl version 1.5.10

How was Istio installed? we are using istioctl to install istio and istioctl verify install passed

Environment where bug was observed (cloud vendor, OS, etc)

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 17 (6 by maintainers)

Most upvoted comments

Closing as both issues here are addressed

OP: has incorrect inejctor config Others: have automountServiceAccountToken=false

howardjohn on Aug 24, 2020

I got the same issue in 1.6.7. fatal Missing JWT, can’t authenticate with control plane. Try using plain text (15010)

[root@K8S-Master ~]# k get po -n fewbox
NAME                      READY   STATUS             RESTARTS   AGE
demo-v1-8b88796cf-2jxx6   1/2     CrashLoopBackOff   20         78m
demo-v1-8b88796cf-n847x   1/2     CrashLoopBackOff   13         47m
demo-v1-8b88796cf-rlmw5   1/2     CrashLoopBackOff   20         78m

[root@K8S-Master ~]# k logs demo-v1-8b88796cf-2jxx6 -n fewbox istio-proxy 
2020-08-11T12:41:10.209748Z	info	FLAG: --concurrency="2"
2020-08-11T12:41:10.209784Z	info	FLAG: --disableInternalTelemetry="false"
2020-08-11T12:41:10.209789Z	info	FLAG: --domain="fewbox.svc.cluster.local"
2020-08-11T12:41:10.209792Z	info	FLAG: --help="false"
2020-08-11T12:41:10.209794Z	info	FLAG: --id=""
2020-08-11T12:41:10.209796Z	info	FLAG: --ip=""
2020-08-11T12:41:10.209799Z	info	FLAG: --log_as_json="false"
2020-08-11T12:41:10.209801Z	info	FLAG: --log_caller=""
2020-08-11T12:41:10.209803Z	info	FLAG: --log_output_level="default:info"
2020-08-11T12:41:10.209805Z	info	FLAG: --log_rotate=""
2020-08-11T12:41:10.209808Z	info	FLAG: --log_rotate_max_age="30"
2020-08-11T12:41:10.209810Z	info	FLAG: --log_rotate_max_backups="1000"
2020-08-11T12:41:10.209812Z	info	FLAG: --log_rotate_max_size="104857600"
2020-08-11T12:41:10.209815Z	info	FLAG: --log_stacktrace_level="default:none"
2020-08-11T12:41:10.209822Z	info	FLAG: --log_target="[stdout]"
2020-08-11T12:41:10.209825Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2020-08-11T12:41:10.209827Z	info	FLAG: --mixerIdentity=""
2020-08-11T12:41:10.209829Z	info	FLAG: --outlierLogPath=""
2020-08-11T12:41:10.209831Z	info	FLAG: --pilotIdentity=""
2020-08-11T12:41:10.209833Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2020-08-11T12:41:10.209836Z	info	FLAG: --proxyLogLevel="warning"
2020-08-11T12:41:10.209838Z	info	FLAG: --serviceCluster="demo.fewbox"
2020-08-11T12:41:10.209840Z	info	FLAG: --serviceregistry="Kubernetes"
2020-08-11T12:41:10.209842Z	info	FLAG: --stsPort="0"
2020-08-11T12:41:10.209845Z	info	FLAG: --templateFile=""
2020-08-11T12:41:10.209847Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2020-08-11T12:41:10.209850Z	info	FLAG: --trust-domain="cluster.local"
2020-08-11T12:41:10.209871Z	info	Version 1.6.7-2511ab8c8c59a203e77bb804846593c3690fcf4a-dirty-Modified
2020-08-11T12:41:10.210008Z	info	Obtained private IP [10.244.235.139]
2020-08-11T12:41:10.210088Z	info	Apply proxy config from env {"proxyMetadata":{"DNS_AGENT":""}}

2020-08-11T12:41:10.211512Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: demo.fewbox
statNameLength: 189
statusPort: 15020
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2020-08-11T12:41:10.211553Z	info	Proxy role: &model.Proxy{ClusterID:"", Type:"sidecar", IPAddresses:[]string{"10.244.235.139"}, ID:"demo-v1-8b88796cf-2jxx6.fewbox", Locality:(*envoy_api_v2_core.Locality)(nil), DNSDomain:"fewbox.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil)}
2020-08-11T12:41:10.211558Z	info	JWT policy is first-party-jwt
2020-08-11T12:41:10.211599Z	warn	Missing JWT token, can't use in process SDS /var/run/secrets/kubernetes.io/serviceaccount/tokenstat /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
2020-08-11T12:41:10.211603Z	fatal	Missing JWT, can't authenticate with control plane. Try using plain text (15010)

I resolve this issue after read some ref docs, but I don’t know if it is the right way. I used the “first-party-jwt” and add the property in deployment(template>spec>[Add automountServiceAccountToken: true]) then the pod is running now, I will find out whether it is work later.

template:
    metadata:
      annotations:
        app: demo
        brand: fewbox
        version: v1
      creationTimestamp: null
      labels:
        app: demo
        brand: fewbox
        version: v1
    spec:
      automountServiceAccountToken: true

FewBox on Aug 11, 2020