linkerd2: Fail to inject when mTLS is enabled and there is no service account

Bug Report

What is the issue?

When linkerd injects it’s proxy into a pod where service account automount is disabled, the pod fails to start up And goes into a restart loop. The root cause is the linkerd-proxy looking for the mounted serviceaccount info but not finding it.

How can it be reproduced?

Try to instrument a pod that has the following attribute set in it’s spec:

automountServiceAccountToken: false

I encountered this issue with the official argocd manifest: https://argoproj.github.io/argo-cd/getting_started/#1-install-argo-cd

Logs, error output, etc

On pod startup the linkerd-proxy container outputs this:

time="2019-05-23T15:47:08Z" level=info msg="running version dev-undefined"
time="2019-05-23T15:47:08Z" level=info msg="Using with pre-existing key: /var/run/linkerd/identity/end-entity/key.p8"
time="2019-05-23T15:47:08Z" level=info msg="Using with pre-existing CSR: /var/run/linkerd/identity/end-entity/key.p8"
ERR! [     0.000310s] linkerd2_proxy::app::config Could not read LINKERD2_PROXY_IDENTITY_TOKEN_FILE: No such file or directory (os error 2)
ERR! [     0.000354s] linkerd2_proxy::app::config LINKERD2_PROXY_IDENTITY_TOKEN_FILE="/var/run/secrets/kubernetes.io/serviceaccount/token" is not valid: InvalidTokenSource
configuration error: InvalidEnvVar

`linkerd check` output

➜ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ no invalid service profiles

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

Status check results are √

Environment

Kubernetes Version:

➜ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:26:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Cluster Environment: minikube in HyperKit
Host OS: MacOS
Linkerd version:

➜ linkerd version
Client version: stable-2.3.0
Server version: stable-2.3.0

Possible solution

Allow the mTLS to work without a service-account.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 17 (7 by maintainers)

Most upvoted comments

I just posted this in the linkerd slack channel, but here is a config that will allow linkerd 2.6 to inject with elastic-cloud-on-k8s, in case this helps anyone else seeing this issue.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.5.2
  nodeSets:
  - name: default
    count: 1
    podTemplate:
      metadata:
        annotations:
          linkerd.io/inject: "enabled"
      spec:
        automountServiceAccountToken: true

kellybirr on Jan 29, 2020

Linkerd sidecar injector should inject an audience bound token into the linkerd proxy container. This would provide better security (token bound to identity service would not be replayable against the Kubernetes API, and vice versa) and would also remove linkerd’s dependency on automountServiceAccountToken. Doc here:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

And more details on the differences to legacy tokens here:

https://github.com/kubernetes/kubernetes/issues/70679#issue-377670415

mikedanese on Aug 2, 2019

btw, if there the Linkerd proxy is outputting errors in its logs, the linkerd check --proxy command should fail. What does the command output on your end?

I think there is a trade-off here between the amount of auto-mutation we want to introduce into the proxy injection process and the user’s experience. Like you said, there are people who explicitly don’t want to mount the SA token for security reasons. Personally, I will have problem with some intermediary flipping on a security flag (in this case, automountServiceAccountToken) that I explicitly disabled. That same goes for automatically disabling the Linkerd mTLS. My preference is if a new workload that I’m about to deploy doesn’t comply with my cluster security settings, I will want the deployment to fail loudly.

ihcsim on May 24, 2019