rancher: Rancher RBAC stops Vault Kubernetes Auth from working

What kind of request is this (question/bug/enhancement/feature request): bug/enhancement

Steps to reproduce (least amount of steps as possible):

I am trying to use Hashicorp Vault Kubernetes Auth to get secrets into my Kubernetes workloads. It appears that the Rancher RBAC is getting in the way of the auth mechanism.

This is NOT the same as the kubernetes-vault Catalog App in Rancher. That one uses Vault’s AppRole authentication, this one uses Kubernetes Service accounts to provide authorization.

Steps:

Create a Kubernetes cluster (I used RKE and Minikube). Start a Vault server (you can use the development server to start quickly: )

After getting vault running, enable Kubernetes Auth.

vault auth enable -path=vaulttest2 kubernetes

From the kube_config_cluster.yml created by RKE, get the certificate-authority-data, and base64 decode it to get the CA cert, and save it as ca.crt. Also save take the server URL, and plug them into the command:

vault write auth/vaulttest2/config \
  kubernetes_host=https://<server-url> \
  kubernetes_ca_cert=@ca.crt

Create and apply a Kubernetes ClusterRoleBinding and Service account for the pods to use:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-tokenreview
  namespace: default

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-tokenreview

Create a secret in Vault:

# vault kv put applications/my-app/secrets foo=bar
Success! Data written to: applications/my-app/secrets

Create a Vault policy that has access to the secret:

policy.hcl

path "applications/my-app/secrets" {
    capabilities = ["read"]
}

Apply the policy:

vault write sys/policy/demo-policy policy=@policy.hcl

Create a Vault role, that has access to the policy:

vault write auth/vaulttest2/role/my-app \
  bound_service_account_names=vault-tokenreview \
  bound_service_account_namespaces=default \
  policies=my-app-auth \
  period=60s

Create a Kubernetes deployment that has the vault-tokenreview serviceaccount set:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: vault-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: vault-test
      serviceAccount: vault-tokenreview
      serviceAccountName: vault-tokenreview

Once the container starts, exec, into it, and run the commands:

KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl --insecure --request POST --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "my-app"}' http://<vault-host>:8200/v1/auth/vaulttest2/login
{"request_id":"d9d576fe-c5b3-29b6-7d6c-a6c717f93240","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.7eMiBpQL9zjX1RQyHhd3PCgx","accessor":"gChuNlhmmL9i1TY1HilxKTGL","policies":["default","kube-auth"],"token_policies":["default","kube-auth"],"metadata":{"role":"demo","service_account_name":"vault-tokenreview","service_account_namespace":"default","service_account_secret_name":"vault-tokenreview-token-s2fkd","service_account_uid":"9a853862-0fb3-11e9-a577-005056982aa7"},"lease_duration":60,"renewable":true,"entity_id":"d08c7fdd-c1d0-0684-645a-71564d57019c","token_type":"service"}}

You can then take this JSON response and take the client_token value to pull secrets from Vault.

This all works fine, except that Vault is going directly to the Kubernetes API, using the self-signed cert that RKE created. I have multiple clusters, and I want to be able to use the Rancher-to-Kubernetes API that has my internal CA-signed certificate.

If I change the Kubernetes Auth configuration to point to my Rancher cluster, using my signed cert:

vault write auth/vaulttest2/config \
  kubernetes_host=https://rancher.my-company.com/k8s/clusters/c-ct5fq \
  kubernetes_ca_cert=@my-cert-chain.crt

Now, when I run the curl command in the container, I get the response:

{"errors":["lookup failed: service account unauthorized; this could mean it has been deleted"]}

I presume that this is because the Rancher API is intercepting the requests in order to do RBAC, and it does not know about my vault-tokenreview service account, so it blocks the access.

How can I get Vault Kubernetes Auth to work using the Rancher API?

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
• Installation option (single install/HA): Rancher 2.1.5, Kubernetes 1.12.4

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Imported
• Machine type (cloud/VM/metal) and specifications (CPU/memory): VM