rancher: Vault Sidecar Injector error with k8s installed by rancher 2.5

Got a toy Vault sidecar injector working - here’s what it took for an RKE cluster launched through the Rancher server.

Create and configure the cluster

When creating the RKE-based cluster in Rancher (Custom or node driver), ensure the following:

• Authorized Cluster Endpoint (ACE) is enabled
• If an FQDN is supplied for ACE, rancher_kubernetes_engine_config.authentication.sans must be an array containing the FQDN. In my case, it looks like the following:

  rancher_kubernetes_engine_config:
    # ... other keys ...
    authentication:
      sans:
        - nikkelma-vault-ds.fe.rancher.space
    strategy: x509|webhook
    # ... continued config ...
  

Optional: create a development Vault server

I used a development Vault server at nikkelma-vault.fe.rancher.space when doing my testing. I ran the following:

vault server -dev -dev-listen-address=0.0.0.0:8200

Configure Vault for Kubernetes, install Injector

I developed the following script to configure Vault, install the Vault Injector, and deploy a test application to confirm correct behavior.

NOTES:

• My KUBE_HOST uses port 6443 because my ACE FQDN was simply a DNS entry pointing to my cluster’s control plane. Use the correct port for your configuration, possibly leaving off the port if using the HTTPS default of 443.
• Change VAULT_ADDR to point to your Vault cluster - I used a local development server here.

# change to point to your kube config, or comment to use default kubectl behavior
export KUBECONFIG="$HOME/kube_config.yaml"

# vault downstream test cluster
# NOTE: I used port 6443 because I simply pointed a DNS record to the cluster's control plane
export KUBE_HOST="https://nikkelma-vault-ds.fe.rancher.space:6443"

export EXTERNAL_VAULT_ADDRESS='http://nikkelma-vault.fe.rancher.space:8200'

# configure vault CLI to talk to vault on local host
export VAULT_ADDR='http://127.0.0.1:8200'

mkdir tmp

# create target namespace
kubectl create ns vault-test-a

# create service account YAML - see below for file contents
echo -n 'LS0tCmFwaVZlcnNpb246IHYxCmtpbmQ6IFNlcnZpY2VBY2NvdW50Cm1ldGFkYXRhOgogIG5hbWU6IGludGVybmFsLWFwcAogIG5hbWVzcGFjZTogdmF1bHQtdGVzdC1hCi0tLQphcGlWZXJzaW9uOiB2MQpraW5kOiBTZXJ2aWNlQWNjb3VudAptZXRhZGF0YToKICBuYW1lOiB2YXVsdC1hdXRoCiAgbmFtZXNwYWNlOiB2YXVsdC10ZXN0LWEKLS0tCmFwaVZlcnNpb246IHYxCmtpbmQ6IFNlY3JldAptZXRhZGF0YToKICBuYW1lOiB2YXVsdC1hdXRoCiAgbmFtZXNwYWNlOiB2YXVsdC10ZXN0LWEKICBhbm5vdGF0aW9uczoKICAgIGt1YmVybmV0ZXMuaW8vc2VydmljZS1hY2NvdW50Lm5hbWU6IHZhdWx0LWF1dGgKdHlwZToga3ViZXJuZXRlcy5pby9zZXJ2aWNlLWFjY291bnQtdG9rZW4KLS0tCmFwaVZlcnNpb246IHJiYWMuYXV0aG9yaXphdGlvbi5rOHMuaW8vdjEKa2luZDogQ2x1c3RlclJvbGVCaW5kaW5nCm1ldGFkYXRhOgogIG5hbWU6IHJvbGUtdG9rZW5yZXZpZXctYmluZGluZwpyb2xlUmVmOgogIGFwaUdyb3VwOiByYmFjLmF1dGhvcml6YXRpb24uazhzLmlvCiAga2luZDogQ2x1c3RlclJvbGUKICBuYW1lOiBzeXN0ZW06YXV0aC1kZWxlZ2F0b3IKc3ViamVjdHM6CiAgLSBraW5kOiBTZXJ2aWNlQWNjb3VudAogICAgbmFtZTogdmF1bHQtYXV0aAogICAgbmFtZXNwYWNlOiB2YXVsdC10ZXN0LWEK' \
  | base64 -d > tmp/sa.yaml

kubectl apply -f tmp/sa.yaml

# enable kubernetes authentication, under a non-default path to match the source issue
vault auth enable -path="rndvault" kubernetes

# sleep to ensure the secret created in sa.yaml is populated with a valid token
sleep 10

export TOKEN_REVIEW_JWT="$(kubectl get secret vault-auth -n vault-test-a -o go-template='{{ .data.token }}' | base64 --decode)"

# save the cluster's CA cert to a file to avoid any issues with environment variables
kubectl get secret vault-auth -n vault-test-a -o go-template='{{ index .data "ca.crt" }}' | base64 --decode > tmp/vault-ca.crt

# configure Vault's kubernetes authentication to use the newly-created service account
vault write auth/rndvault/config \
         token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
         kubernetes_host="$KUBE_HOST" \
         kubernetes_ca_cert=@tmp/vault-ca.crt

# create the policy file used for testing injection - see below for file contents
echo -n 'IyBFbmFibGUgYW5kIG1hbmFnZSB0aGUga2V5L3ZhbHVlIHNlY3JldHMgZW5naW5lIGF0IGBzZWNyZXQvYCBwYXRoCnBhdGggInNlY3JldC8qIgp7CiAgY2FwYWJpbGl0aWVzID0gWyJyZWFkIl0KfQo=' \
  | base64 -d > tmp/internal-app-policy.hcl

# create app policy in Vault
vault policy write internal-app-pol tmp/internal-app-policy.hcl

# grant the app's service account permissions to get the secrets it wants
# vault-app will be the role referenced in the test deployment
vault write auth/rndvault/role/vault-app \
         bound_service_account_names=internal-app \
         bound_service_account_namespaces=vault-test-a \
         policies=internal-app-pol \
         ttl=24h

# install the Vault sidecar injector
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# NOTE: the Vault server is NOT installed in this package because injector.externalVaultAddr is non-empty
helm upgrade --install vault hashicorp/vault \
  -n vault-test-a \
  --version 0.9.1 \
  --set "injector.externalVaultAddr=${EXTERNAL_VAULT_ADDRESS}" \
  --set "injector.authPath=auth/rndvault"\
  --set "server.serviceAccount.name=vault-auth" \
  --set "server.serviceAccount.create=false" \
  --wait

# create a test secret for the deployment
vault kv put secret/hello foo=world

# create the deployment file used to test injection - see below for file contents
echo -n 'YXBpVmVyc2lvbjogYXBwcy92MQpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIGxhYmVsczoKICAgIGFwcDogbmdpbngKICBuYW1lOiBuZ2lueApzcGVjOgogIHJlcGxpY2FzOiAxCiAgc2VsZWN0b3I6CiAgICBtYXRjaExhYmVsczoKICAgICAgYXBwOiBuZ2lueAogIHN0cmF0ZWd5OiB7fQogIHRlbXBsYXRlOgogICAgbWV0YWRhdGE6CiAgICAgIGFubm90YXRpb25zOgogICAgICAgIHZhdWx0Lmhhc2hpY29ycC5jb20vYWdlbnQtaW5qZWN0OiAidHJ1ZSIKICAgICAgICB2YXVsdC5oYXNoaWNvcnAuY29tL2FnZW50LWluamVjdC1zZWNyZXQtaGVsbG86ICJzZWNyZXQvaGVsbG8iCiAgICAgICAgdmF1bHQuaGFzaGljb3JwLmNvbS9yb2xlOiAidmF1bHQtYXBwIgogICAgICBsYWJlbHM6CiAgICAgICAgYXBwOiBuZ2lueAogICAgc3BlYzoKICAgICAgc2VydmljZUFjY291bnROYW1lOiBpbnRlcm5hbC1hcHAKICAgICAgY29udGFpbmVyczoKICAgICAgLSBpbWFnZTogbmdpbng6MS4xOAogICAgICAgIG5hbWU6IG5naW54CiAgICAgICAgcmVzb3VyY2VzOiB7fQo=' \
  | base64 -d > tmp/deploy.yaml

# install test deployment
kubectl apply -f tmp/deploy.yaml -n vault-test-a

rm -rf ./tmp

File contents reference

sa.yaml: NOTE: edited from source issue to grant only vault-auth service account access the system:auth-delegator role

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
  namespace: vault-test-a
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: vault-test-a
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  namespace: vault-test-a
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault-test-a

internal-app-policy.hcl:

# Enable and manage the key/value secrets engine at `secret/` path
path "secret/*"
{
  capabilities = ["read"]
}

deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-hello: "secret/hello"
        vault.hashicorp.com/role: "vault-app"
      labels:
        app: nginx
    spec:
      serviceAccountName: internal-app
      containers:
      - image: nginx:1.18
        name: nginx
        resources: {}

@bienkma the difference is the activation of Authorized Cluster Endpoint (ACE) to be able to use internal Kube JWT token instead of rancher tokens.

using kubespray is not an option in my case since we’re using vsphere to automatically provide de cluster. got some workarounds going on, if any fits i will post it here. thanks