vault-k8s: vault-agent sidecar doesn't extend vault-agent-init's secret, but fetches new ones

Hello there!

Issue

I have a PostgresSQL database and I have set up vault with dynamic roles in order to dynamically generate database credentials for my application. By closely examining the logs I found a (potential?) bug:

• The vault-agent-init container asks the vault server for database secrets. This means that a new lease is created. Vault-agent-init writes the credentials to a file.
• Vault-agent and the application container start simultaneously.
• Vault-agent takes some time (8 seconds - more on this later) to fetch the (new!) secrets.
• In the meantime my application reads the secrets generated by the vault-agent-init and connects to the database. However as the vault-agent-init is completed, its lease is not renewed, hence the secrets picked up by the application are invalidated shortly.
• Once vault-agent initializes itself it fetches a new pair of secrets, writes these to the file.

Expected behavior: Shouldn’t the vault-agent keep using the vault-agent-init’s lease and thus renewing the secret fetched by vault-agent-init?

Specs

Vault versions

• image: "hashicorp/vault-k8s:0.3.0"
• image: vault:1.3.2

Auth methods

• I’m running my setup on a GCP cluster
• I have tried it with the k8s auth method. Then the entire authentication and rendering (of vault-agent) takes 200ms, so the application has no time to pick up the (soon to be expired) secrets of vault-agent-init => it picks up the new (vault-agent) secrets and things work out fine
• When using GCP Auth method I first get a timeout (net/http: timeout awaiting response headers), and the authentication only succeeds on the second attempt. (This whole process takes around 8 seconds…

[ERROR] auth.handler: error getting path or data from method: error="unable to sign JWT for projects/xxx/serviceAccounts/xyz@abc.iam.gserviceaccount.com using given Vault credentials: Post https://iam.googleapis.com/v1/projects/xxx/serviceAccounts/xyz@abc.iam.gserviceaccount.com:signJwt?alt=json&prettyPrint=false: Get http://IP/computeMetadata/v1/instance/service-accounts/default/token?scopes=xyz: net/http: timeout awaiting response headers" backoff=1.164255094

• The it backs off for a couple of seconds, retries the authentication, succeeds and writes the secrets to the file.
• But as said, the application has already picked up the files of the init-container…

Agent config:

    "auto_auth" = {
      "method" = {
        "config" = {
          "role" = "xyz-role"
          "type" = "iam"
          "project" = "xyz"
          "service_account" = "abc@xyz.iam.gserviceaccount.com"
          "jwt_exp" = 10
        }
        "mount_path" = "auth/ourGcpAuthPath"
        "type" = "gcp"
      }

      "sink" = {
        "config" = {
          "path" = "/home/vault/.token"
        }

        "type" = "file"
      }
    }

    "exit_after_auth" = true
    "pid_file" = "/home/vault/.pid"

    "template" = {
      "contents" = "nothing special"
      "destination" = "/vault/secrets/jdbc.yaml"
      "create_dest_dirs" = true
    }

    "vault" = {
      "address" = "https://vault.abc.cde"
    }

• The agent config is the same, except the exit_after_auth is set to false.

Deployment annotations and istio config

• Our deployment runs with the following annotations:

  vault.hashicorp.com/agent-init-first: "true"
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-configmap: our-vault-configmap
  

• We also have an istio sidecar running. To make it work we use a ServiceEntry:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: vault-service-entry
  namespace: someNamespace
spec:
  hosts:
  - vault.vault.svc.cluster.local
  location: MESH_EXTERNAL
  ports:
  - name: http
    number: 8200
    protocol: HTTP
  resolution: DNS

• The ServiceEntry is NOT located in the same namespace as our (current) deployment, however I don’t believe this would be a problem.
• Istio is on the latest (1.5.1) version.