vault-k8s: Sporadic tls: bad certificate Issue

Env Versions:

GKE 1.14.10-gke.36 vault:1.2.1 agent_inject_vault_image: vault:1.3.2 vault-agent-injector: hashicorp/vault-k8s:0.3.0

I’m seeing sporadic tls: bad certificate error log entries in the vault-agent-injector container. This is causing vault-agent containers not being injected into our pods and causing apps to crash. Wonder if anyone has seen this or better yet could point out my mistakes 🙏🏻

vault-agent-injector.log

2020-06-02T05:47:13.961421898Z 2020-06-02T05:47:13.953Z [INFO]  handler: Starting handler..
2020-06-02T05:47:13.961490627Z Listening on ":8080"...
2020-06-02T05:47:13.980111277Z Updated certificate bundle received. Updating certs...
2020-06-02T06:02:49.031206881Z 2020-06-02T06:02:49.030Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:16:58.772934263Z 2020-06-02T06:16:58.771Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:19:12.222721585Z 2020-06-02T06:19:12.217Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:29:45.604690396Z 2020-06-02T06:29:45.603Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:38:44.372917575Z 2020-06-02T06:38:44.372Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:59:32.662445342Z 2020-06-02T06:59:32.662Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:59:33.312342263Z 2020-06-02T06:59:33.312Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:59:33.325205594Z 2020-06-02T06:59:33.325Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:59:47.241623679Z 2020-06-02T06:59:47.239Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T06:59:47.415966009Z 2020-06-02T06:59:47.413Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:00:21.976433396Z 2020-06-02T07:00:21.966Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:00:22.289668763Z 2020-06-02T07:00:22.219Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:01:50.033803963Z 2020-06-02T07:01:50.031Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:02:27.804240219Z 2020-06-02T07:02:27.804Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:02:27.831802107Z 2020-06-02T07:02:27.831Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:02:28.618867665Z 2020-06-02T07:02:28.618Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:07:30.679237828Z 2020-06-02T07:07:30.676Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:07:31.055053655Z 2020-06-02T07:07:31.034Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:07:36.267515821Z 2020-06-02T07:07:36.267Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:07:36.279054864Z 2020-06-02T07:07:36.278Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:07:52.096649591Z 2020-06-02T07:07:52.079Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:07:52.096678807Z 2020-06-02T07:07:52.091Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:08:04.433026731Z 2020-06-02T07:08:04.432Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:08:10.692660442Z 2020-06-02T07:08:10.689Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:18:40.396171467Z 2020-06-02T07:18:40.395Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:18:40.578473213Z 2020-06-02T07:18:40.578Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:18:40.919113638Z 2020-06-02T07:18:40.918Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:20:14.334225402Z 2020-06-02T07:20:14.319Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:28:07.110445901Z 2020-06-02T07:28:07.109Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T07:28:30.00637003Z 2020-06-02T07:28:30.004Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T09:10:52.530310714Z 2020-06-02T09:10:52.530Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T09:12:56.065020085Z 2020-06-02T09:12:56.056Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T09:13:35.498441485Z 2020-06-02T09:13:35.493Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T09:13:35.502775497Z 2020-06-02T09:13:35.501Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T10:30:28.366852121Z 2020-06-02T10:30:28.366Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T10:30:28.588449658Z 2020-06-02T10:30:28.588Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T11:37:04.189491886Z 2020-06-02T11:37:04.189Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T11:37:04.197282233Z 2020-06-02T11:37:04.197Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T11:37:04.419698687Z 2020-06-02T11:37:04.387Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T17:13:34.390024893Z 2020-06-02T17:13:34.389Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T20:57:42.896885748Z 2020-06-02T20:57:42.895Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T20:57:58.012645084Z 2020-06-02T20:57:57.993Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T20:58:03.903044028Z 2020/06/02 20:58:03 http: TLS handshake error from 10.154.0.89:44848: remote error: tls: bad certificate
2020-06-02T20:59:59.111683584Z 2020-06-02T20:59:59.111Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T20:59:59.451507454Z 2020-06-02T20:59:59.449Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:00:08.96931677Z 2020-06-02T21:00:08.969Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:00:08.972082809Z 2020-06-02T21:00:08.971Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:00:09.085292012Z 2020-06-02T21:00:09.083Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:00:09.088025244Z 2020-06-02T21:00:09.087Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:00:10.05514469Z 2020-06-02T21:00:10.052Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:00:26.386882258Z 2020/06/02 21:00:26 http: TLS handshake error from 10.154.0.3:42088: remote error: tls: bad certificate
2020-06-02T21:06:07.254377869Z 2020-06-02T21:06:07.252Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:16:04.834988005Z 2020-06-02T21:16:04.834Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:23:24.13032099Z 2020-06-02T21:23:24.129Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T21:23:31.127620578Z 2020-06-02T21:23:31.124Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:31:20.399379715Z 2020-06-02T22:31:20.397Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:33.014280792Z 2020-06-02T22:34:33.014Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:33.019973546Z 2020-06-02T22:34:33.019Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:33.071176991Z 2020-06-02T22:34:33.070Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:33.084315252Z 2020-06-02T22:34:33.084Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:35.328848206Z 2020-06-02T22:34:35.327Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:38.340745517Z 2020-06-02T22:34:38.340Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-06-02T22:34:39.476182565Z 2020-06-02T22:34:39.467Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (5 by maintainers)

Most upvoted comments

Hi all, we’re working on a fix for this, but wanted to give an update. Likely the fix is going to land in Vault Helm to generate certificates for both the injector and Vault deployments out of the box.

While the auto-tls feature here is handy, it was sourced from another HashiCorp web hook (Consul Connect) and wasn’t designed for multiple replicas. This is evident in how the software creates a CA and signs certs. We would rather not introduce state into the injector and there’s no way to coordinate this without it (such as global locking).

This effort is likely going to launch with Vault 1.6. We’re working on some features in the Vault project for bootstrapping TLS and we’ll likely leverage that as our pre-start job.

What Can I Do Today?

While auto-tls is the default feature, the injector does support user supplied certificates. I’ve created a document with instructions on setting this up: https://www.vaultproject.io/docs/platform/k8s/helm/examples/injector-tls. Using manual certs you can scale as fit!

If anyone is not using Helm to install the injector, please let me know and I can update the Vault K8s documentation for this use case.

jasonodonnell on Sep 2, 2020