cert-manager: Server Key Usage by Default Breaks GRPC Certs

Bugs should be filed for issues encountered whilst operating cert-manager. You should first attempt to resolve your issues through the community support channels, e.g. Slack, in order to rule out individual configuration errors. Please provide as much detail as possible.

Describe the bug: After the merge of #2351, certificates generated with default settings break GRPC client<->server communication. This is specifically noticed with KIAM, which uses certificates to authenticate/encrypt communication via the client/server.

WARNING: 2019/11/27 18:29:07 grpc: Server.Serve failed to complete security handshake from "127.0.0.1:51648": tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage

Expected behaviour: The same certificates used by previous versions of KIAM continue to work after being upgraded.

Since the apiGroup is changed from cert-manager.k8s.io -> cert-manager.io, the certificates are regenerated with incorrect Extended Key Usages for this situation.

Steps to reproduce the bug:

Deploy quay.io/jetstack/cert-manager-controller:v0.10.1
Create the following objects in your Kubernetes cluster:

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-ca
  namespace: kiam
spec:
  commonName: kiam-ca
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: selfsigning
  secretName: kiam-ca
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-server
  namespace: kiam
spec:
  dnsNames:
  - localhost
  - kiam-server
  ipAddresses:
  - 127.0.0.1
  issuerRef:
    name: kiam-issuer
  secretName: kiam-server
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-agent
  namespace: kiam
spec:
  commonName: agent
  issuerRef:
    name: kiam-issuer
  secretName: kiam-agent

Upgrade to the latest cert-manager, which requires changing API groups.
Tail the logs of the kiam-server pod and notice that communication is now broken.

Anything else we need to know?:

Environment details::

Kubernetes version 1.14.7
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS
cert-manager version (e.g. v0.4.0): v0.10.1 / latest
Install method (e.g. helm or static manifests): static manifests

/kind bug

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 18 (4 by maintainers)

Most upvoted comments

Another gRPC user here.

What has triggered this behavior for those of us using mTLS with frameworks like gRPC, is specified in the Internet X.509 Public Key Infrastructure Certificate spec: RFC 5280.

In RFC 5280 under the section “Extended Key Usage” it states:

If the extension is present, then the certificate MUST only be used for one of the purposes indicated.

This indicates that if no extended key usage is present then it can be used for client auth or server auth. By specifying server auth as a default in #2351 we now have an extended key usage that says the certs created can only be used for the purpose of server auth and not client auth.

This prevented an upgrade from 0.11 to 0.13 for us and it would be really inconvenient to have to now specify usages on all our certificates.

Since #2351 was a fix for temporary certificates, can we add the usages to the temporary certificates instead of changing the default?

jessedearing on Feb 27, 2020

The solution here would be to add:
spec:
  ...
  keyUsages:
  - digital signature
  - key encipherment
  - server auth
  - client auth
explicitly to your Certificate resources.

Setting server auth by default is required for Safari/Mac OSX Catalina to properly validate a certificate for serving (else it’ll be marked untrusted), in your case, client auth is the exception here and you should explicitly denote it as a keyUsage when requesting your Certificate 😄

For anyone using this the keyUsages: key should be usages:

jsharpe on Jan 7, 2020

I was digging into the “why” on this today and was surprised to see how (A) the Go standard library validates the non-leaf parts of the chain and (B) how the gRPC Go implementation sets expected key usages for the non-leaf parts of the chain:

dhermes on Dec 16, 2020

Got bitten by this as well.

agusti-t on Mar 7, 2020