kyverno: [Bug] Webhook server: x509: certificate signed by unknown authority

Kyverno Version

1.6.x

Kubernetes Version

1.20.x

Kubernetes Platform

EKS

Kyverno Rule Type

Other

Description

This error occurs then you retry and it is okay. For example when creating a pod:

Error from server (InternalError): Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": Post "https://infra-kyverno-svc.infra.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "*.kyverno.svc")

Steps to reproduce

Create a pod

Expected behavior

No error message

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

I have read and followed the documentation AND the troubleshooting guide.
I have searched other issues in this repository and mine is not recorded.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 38 (21 by maintainers)

Most upvoted comments

Steps to reproduce:

helm install …
helm upgrade …

with --set createSelfSignedCert=true

The effect is that the cert CA is re-created, but that does not cause the PODs to restart and a such the cert on the POD is not signed anymore by the CA.

The helm chart could gate the creation by checking for the existence:

{{- if not (lookup "v1" "Secret" .Release.Namespace "kyverno-svc.kyverno.svc.kyverno-tls-ca") }}
... everything like before with helm annotation ... 
metadata:
  annotations:
    "helm.sh/resource-policy": "keep"
{{- end }}

The same section has another issue: The cert generated by the helm function will be rejected by clusters above 1.21

The generation only sets the CN which is rejected by golang 1.16+

This would be around line 2/3 of the original helm chart:

{{- $ca := .ca | default (genCA (printf "*.%s.svc" (include "kyverno.namespace" .)) 1024) -}}
{{- $svcName := (printf "%s.%s.svc" (include "kyverno.serviceName" .) (include "kyverno.namespace" .)) -}}
{{- $cert := genSignedCert $svcName nil (list $svcName) 1024 $ca -}}

This solution is far from perfect, one could also just have a sha check over the secret as part of the deployment to make the pods restart on CA change (and cert change which also happens, but isn’t picked up in a good way by kyverno yet)

  annotations:
    checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

P-S-bits on Apr 6, 2022