ingress-nginx: Validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format

NGINX Ingress controller version: 0.33.0

Kubernetes version (use kubectl version): 1.16.9 & 1.18.3

Environment:

Bare-metal and minikube (for testing) Installed from https://github.com/kubernetes/ingress-nginx/blob/master/deploy/static/provider/baremetal/deploy.yaml and from the Helm Chart (chart: ingress-nginx-2.6.0) - ingress-nginx 0.33.0 in both situations

What happened: I tried to create invalid ingress object:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: k8s-debug
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      blah;
spec:
  rules:
  - host: blah.blah.test
    http:
      paths:
      - backend:
          serviceName: k8s-debug
          servicePort: 80

It was refused to create it as expected, but Kube-apiserver also logs fail message. Probably there is something broken in message exchange between validation webhook and kube-apiserver. The important part is Failed to set admission audit annotation validate.nginx.ingress.kubernetes.io/nginx.ingress.kubernetes.io/error to … for validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format, the right format is a DNS subdomain prefix and ‘/’ and key name. (e.g. ‘podsecuritypolicy.admission.k8s.io/admit-policy’).

W0618 05:54:38.001390       1 dispatcher.go:227] Failed to set admission audit annotation validate.nginx.ingress.kubernetes.io/nginx.ingress.kubernetes.io/error to
-------------------------------------------------------------------------------
Error: exit status 1
2020/06/18 05:54:37 [emerg] 126#126: unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: [emerg] unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: configuration file /tmp/nginx-cfg429015755 test failed

-------------------------------------------------------------------------------
 for validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy')
W0618 05:54:38.001533       1 dispatcher.go:141] rejected by webhook "validate.nginx.ingress.kubernetes.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validate.nginx.ingress.kubernetes.io\" denied the request: \n-------------------------------------------------------------------------------\nError: exit status 1\n2020/06/18 05:54:37 [emerg] 126#126: unknown directive \"blah\" in /tmp/nginx-cfg429015755:546\nnginx: [emerg] unknown directive \"blah\" in /tmp/nginx-cfg429015755:546\nnginx: configuration file /tmp/nginx-cfg429015755 test failed\n\n-------------------------------------------------------------------------------\n", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}

What you expected to happen:

There shouldn’t be this part of message:

W0618 05:54:38.001390       1 dispatcher.go:227] Failed to set admission audit annotation validate.nginx.ingress.kubernetes.io/nginx.ingress.kubernetes.io/error to
-------------------------------------------------------------------------------
Error: exit status 1
2020/06/18 05:54:37 [emerg] 126#126: unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: [emerg] unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: configuration file /tmp/nginx-cfg429015755 test failed

-------------------------------------------------------------------------------
 for validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy')

How to reproduce it: start minikube

minikube start

deploy Ingress-nginx from static yaml file

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

or from Helm Chart

kubectl create ns ingress-nginx
helm -n ingress-nginx install my-release ingress-nginx/ingress-nginx

check the logs of the Kube-apiserver and Ingress-nginx in two separate consoles

kubectl -n kube-system logs -f kube-apiserver-minikube
kubectl -n ingress-nginx logs -f ingress-nginx-controller-pod

try to create invalid ingress (yaml is above)

kubectl apply -f invalid-ingress.yaml

/kind bug

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (7 by maintainers)

Commits related to this issue

charts,salt,build: Downgrade NGINX Ingress chart to v4.0.6 Downgrade the NGINX Ingress chart to v4.0.6 and also the ingress controller image to v1.0.4 since the version v1.0.5 contains a bug that pre... — committed to scality/metalk8s by TeddyAndrieux 2 years ago

Most upvoted comments

I am getting the above message

> kubectl apply -f ingress.yaml

Error from server: error when creating "ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io"
denied the request: rejecting admission review because the request does not contains an Ingress resource but
networking.k8s.io/v1, Resource=ingresses with name example-ingress in namespace default

What does this mean?

kilianc on Sep 17, 2020

API group is err in doc

https://github.com/kubernetes/ingress-nginx/blob/nginx-0.30.0/docs/deploy/validating-webhook.md

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: check-ingress
webhooks:
- name: validate.nginx.ingress.kubernetes.io
  rules:
  - apiGroups:
    - extensions #1.16 -
    - networking.k8s.io  #1.16+
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
  failurePolicy: Fail
  clientConfig:
    service:
      namespace: ingress-nginx
      name: ingress-validation-webhook
      path: /networking.k8s.io/v1beta1/ingress
    caBundle: <pem encoded ca cert that signs the server cert used by the webhook>

https://github.com/kubernetes/ingress-nginx/commit/0dce5be743ae012dc5b22c1cf4b91dcdb8b64d88

vsxen on Jun 30, 2020

I’m still having the issue with 0.48.1 and chart 3.34 and the validationwebhookconfigurations is:

kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: spinnaker
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.47.0
    helm.sh/chart: ingress-nginx-3.34.0
  name: nginx-ingress-admission
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    caBundle: <ca bundle>
    service:
      name: nginx-ingress-controller-admission
      namespace: nginx-ingress
      path: /networking/v1beta1/ingresses
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: validate.nginx.ingress.kubernetes.io
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - networking.k8s.io
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

My kubernetes cluster is in v1.19.10 version

Lord-Y on Jul 20, 2021

I am getting the above message

> kubectl apply -f ingress.yaml

Error from server: error when creating "ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io"
denied the request: rejecting admission review because the request does not contains an Ingress resource but
networking.k8s.io/v1, Resource=ingresses with name example-ingress in namespace default

What does this mean?

This issue is fixed in 0.40.0. Before this version ingresses networking.k8s.io/v1 (you are running k8s 1.19) were not supported in the validation webhook

aledbf on Nov 4, 2020

same with nextcloud chart and its default snippet

k8s 1.21.6
ingress-nginx 4.0.8 (image 1.0.5)

 nginx.ingress.kubernetes.io/server-snippet: |
          server_tokens off;
          proxy_hide_header X-Powered-By;
          rewrite ^/.well-known/webfinger /public.php?service=webfinger last;
          rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
          rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
          location = /.well-known/carddav {
            return 301 $scheme://$host/remote.php/dav;
          }
          location = /.well-known/caldav {
            return 301 $scheme://$host/remote.php/dav;
          }
          location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
          }
          location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
            deny all;
          }
          location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
            deny all;
          }

Helm upgrade failed: cannot patch "nextcloud" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/server-snippet annotation contains invalid word location

zzorica on Nov 20, 2021

I have the same issue on 1.19.0 and 1.20.0 and 1.22.0 kubernetes using NGINX helm chart ingress-nginx-4.0.7 (version 1.0.5). It rejects the use of the “location” word when in the ingress.

metadata: name: {{ template “content-services.shortname” . }}-share labels: {{- include “share.labels” . | nindent 4 }} annotations: kubernetes.io/ingress.class: “nginx” # Default limit is 1m, document(s) above this size will throw 413 (Request Entity Too Large) error nginx.ingress.kubernetes.io/proxy-body-size: {{ .Values.repository.ingress.maxUploadSize }} nginx.ingress.kubernetes.io/affinity: “cookie” nginx.ingress.kubernetes.io/session-cookie-name: “alfrescoShare” nginx.ingress.kubernetes.io/session-cookie-path: “/share” nginx.ingress.kubernetes.io/session-cookie-max-age: “604800” nginx.ingress.kubernetes.io/session-cookie-expires: “604800” nginx.ingress.kubernetes.io/configuration-snippet: | location ~ ^(/./proxy/./api/solr/.)$ {return 403 ;} location ~ ^(/./-default-/proxy/./api/.)$ {return 403;}

Error: INSTALLATION FAILED: release acs failed, and has been uninstalled due to atomic being set: admission webhook “validate.nginx.ingress.kubernetes.io” denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation contains invalid word location helm.go:88: [debug] admission webhook “validate.nginx.ingress.kubernetes.io” denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation contains invalid word location

torgerss on Nov 17, 2021