ingress-nginx: CVE-2023-5043: Ingress nginx annotation injection causes arbitrary command execution

Issue Details

A security issue was identified in ingress-nginx where the nginx.ingress.kubernetes.io/configuration-snippet annotation on an Ingress object (in the networking.k8s.io or extensions API group) can be used to inject arbitrary commands, and obtain the credentials of the ingress-nginx controller. In the default configuration, that credential has access to all secrets in the cluster.

This issue has been rated High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L), and assigned CVE-2023-5043.

Affected Components and Configurations

This bug affects ingress-nginx. If you do not have ingress-nginx installed on your cluster, you are not affected. You can check this by running kubectl get po -n ingress-nginx.

If you are running the “chrooted” ingress-nginx controller introduced in v1.2.0 (gcr.io/k8s-staging-ingress-nginx/controller-chroot), command execution is possible but credential extraction is not, so the High severity does not apply.

Multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.

Affected Versions

  • <v1.9.0

Versions allowing mitigation

  • v1.9.0

Mitigation

Ingress Administrators should set the --enable-annotation-validation flag to enforce restrictions on the contents of ingress-nginx annotation fields.

Detection

If you find evidence that this vulnerability has been exploited, please contact security@kubernetes.io

Additional Details

See ingress-nginx Issue #10571 for more details.

Acknowledgements

This vulnerability was reported by suanve

Thank You, CJ Cullen on behalf of the Kubernetes Security Response Committee

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Comments: 25 (8 by maintainers)

Most upvoted comments

sorry…to confirm we need to add --enable-annotation-validation to kind: Deployment of the ingress-controller right (https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/cloud/deploy.yaml#L424) ?

@thiDucTran, You will need to update your local values.yaml file with this toggle set to true

https://github.com/kubernetes/ingress-nginx/blob/cf156c73905792c6ce1f02822fe9ffca24381d3b/charts/ingress-nginx/values.yaml#L21

We are actively trying to brake out the security model for data plane and the control plane to help increase the security of the controller.

Good to hear! My suggestion is to avoid closing security vulnerabilities without investing sufficient time into validating the provided mitigations. I understand it is usually not trivial and time consuming. Luckily, the community provided quite some doubts already to reconsider the previously made decision.

I would suggest pinpointing that for this particular finding the mitigations are the following:

  • Keeping the (now default) setting allowSnippetAnnotations: false
  • If the above is not possible, limiting and auditing the number of users which are allowed to use configuration-snippet annotation. OPA Gatekeeper or Kyverno policies can do their job for this.

For vulnerability CVE-2023-5044 (permanent-redirect) the provided mitigation (enable-annotation-validation) will work, as basically only valid URL is expected.

Vulnerability CVE-2022-4886 (ImplementationSpecific) path is a really complicated beast. By definition user is allowed to use regular expressions there, which means “weird” symbols. However, as controller just inserts the provided setting into nginx config file, this might lead to config context escape. In order to avoid that one may just forbid ImplementationSpecific types via OPA Gatekeeper or Kyverno. However, this feature is way more used than even configuration-snippet. The provided mitigation in the closed finding is vague, enable-annotation-validation is not complete in this case as well, because dangerous symbols are allowed. I would adjust validation on controller’ side and allow less characters in regex.

Any time you allow users to run untrusted code it could result in security flaws.

That is why we are relying on containerization. 😃 In this case, the controller provides functionality to adjust web server’s configuration directly, without doing context-aware escape. It is similar to granting users a freedom of editing web page they are viewing: everything is fine, until there is an adversary (compromised user) which injects JavaScript to this page and benefit from others.

Are you saying that the remediation suggested of --enable-annotation-validation doesn’t work for configuration-snippet and disabling the annotation with controller.allowSnippetAnnotations: false should be used instead ?

enable-annotation-validation does enable additional validations. Unfortunately, they are just regular expressions working in blacklist mode. This never gives 100% assurance that mitigation works. https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/annotations/parser/validators.go#L62

There are some checks here: https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go#L25 but they are easily bypassable.

As you can see here, configuration-snippet is not validated: https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/annotations/snippet/main.go#L36 anyway.

Many people are using the remediation suggested by @cjcullen in their k8s cluster. If it is not effective, can we work and update the description to mention controller.allowSnippetAnnotations: false instead ?

I have a lot of questions why these stories were closed at all, as the product is still affected and suggested mitigations are “mitigations” at best (even if they would work), but not fixes.

On the other hand, configuration snippet annotation is indeed now disabled by default, which is indeed the best way to tackle this particular finding. 😃

Can someone suggest an invalid value for nginx.ingress.kubernetes.io/configuration-snippet one can use in order to validate that the remediation works and do a before and after test ?

For all three these issues the exploitation is similar to what is described on this page: https://raesene.github.io/blog/2023/10/29/exploiting-CVE-2023-5044/ Only the approach to escape from the current nginx config context differs. The existing regex-based validations do not work or can be bypassed. In case of “configuration-snippet” what works as remediation: disabling this annotation.

If the ingress controller is configured withallowSnippetAnnotations: false, is this vulnerability mitigated?

@phidlipus You can upgrade, there are no v1.25+ specific features in the new version of Ingress NGINX.

But please also make sure to test before upgrading.

Is there any way how to mitigate this for versions <1.9.0? We are using v1.8.X version and according to compatility matrix we can’t upgrade to later versions, because we are using Kubernetes <1.25. Do you have any recommandations? Or is possible just ignore the compatibility matrix and update to the latest version? Same question is also valid for CVE-2023-5044.

Hey folks,

As this CVE has been opened for 1 week now, I’m closing the issue.

The description of the issue contains all the required mitigations, and we plan in future releases to turn the validation on by default and also implement more safety measures.

Thank you all for using the project, and for your continuous support for us.

/close

thanks all…fyi…if you are not using helm…the equivalent is to add --enable-annotation-validation=true as an argument in https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/cloud/deploy.yaml#L424