ingress-nginx: CVE-2023-5043: Ingress nginx annotation injection causes arbitrary command execution
Issue Details
A security issue was identified in ingress-nginx where the nginx.ingress.kubernetes.io/configuration-snippet annotation on an Ingress object (in the networking.k8s.io
or extensions
API group) can be used to inject arbitrary commands, and obtain the credentials of the ingress-nginx controller. In the default configuration, that credential has access to all secrets in the cluster.
This issue has been rated High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L), and assigned CVE-2023-5043.
Affected Components and Configurations
This bug affects ingress-nginx. If you do not have ingress-nginx installed on your cluster, you are not affected. You can check this by running kubectl get po -n ingress-nginx
.
If you are running the “chrooted” ingress-nginx controller introduced in v1.2.0 (gcr.io/k8s-staging-ingress-nginx/controller-chroot), command execution is possible but credential extraction is not, so the High severity does not apply.
Multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Affected Versions
- <v1.9.0
Versions allowing mitigation
- v1.9.0
Mitigation
Ingress Administrators should set the --enable-annotation-validation flag to enforce restrictions on the contents of ingress-nginx annotation fields.
Detection
If you find evidence that this vulnerability has been exploited, please contact security@kubernetes.io
Additional Details
See ingress-nginx Issue #10571 for more details.
Acknowledgements
This vulnerability was reported by suanve
Thank You, CJ Cullen on behalf of the Kubernetes Security Response Committee
About this issue
- Original URL
- State: closed
- Created 8 months ago
- Comments: 25 (8 by maintainers)
@thiDucTran, You will need to update your local values.yaml file with this toggle set to true
https://github.com/kubernetes/ingress-nginx/blob/cf156c73905792c6ce1f02822fe9ffca24381d3b/charts/ingress-nginx/values.yaml#L21
Good to hear! My suggestion is to avoid closing security vulnerabilities without investing sufficient time into validating the provided mitigations. I understand it is usually not trivial and time consuming. Luckily, the community provided quite some doubts already to reconsider the previously made decision.
I would suggest pinpointing that for this particular finding the mitigations are the following:
allowSnippetAnnotations: false
configuration-snippet
annotation. OPA Gatekeeper or Kyverno policies can do their job for this.For vulnerability CVE-2023-5044 (
permanent-redirect
) the provided mitigation (enable-annotation-validation
) will work, as basically only valid URL is expected.Vulnerability CVE-2022-4886 (
ImplementationSpecific
) path is a really complicated beast. By definition user is allowed to use regular expressions there, which means “weird” symbols. However, as controller just inserts the provided setting into nginx config file, this might lead to config context escape. In order to avoid that one may just forbidImplementationSpecific
types via OPA Gatekeeper or Kyverno. However, this feature is way more used than evenconfiguration-snippet
. The provided mitigation in the closed finding is vague,enable-annotation-validation
is not complete in this case as well, because dangerous symbols are allowed. I would adjust validation on controller’ side and allow less characters in regex.That is why we are relying on containerization. 😃 In this case, the controller provides functionality to adjust web server’s configuration directly, without doing context-aware escape. It is similar to granting users a freedom of editing web page they are viewing: everything is fine, until there is an adversary (compromised user) which injects JavaScript to this page and benefit from others.
enable-annotation-validation
does enable additional validations. Unfortunately, they are just regular expressions working in blacklist mode. This never gives 100% assurance that mitigation works. https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/annotations/parser/validators.go#L62There are some checks here: https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go#L25 but they are easily bypassable.
As you can see here, configuration-snippet is not validated: https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/annotations/snippet/main.go#L36 anyway.
I have a lot of questions why these stories were closed at all, as the product is still affected and suggested mitigations are “mitigations” at best (even if they would work), but not fixes.
On the other hand, configuration snippet annotation is indeed now disabled by default, which is indeed the best way to tackle this particular finding. 😃
For all three these issues the exploitation is similar to what is described on this page: https://raesene.github.io/blog/2023/10/29/exploiting-CVE-2023-5044/ Only the approach to escape from the current nginx config context differs. The existing regex-based validations do not work or can be bypassed. In case of “configuration-snippet” what works as remediation: disabling this annotation.
If the ingress controller is configured with
allowSnippetAnnotations: false
, is this vulnerability mitigated?@phidlipus You can upgrade, there are no v1.25+ specific features in the new version of Ingress NGINX.
But please also make sure to test before upgrading.
Is there any way how to mitigate this for versions <1.9.0? We are using v1.8.X version and according to compatility matrix we can’t upgrade to later versions, because we are using Kubernetes <1.25. Do you have any recommandations? Or is possible just ignore the compatibility matrix and update to the latest version? Same question is also valid for CVE-2023-5044.
Hey folks,
As this CVE has been opened for 1 week now, I’m closing the issue.
The description of the issue contains all the required mitigations, and we plan in future releases to turn the validation on by default and also implement more safety measures.
Thank you all for using the project, and for your continuous support for us.
/close
thanks all…fyi…if you are not using helm…the equivalent is to add
--enable-annotation-validation=true
as an argument in https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/cloud/deploy.yaml#L424