ingress-nginx: Release 1.0.5 is breaking (Chart release 4.0.8), major version should be bumped
Notice
Everyone reading this: Please save your “this is unacceptable“ comments. This is an open source project, if you want to berate someone, ask their consent first and pay them for it.
If you want to show that you share my opinion that this did not go well, please +1 or like this post, but keep the comments spam free.
Issue
NGINX Ingress controller version 1.0.5
Kubernetes version (use kubectl version
): v1.22.2+k3s1, but this is irrelevant
How was the ingress-nginx-controller installed: helm chart, version 4.0.8
What happened: ingress-nginx Release 1.0.5 contains breaking changes, but no major version bump occurred. Helm chart version 4.0.8 contains breaking changes, but no major version bump occured.
1.0.5 introduces the annotation-value-word-blocklist
option to the configmap that will block ingresses that have specific words or characters in them. The default for this is load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\
, making this i breaking change.
What you expected to happen: Major version bump for breaking changes.
The changelog contains the hint Possible Breaking Change. If a change is breaking to some configurations, it is a breaking change and therefore requires a major version bump.
Please release a 1.0.6 that uses an empty default value for this setting and include the current default of load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\
in a 2.0.0 ingress-nginx and 5.0.0 helm chart release.
Please also consider retracting the 1.0.5 ingress-nginx and 4.0.8 chart release as this might break many installations.
How to reproduce it:
Use any install method for ingress-nginx in version 1.0.5 with the Helm chart version 4.0.8, using the default configuration.
Create an ingress (please add any additional annotation required)
echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-bar
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/server-snippet: |-
location = /.well-known/carddav {
return 301 $scheme://$host/remote.php/dav;
}
spec:
ingressClassName: nginx # omit this if you're on controller version below 1.0.0
rules:
- host: foo.bar
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80
" | kubectl apply -f -
make a request
POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME)
kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' localhost
Anything else we need to know:
More and more systems are automatically deploying patch version upgrades automatically, expecting that a patch version only contains patches. This bug really caught me unaware (and definitely showed a lack of automated tests on my side, even before merging patch versions, this should be standard.
I’m bringing this to your attention as I am of the opinion that if we should not fear bumping major versions. This incident looks like a manifestation of that fear to me as I’ve seen it in the past.
/kind bug
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 15
- Comments: 20 (6 by maintainers)
/triage accepted /priority critical-urgent
That is a valid point, I will discuss with it @rikatz . We were prioritizing the CVE fix and should be more thoughtful when we release new features like this next.
Hey folks.
We released v1.1.0 where the blocklist is empty by default
Please help us: test, check if the breaking change still persists, and also please please please configure at least a small sane default for your environment.
Thanks once more for the patience and help on this!
@morremeyer thanks for the nice reporting on this and the consideration 😃 really appreciate it!
We are working on a different way to secure the data and control plane rather than blocking known “bad words” Folks were very vocal when we broke this feature. I don’t think they would migrate to v2.0.0 if we made that breaking change.
@BloodyIron thank you for your opinion and we welcome them in our community meetings and slack to help further the discussion.
@janosmiko Yes, you can set the value of
annotation-value-word-blocklist
to an empty string""
.@strongjz @rikatz Thanks for taking care of it, I really appreciate it.
I tried setting it to empty on my setup, which weirdly failed and was still rejecting ingresses for blocked characters. I didn’t debug that much further though as I needed to get things running again.
I therefore set it to
thisvaluewillneverappear
, which worked.Just want to mention this in case there is a possibility to add a test for that.
Agreed! Sorry for the mess, we were so in a rush to fix this that the “we are breaking users” was not accounted 😕
@strongjz you wanna tackle defaulting this to empty? Otherwise I can take care of this tomorrow 😃