ingress-nginx: Global Config map not loaded even when explicitly defined in args and documentation on topic is severely lacking.
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.1.0
Build: cacbee86b6ccc45bde8ffc184521bed3022e7dee
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.9
-------------------------------------------------------------------------------
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"392a7e3e62996d05ada2157090ee6ed616af1a8c", GitTreeState:"clean", BuildDate:"2021-11-15T08:35:15Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
Environment:
-
Cloud provider or hardware configuration: Azure
-
OS (e.g. from /etc/os-release): linux
-
How was the ingress-nginx-controller installed:
- If helm was used then please show output of
helm ls -A | grep -i ingress
- If helm was used then please show output of
ingress-nginx ingress-basic 1 2021-12-17 10:38:06.5524382 -0500 EST deployed ingress-nginx-4.0.13 1.1.0
- If helm was used then please show output of
helm -n <ingresscontrollernamepspace> get values <helmreleasename>
controller:
nodeSelector:
kubernetes.io/os: linux
replicaCount: 2
service:
externalTrafficPolicy: Local
loadBalancerIP: 52.186.101.40
default-backend-service: cadlearning/api-svc
default-ssl-certificate: cadlearning/cadlearning-tls-secret
defaultBackend:
enabled: false
nodeSelector:
kubernetes.io/os: linux
- Current State of the controller:
-
kubectl describe ingressclassesName: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/version=1.1.0 helm.sh/chart=ingress-nginx-4.0.13 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-basic Controller: k8s.io/ingress-nginx Events: <none> -
kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
-
Name: ingress-nginx-controller-bb8d97988-5p6nx
Namespace: ingress-basic
Priority: 0
Node: aks-agentpool-25177618-vmss000000/10.240.0.4
Start Time: Fri, 17 Dec 2021 11:10:43 -0500
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=bb8d97988
Annotations: <none>
Status: Running
IP: 10.240.0.29
IPs:
IP: 10.240.0.29
Controlled By: ReplicaSet/ingress-nginx-controller-bb8d97988
Containers:
controller:
Container ID: containerd://c0b4e05f1e06ca5b31faafed5469bb33be62c0cc1f7bc88dfb0b0c9e23cfe740
Image: k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
Image ID: k8s.gcr.io/ingress-nginx/controller@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--configmap=ingress-basic/nginx-config
--default-ssl-certificate=cadlearning/cadlearning-tls-secret
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Running
Started: Fri, 17 Dec 2021 11:10:44 -0500
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-bb8d97988-5p6nx (v1:metadata.name)
POD_NAMESPACE: ingress-basic (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-q8kfj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
ingress-nginx-token-q8kfj:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-token-q8kfj
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m32s default-scheduler Successfully assigned ingress-basic/ingress-nginx-controller-bb8d97988-5p6nx to aks-agentpool-25177618-vmss000000
Normal Pulled 7m32s kubelet Container image "k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a" already present on machine
Normal Created 7m32s kubelet Created container controller
Normal Started 7m32s kubelet Started container controller
Normal RELOAD 7m30s nginx-ingress-controller NGINX reload triggered due to a change in configuration
kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name: ingress-nginx-controller
Namespace: ingress-basic
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=1.1.0
helm.sh/chart=ingress-nginx-4.0.13
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-basic
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Families: <none>
IP: 10.0.90.198
IPs: 10.0.90.198
IP: 52.186.101.40
LoadBalancer Ingress: 52.186.101.40
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30875/TCP
Endpoints: 10.240.0.29:80,10.240.0.72:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31284/TCP
Endpoints: 10.240.0.29:443,10.240.0.72:443
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 30729
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 41m service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 41m service-controller Ensured load balancer
- Others: ingress-basic/nginx-config:
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
namespace: ingress-basic
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
proxy-connect-timeout: "10"
proxy-read-timeout: "120"
proxy-send-timeout: "120"
enable-access-log-for-default-backend: "true"
use-http2: "true"
add-headers: "ingress-basic/custom-headers"
proxy-set-headers: "ingress-basic/proxy-headers"
hsts: "true"
customer-headers:
apiVersion: v1
data:
X-Different-Name: "true"
X-Request-Start: "t=${msec}"
X-Using-Nginx-Controller: "true"
X-Content-Type-Options: "nosniff"
X-XSS-Protection: "1; mode=block"
Referrer-Policy: "no-referrer-when-downgrade"
Feature-Policy: "notifications 'self'; usemedia *;gyroscope: 'none'"
kind: ConfigMap
metadata:
name: custom-headers
namespace: ingress-basic
proxy-headers:
apiVersion: v1
data:
X-Forwarded-Proto: https;
kind: ConfigMap
metadata:
name: proxy-headers
namespace: ingress-basic
What happened:
No matter what I do or set, the custom config map that I have defined and is clearly in the configuration under the arguments run. (which the documentation doesn’t even tell you what the default name of the config map should be under the config map documentation area so you’re guessing). When I request a page, the page comes up fine, but the custom headers are not added and it doesn’t appear that any of the other items in the nginx-config file is being used.
What you expected to happen:
It should load in all of the configuration in the nginx-config properly, and append all of the custom headers in the custom-headers config map as defined in add-headers in nginx-config to every client response.
How to reproduce it: Install with helm:
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-basic --set controller.replicaCount=2 --set controller.nodeSelector.“kubernetes.io/os”=linux --set defaultBackend.nodeSelector.“kubernetes.io/os”=linux --set controller.service.loadBalancerIP=“PUBLIC_IP” --set defaultBackend.enabled=“false” --set controller.service.externalTrafficPolicy=Local
Edit the config like this: kubectl edit deployment ingress-nginx-controller -n ingress-basic
Add: - --configmap=ingress-basic/nginx-config To the args section, save and close and note it applies it, and the pods restart.
Notes
- The documentation on how to use a custom config map is seriously lacking and doesn’t tell you how to add it, where to add it or even what it should be named. And the example doesn’t do this either and there are multiple conflicting examples with multiple different names for the config map. The documentation should be updated (which you can’t do because clicking the edit on this page: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ gets you a 404 error.
- There doesn’t appear to be any documentation on how to verify that your configmap is working, that it loaded your config map (other than looking and finding it not in your nginx.conf) and know way to know what it actually loaded or what it tried to find as the config map.
As a result this is a double bug which may just be a problem with the documentation. The first is, the documentation isn’t clear so you’re guessing. The second is that it’s ignoring the config map even when explicitly defined AND there is no way to tell what it actually tried to load, or at least no documentation to tell you how to find the logs that tell you what it tried to load/merge.
/kind bug
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (10 by maintainers)
@manikanth04 How is the docs here coming? The docs needs to discuss how to update already existing proxy headers like
X-Forwarded-Proto, I’ve been looking for how to properly set this, but can’t find it anywhere.@longwuyuan I’d suggest that that’s guessing happening until they get it right, not because of good documentation. And guessing leads to mistakes which lead to security issues etc.
I’ve already stated exactly what’s wrong with the docs:
These are all deficiencies that lead to guessing and hoping. (Which I did and eventually got it working but wow.)
A proper documentation would say “The config map is anmed xxxx in the same name space as you installed it in. To edit the file run the following yyyyy command making sure not to alter the current values…” And then give an example of commands to create the add headers configmap file, and then reference it properly in the root config map file.