ingress-nginx: func NumCPU() fails to return correct cpus when using cgroups2
What happened:
Installing ingress-nginx helm chart into k8s 1.24 with base OS ubuntu 22 (cgroups v2) doensn’t generate correct worker_process based on k8s resources, instead it get max cpu cores of the node.
- values.yaml resourfes part
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 1
memory: 512Mi
On previous ubuntu 20 (cgroups v1) the /etc/nginx/nginx.conf
file into the container got configured to
worker_processes 1;
On ubuntu 22 base os (groups v2) the /etc/nginx/nginx.conf
file into the container gets configured all the posible cores in the node (64 in my case),
worker_process 64;
What you expected to happen:
- /etc/nginx/nginx.conf:
worker_process 1;
I think the way is calculated only works for cgroup v1 and not for cgroup v2 returning max cpus when not finding cgrup v1 files:
https://github.com/kubernetes/ingress-nginx/blob/main/pkg/util/runtime/cpu_linux.go#L38-L54
func NumCPU()
uses cgroup v1 paths to calculate:
- /sys/fs/cgroup/cpu/cpu.cfs_quota_us
- /sys/fs/cgroup/cpu/cpu.cfs_period_us In cgroup v2, this was replaced by one file containing the previous 2 values from v1
- /sys/fs/cgroup/cpu.max
NGINX Ingress controller version ):
NGINX Ingress controller
Release: v1.6.4
Build: 69e8833858fb6bda12a44990f1d5eaa7b13f4b75
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
-------------------------------------------------------------------------------
Kubernetes version: v1.24.6
Environment:
- Cloud provider or hardware configuration: OnPremise
- OS: Ubuntu 22.04
- Kernel (e.g.
uname -a
): Linux worker-4 5.15.0-60-generic - Install tools:
- kubespray v2.10 k8s
- Basic cluster related info:
kubectl version
1.24.6
How to reproduce this issue: nginx controller being executed in any node with cgroups 2 and limited by 1 CPU
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 3
- Comments: 17 (5 by maintainers)
We have currently 400+ ingress controller on newly upgraded 1.25 Azure AKS clusters waiting for a solution. Azure brings Ubuntu 22 with cgroup v2 with AKS 1.25 !
Yes, should push something in the next few days