ingress-nginx: func NumCPU() fails to return correct cpus when using cgroups2

What happened:

Installing ingress-nginx helm chart into k8s 1.24 with base OS ubuntu 22 (cgroups v2) doensn’t generate correct worker_process based on k8s resources, instead it get max cpu cores of the node.

  • values.yaml resourfes part
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi

On previous ubuntu 20 (cgroups v1) the /etc/nginx/nginx.conf file into the container got configured to

worker_processes 1;

On ubuntu 22 base os (groups v2) the /etc/nginx/nginx.conf file into the container gets configured all the posible cores in the node (64 in my case),

worker_process 64;

What you expected to happen:

  • /etc/nginx/nginx.conf:
 worker_process 1;

I think the way is calculated only works for cgroup v1 and not for cgroup v2 returning max cpus when not finding cgrup v1 files:

https://github.com/kubernetes/ingress-nginx/blob/main/pkg/util/runtime/cpu_linux.go#L38-L54

func NumCPU() uses cgroup v1 paths to calculate:

  • /sys/fs/cgroup/cpu/cpu.cfs_quota_us
  • /sys/fs/cgroup/cpu/cpu.cfs_period_us In cgroup v2, this was replaced by one file containing the previous 2 values from v1
  • /sys/fs/cgroup/cpu.max

NGINX Ingress controller version ):

NGINX Ingress controller
  Release:       v1.6.4
  Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6
-------------------------------------------------------------------------------

Kubernetes version: v1.24.6

Environment:

  • Cloud provider or hardware configuration: OnPremise
  • OS: Ubuntu 22.04
  • Kernel (e.g. uname -a): Linux worker-4 5.15.0-60-generic
  • Install tools:
    • kubespray v2.10 k8s
  • Basic cluster related info:
    • kubectl version 1.24.6

How to reproduce this issue: nginx controller being executed in any node with cgroups 2 and limited by 1 CPU

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 3
  • Comments: 17 (5 by maintainers)

Most upvoted comments

We have currently 400+ ingress controller on newly upgraded 1.25 Azure AKS clusters waiting for a solution. Azure brings Ubuntu 22 with cgroup v2 with AKS 1.25 !

Yes, should push something in the next few days