cri-o: high performace hook: irqbalance: incorrect handling of short affinity mask

What happened?

The high performance hook support the dynamic IRQ balancing thanks some well known annotation. Pods can opt out from IRQ handling exposing this annotation. On pod startup, crio will recompute the irqbalance ban mask and reapply irqbalance to make sure the cpus assigned to these pods will not handle IRQs - being added to the irqbalance ban mask. This works only for guaranteed pods requesting integral CPUs.

To support this feature, crio needs to recompute the cpu ban mask. In turn, crio depends on encoding/hex.DecodeString for some heavy lifting. The function expects to deal with even-sized inbound strings (which are trivial transformations of the irqbalance ban mask).

So crio does left-aligned zero padding of the irq affinity mask at the beginning of computation.

The left-padded mask is then adjusted excluding any relevant cpu, and then inverted. Here lies the problem: the padding character should always be “0” (ASCII zero), not “f”, which is obtained inverting the original correct zero character.

The outcome of this behavior is that stray ‘f’ end up in the IRQ ban list in the irqbalance config file, and they are never removed.

To the best of my knowledge, this happens only when the IRQ affinity mask is a multiple of 4 but not of 8, so on systems with 4, 12, 20… cpus, which will have odd-numbered irq affinity mask reported by the kernel. Machines like this are expected to be quite rare in production environment, but relatively common on CI (especially 4 and 12 cpus)

What did you expect to happen?

cri-o should compute the padding characters correctly using ASCII zero (“0”) and not just trivially invert the computed mask, which however was padded correctly. IOW, the problem is the computation of the inverted mask (= the ban list)

How can we reproduce it (as minimally and precisely as possible)?

run the dynamic IRQ balancing code on a system with 4 cpus, like the openshift CI: see https://github.com/openshift/cluster-node-tuning-operator/pull/396/commits/4821953dc2afd162dd7a02572c7c5cb5c7e1a384 (e2e test added as part of https://github.com/openshift/cluster-node-tuning-operator/pull/396)

Excerpt of a test run which demonstrates the behavior

[performance] Checking IRQBalance settings Verify irqbalance configuration handling 
  Should not overwrite the banned CPU set on tuned restart
  /home/fromani/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/irqbalance.go:70
Aug 11 13:39:53.221: [INFO]: MCP "worker" is targeting 3 node(s)
Aug 11 13:39:53.786: [INFO]: MCP "worker" is targeting 3 node(s)
STEP: verifying worker node "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm"
Aug 11 13:39:54.058: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:39:55.062: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000f0\""
Aug 11 13:39:55.062: [INFO]: banned CPUs: "00000000,000000f0"
Aug 11 13:39:55.062: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" when test begins: {4-7}
Aug 11 13:39:55.200: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:39:56.363: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:39:57.456: [INFO]: using testpod:
{"metadata":{"generateName":"test-","namespace":"performance-addon-operators-testing","creationTimestamp":null,"labels":{"test":""},"annotations":{"cpu-quota.crio.io":"disable","irq-load-balancing.crio.io":"disable"}},"spec":{"containers":[{"name":"test","image":"quay.io/openshift-kni/cnf-tests:4.9","command":["sleep","10h"],"resources":{"limits":{"cpu":"2","memory":"256Mi"}}}],"nodeName":"ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm","runtimeClassName":"performance-perfprof"},"status":{}}
Aug 11 13:40:02.009: [WARNING]: ->  AddedInterface Add eth0 [10.128.2.19/23] from ovn-kubernetes
Aug 11 13:40:02.009: [WARNING]: ->  Pulled Container image "quay.io/openshift-kni/cnf-tests:4.9" already present on machine
Aug 11 13:40:02.009: [WARNING]: ->  Created Created container test
Aug 11 13:40:02.009: [WARNING]: ->  Started Started container test
Aug 11 13:40:02.146: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:40:03.405: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000fa\""
Aug 11 13:40:03.405: [INFO]: banned CPUs: "00000000,000000fa"
Aug 11 13:40:03.405: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" just before the tuned restart: {1,3-7}
STEP: getting a TuneD Pod running on node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
STEP: causing a restart of the tuned pod (deleting the pod) on ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
run command 'oc [delete pod --wait=true -n openshift-cluster-node-tuning-operator tuned-2cb42]':
  out=pod "tuned-2cb42" deleted
  err=
  ret=<nil>
STEP: getting again a TuneD Pod running on node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
STEP: waiting for the TuneD daemon running on node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
STEP: re-verifying worker node "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" after TuneD restart
Aug 11 13:40:13.896: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:40:14.880: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000fa\""
Aug 11 13:40:14.880: [INFO]: banned CPUs: "00000000,000000fa"
Aug 11 13:40:14.880: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" after the tuned restart: {1,3-7}
Aug 11 13:40:14.880: [INFO]: deleting pod "test-w577h"
Aug 11 13:40:47.564: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:40:48.577: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000f0\""
Aug 11 13:40:48.577: [INFO]: banned CPUs: "00000000,000000f0"
Aug 11 13:40:48.577: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" when test ends: {4-7}

note this:

Aug 11 13:40:48.577: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000f0\""

stray unexpected “f” in the banned cpu mask

Anything else we need to know?

this issue is likely present since the introdution of the dynamic IRQ balancing feature.

CRI-O and Kubernetes version

$ crio --version
crio version 1.24.1-11.rhaos4.11.gitb0d2ef3.el8
Version:          1.24.1-11.rhaos4.11.gitb0d2ef3.el8
GoVersion:        go1.18.1
Compiler:         gc
Platform:         linux/amd64
Linkmode:         dynamic
BuildTags:        exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  false
$ kubectl --version
$ oc version
Client Version: 4.10.9
Server Version: 4.11.0-0.nightly-2022-07-24-151159
Kubernetes Version: v1.24.0+9546431

OS version

# On Linux:
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION="411.86.202207210724-0"
VERSION_ID="4.11"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 411.86.202207210724-0 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://docs.openshift.com/container-platform/4.11/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.11"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.11"
OPENSHIFT_VERSION="4.11"
RHEL_VERSION="8.6"
OSTREE_VERSION="411.86.202207210724-0"

$ uname -a
Linux cnfdt12 4.18.0-372.16.1.el8_6.x86_64 #1 SMP Tue Jun 28 03:02:21 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

bare metal

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (6 by maintainers)

Commits related to this issue

Most upvoted comments

/reopen

the issue, AFAIK, still exists