cri-o: high performace hook: irqbalance: incorrect handling of short affinity mask
What happened?
The high performance hook support the dynamic IRQ balancing thanks some well known annotation. Pods can opt out from IRQ handling exposing this annotation. On pod startup, crio will recompute the irqbalance ban mask and reapply irqbalance to make sure the cpus assigned to these pods will not handle IRQs - being added to the irqbalance ban mask. This works only for guaranteed pods requesting integral CPUs.
To support this feature, crio needs to recompute the cpu ban mask. In turn, crio depends on encoding/hex.DecodeString for some heavy lifting. The function expects to deal with even-sized inbound strings (which are trivial transformations of the irqbalance ban mask).
So crio does left-aligned zero padding of the irq affinity mask at the beginning of computation.
The left-padded mask is then adjusted excluding any relevant cpu, and then inverted. Here lies the problem: the padding character should always be “0” (ASCII zero), not “f”, which is obtained inverting the original correct zero character.
The outcome of this behavior is that stray ‘f’ end up in the IRQ ban list in the irqbalance config file, and they are never removed.
To the best of my knowledge, this happens only when the IRQ affinity mask is a multiple of 4 but not of 8, so on systems with 4, 12, 20… cpus, which will have odd-numbered irq affinity mask reported by the kernel. Machines like this are expected to be quite rare in production environment, but relatively common on CI (especially 4 and 12 cpus)
What did you expect to happen?
cri-o should compute the padding characters correctly using ASCII zero (“0”) and not just trivially invert the computed mask, which however was padded correctly. IOW, the problem is the computation of the inverted mask (= the ban list)
How can we reproduce it (as minimally and precisely as possible)?
run the dynamic IRQ balancing code on a system with 4 cpus, like the openshift CI: see https://github.com/openshift/cluster-node-tuning-operator/pull/396/commits/4821953dc2afd162dd7a02572c7c5cb5c7e1a384 (e2e test added as part of https://github.com/openshift/cluster-node-tuning-operator/pull/396)
Excerpt of a test run which demonstrates the behavior
[performance] Checking IRQBalance settings Verify irqbalance configuration handling
Should not overwrite the banned CPU set on tuned restart
/home/fromani/go/src/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/irqbalance.go:70
Aug 11 13:39:53.221: [INFO]: MCP "worker" is targeting 3 node(s)
Aug 11 13:39:53.786: [INFO]: MCP "worker" is targeting 3 node(s)
STEP: verifying worker node "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm"
Aug 11 13:39:54.058: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:39:55.062: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000f0\""
Aug 11 13:39:55.062: [INFO]: banned CPUs: "00000000,000000f0"
Aug 11 13:39:55.062: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" when test begins: {4-7}
Aug 11 13:39:55.200: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:39:56.363: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:39:57.456: [INFO]: using testpod:
{"metadata":{"generateName":"test-","namespace":"performance-addon-operators-testing","creationTimestamp":null,"labels":{"test":""},"annotations":{"cpu-quota.crio.io":"disable","irq-load-balancing.crio.io":"disable"}},"spec":{"containers":[{"name":"test","image":"quay.io/openshift-kni/cnf-tests:4.9","command":["sleep","10h"],"resources":{"limits":{"cpu":"2","memory":"256Mi"}}}],"nodeName":"ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm","runtimeClassName":"performance-perfprof"},"status":{}}
Aug 11 13:40:02.009: [WARNING]: -> AddedInterface Add eth0 [10.128.2.19/23] from ovn-kubernetes
Aug 11 13:40:02.009: [WARNING]: -> Pulled Container image "quay.io/openshift-kni/cnf-tests:4.9" already present on machine
Aug 11 13:40:02.009: [WARNING]: -> Created Created container test
Aug 11 13:40:02.009: [WARNING]: -> Started Started container test
Aug 11 13:40:02.146: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:40:03.405: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000fa\""
Aug 11 13:40:03.405: [INFO]: banned CPUs: "00000000,000000fa"
Aug 11 13:40:03.405: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" just before the tuned restart: {1,3-7}
STEP: getting a TuneD Pod running on node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
STEP: causing a restart of the tuned pod (deleting the pod) on ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
run command 'oc [delete pod --wait=true -n openshift-cluster-node-tuning-operator tuned-2cb42]':
out=pod "tuned-2cb42" deleted
err=
ret=<nil>
STEP: getting again a TuneD Pod running on node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
STEP: waiting for the TuneD daemon running on node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
STEP: re-verifying worker node "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" after TuneD restart
Aug 11 13:40:13.896: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:40:14.880: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000fa\""
Aug 11 13:40:14.880: [INFO]: banned CPUs: "00000000,000000fa"
Aug 11 13:40:14.880: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" after the tuned restart: {1,3-7}
Aug 11 13:40:14.880: [INFO]: deleting pod "test-w577h"
Aug 11 13:40:47.564: [INFO]: found mcd machine-config-daemon-s5g7t for node ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm
Aug 11 13:40:48.577: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000f0\""
Aug 11 13:40:48.577: [INFO]: banned CPUs: "00000000,000000f0"
Aug 11 13:40:48.577: [INFO]: banned CPUs on "ci-ln-2h3mjpb-72292-2wnt7-worker-a-zbxpm" when test ends: {4-7}
note this:
Aug 11 13:40:48.577: [INFO]: banned CPUs setting: "IRQBALANCE_BANNED_CPUS=\"00000000,000000f0\""
stray unexpected “f” in the banned cpu mask
Anything else we need to know?
this issue is likely present since the introdution of the dynamic IRQ balancing feature.
CRI-O and Kubernetes version
$ crio --version
crio version 1.24.1-11.rhaos4.11.gitb0d2ef3.el8
Version: 1.24.1-11.rhaos4.11.gitb0d2ef3.el8
GoVersion: go1.18.1
Compiler: gc
Platform: linux/amd64
Linkmode: dynamic
BuildTags: exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp, selinux
SeccompEnabled: true
AppArmorEnabled: false
$ kubectl --version
$ oc version
Client Version: 4.10.9
Server Version: 4.11.0-0.nightly-2022-07-24-151159
Kubernetes Version: v1.24.0+9546431
OS version
# On Linux:
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION="411.86.202207210724-0"
VERSION_ID="4.11"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 411.86.202207210724-0 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://docs.openshift.com/container-platform/4.11/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.11"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.11"
OPENSHIFT_VERSION="4.11"
RHEL_VERSION="8.6"
OSTREE_VERSION="411.86.202207210724-0"
$ uname -a
Linux cnfdt12 4.18.0-372.16.1.el8_6.x86_64 #1 SMP Tue Jun 28 03:02:21 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Additional environment details (AWS, VirtualBox, physical, etc.)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (6 by maintainers)
Commits related to this issue
- irqbalance: fix padding of odd-sized cpu mask The high performance hook support the dynamic IRQ balancing thanks some well known annotation. Pods can opt out from IRQ handling exposing this annotatio... — committed to ffromani/cri-o by ffromani 4 years ago
- perfprof: functests: utils: fix cpu mask padding Testing the irqbalance handling on tuned restart highlighted a crio bug when handling odd-numbered cpu affinity masks. We expect this bug to have litt... — committed to ffromani/cluster-node-tuning-operator by ffromani 4 years ago
- irqbalance: fix padding of odd-sized cpu mask The high performance hook support the dynamic IRQ balancing thanks some well known annotation. Pods can opt out from IRQ handling exposing this annotatio... — committed to ffromani/cri-o by ffromani 4 years ago
- perfprof: functests: utils: fix cpu mask padding Testing the irqbalance handling on tuned restart highlighted a crio bug when handling odd-numbered cpu affinity masks. We expect this bug to have litt... — committed to ffromani/cluster-node-tuning-operator by ffromani 4 years ago
- Bug 2105123: tuned: disable irqbalance (#396) * test: perfprof: utils: make sure to unquote cpus Make sure to unquote the cpumask output to prevent false negatives Signed-off-by: Francesco Romani <... — committed to openshift/cluster-node-tuning-operator by ffromani 4 years ago
- perfprof: functests: utils: fix cpu mask padding Testing the irqbalance handling on tuned restart highlighted a crio bug when handling odd-numbered cpu affinity masks. We expect this bug to have litt... — committed to openshift-cherrypick-robot/cluster-node-tuning-operator by ffromani 4 years ago
- e2e: perfprof: crio mask fix workaround part 2 In commit c5cf0bd33c5afae7695fd86d08c62aef269105d8 we added a workaround for the incorrect crio mask padding (see: https://github.com/cri-o/cri-o/issues... — committed to ffromani/cluster-node-tuning-operator by ffromani 4 years ago
- e2e: perfprof: crio mask fix workaround part 2 In commit c5cf0bd33c5afae7695fd86d08c62aef269105d8 we added a workaround for the incorrect crio mask padding (see: https://github.com/cri-o/cri-o/issues... — committed to ffromani/cluster-node-tuning-operator by ffromani 4 years ago
- e2e: perfprof: crio mask fix workaround part 2 In commit c5cf0bd33c5afae7695fd86d08c62aef269105d8 we added a workaround for the incorrect crio mask padding (see: https://github.com/cri-o/cri-o/issues... — committed to ffromani/cluster-node-tuning-operator by ffromani 4 years ago
- e2e: perfprof: unbreak the e2e-gcp PAO lane (#438) * podsecurity: disable OCP label sync Per last OCP recommendation (internal doc) is not sufficient to declare the pod security labels, we also need... — committed to openshift/cluster-node-tuning-operator by ffromani 4 years ago
- [release-4.11] OCPBUGS-263: tuned: disable irqbalance (#435) * test: perfprof: utils: make sure to unquote cpus Make sure to unquote the cpumask output to prevent false negatives Signed-off-by: Fra... — committed to openshift/cluster-node-tuning-operator by openshift-cherrypick-robot 4 years ago
- e2e: perfprof: crio mask fix workaround part 2 In commit c5cf0bd33c5afae7695fd86d08c62aef269105d8 we added a workaround for the incorrect crio mask padding (see: https://github.com/cri-o/cri-o/issues... — committed to marioferh/cluster-node-tuning-operator by ffromani 4 years ago
- OCPBUGS-1131: [release-4.11] e2e: perfprof: unbreak the e2e-gcp PAO lane (#467) * podsecurity: disable OCP label sync Per last OCP recommendation (internal doc) is not sufficient to declare the pod ... — committed to openshift/cluster-node-tuning-operator by marioferh 4 years ago
- Bug 2105123: tuned: disable irqbalance (#396) * test: perfprof: utils: make sure to unquote cpus Make sure to unquote the cpumask output to prevent false negatives Signed-off-by: Francesco Romani <... — committed to IlyaTyomkin/cluster-node-tuning-operator by ffromani 4 years ago
- e2e: perfprof: unbreak the e2e-gcp PAO lane (#438) * podsecurity: disable OCP label sync Per last OCP recommendation (internal doc) is not sufficient to declare the pod security labels, we also need... — committed to IlyaTyomkin/cluster-node-tuning-operator by ffromani 4 years ago
- Bug 2105123: tuned: disable irqbalance (#396) * test: perfprof: utils: make sure to unquote cpus Make sure to unquote the cpumask output to prevent false negatives Signed-off-by: Francesco Romani <... — committed to IlyaTyomkin/cluster-node-tuning-operator by ffromani 4 years ago
- e2e: perfprof: unbreak the e2e-gcp PAO lane (#438) * podsecurity: disable OCP label sync Per last OCP recommendation (internal doc) is not sufficient to declare the pod security labels, we also need... — committed to IlyaTyomkin/cluster-node-tuning-operator by ffromani 4 years ago
- irqbalance: fix padding of odd-sized cpu mask The high performance hook support the dynamic IRQ balancing thanks some well known annotation. Pods can opt out from IRQ handling exposing this annotatio... — committed to ffromani/cri-o by ffromani 4 years ago
- irqbalance: fix padding of odd-sized cpu mask The high performance hook support the dynamic IRQ balancing thanks some well known annotation. Pods can opt out from IRQ handling exposing this annotatio... — committed to ffromani/cri-o by ffromani 4 years ago
/reopen
the issue, AFAIK, still exists