kyverno: [Bug] fatal error: concurrent map read and map write

Kyverno Version

1.6.x

Description

This is similar to #1749 that is marked as solved, and it’s happening very often (like evry 20-30 minutes a kyverno pod crashes)

• Kubernetes version: 1.22.3+b93fd35
• Openshift version 4.9.23
• Kyverno version: 1.6.1
• Kyverno pods runnings: 3

I’ve changed the configmap of kyverno to:

• discard openshift-* like namespaces
• enlarged limits for RAM to 16Gb
• added --clientRateLimitQPS=20 --clientRateLimitBurst=50 to commandline (this bug happens without this flags as well)

Error:

fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 1326758 [running]:
runtime.throw({0x26a8294, 0xc0011babd0})
        /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0011baba0 sp=0xc0011bab70 pc=0x435b51
runtime.mapassign_faststr(0x22c4c00, 0xc0026f2240, {0x26c98cd, 0x21})
        /usr/local/go/src/runtime/map_faststr.go:294 +0x38b fp=0xc0011bac08 sp=0xc0011baba0 pc=0x4130eb
github.com/kyverno/kyverno/pkg/webhooks.(*WebhookServer).updateAnnotationInGR(0xc002d49ce0, 0xc003c98240, {{0x3150f78, 0xc00004c5a0}, 0xc0019d56b0})
        /kyverno/pkg/webhooks/generation.go:210 +0xc5 fp=0xc0011bacd8 sp=0xc0011bac08 pc=0x200c7e5
github.com/kyverno/kyverno/pkg/webhooks.(*WebhookServer).handleUpdateGenerateTargetResource(0xc002d49ce0, 0xc0008d1f48, {0x26bdf83, 0x1d, 0xc000dbcea8}, 0x30b2948, {{0x3150f78, 0xc00004c5a0}, 0xc000dbcf98})
        /kyverno/pkg/webhooks/generation.go:277 +0x4ad fp=0xc0011bbe60 sp=0xc0011bacd8 pc=0x200cead
github.com/kyverno/kyverno/pkg/webhooks.(*WebhookServer).handleUpdatesForGenerateRules(0xc002d49ce0, 0xc006c20000, {0x465cc48, 0x0, 0x0})
        /kyverno/pkg/webhooks/generation.go:167 +0x9d4 fp=0xc0011bbfa8 sp=0xc0011bbe60 pc=0x200c174
github.com/kyverno/kyverno/pkg/webhooks.(*WebhookServer).resourceValidation·dwrap·16()
        /kyverno/pkg/webhooks/server.go:511 +0x36 fp=0xc0011bbfe0 sp=0xc0011bbfa8 pc=0x201a1b6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0011bbfe8 sp=0xc0011bbfe0 pc=0x468ea1
created by github.com/kyverno/kyverno/pkg/webhooks.(*WebhookServer).resourceValidation
        /kyverno/pkg/webhooks/server.go:511 +0x8bf

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.