keda: Scaled object panics in deref of nil target ref

Report

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x167ddc5]

goroutine 160951 [running]: github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestScale(0xc00002c780, {0x40bf968, 0xc002df8e80}, 0xc001f3ca00, 0x0, 0x0) /workspace/pkg/scaling/executor/scale_scaledobjects.go:45 +0x225 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc000dfefc0, {0x40bf968, 0xc002df8e80}, {0x3873b80?, 0xc001f3ca00?}, {0x40accf8, 0xc004783c18}) /workspace/pkg/scaling/scale_handler.go:361 +0x559 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0xc000dfefc0, {0x40bf968, 0xc002df8e80}, 0xc002b2b540, {0x3873b80, 0xc001f3ca00}, {0x40accf8, 0xc004783c18}) /workspace/pkg/scaling/scale_handler.go:162 +0x32a created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject /workspace/pkg/scaling/scale_handler.go:118 +0x71d

Expected Behavior

The code should check nil here or further up. My guess is the target ref is in flux (a controller may have torn it down).

Actual Behavior

The nil GVKR is deref’d to a panic. Controller restarts.

Steps to Reproduce the Problem

I am not able to reliably reproduce the condition. We do see it quite often but we’ve also got a lot of controllers tinkering with state.

Logs from KEDA operator

2023-01-26T01:25:16Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"assure-anilm-combined","namespace":"assure-anil-dev"}, "namespace": "assure-anil-dev", "name": "assure-anilm-combined", "reconcileID": "e71c917c-a2b0-42e5-a98c-c2845e3b8ed6"}
2023-01-26T01:25:17Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"dmacapd-qinyux-capd-combined","namespace":"dmacapd-dev"}, "namespace": "dmacapd-dev", "name": "dmacapd-qinyux-capd-combined", "reconcileID": "03287421-90f0-483e-bce7-f77a28c2c72e"}
2023-01-26T01:25:17Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"musesmoke-combined","namespace":"muse-dev"}, "namespace": "muse-dev", "name": "musesmoke-combined", "reconcileID": "8fabfffa-b0f1-4139-a34c-9f2bc5e6f304"}
2023-01-26T01:25:17Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"dmacapd-dev-willb-capd-combined","namespace":"dmacapd-dev"}, "namespace": "dmacapd-dev", "name": "dmacapd-dev-willb-capd-combined", "reconcileID": "965c041b-c2cc-46d7-93b1-d7bf3520469d"}
2023-01-26T01:25:17Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"dma-qinyux-capd-combined","namespace":"dma-dev"}, "namespace": "dma-dev", "name": "dma-qinyux-capd-combined", "reconcileID": "4f6f9d91-53c6-4c0c-864f-07c4b3b5435e"}
2023-01-26T01:25:21Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"enrcdi-dev-clu-clu360-combined","namespace":"enrcdi-dev"}, "namespace": "enrcdi-dev", "name": "enrcdi-dev-clu-clu360-combined", "reconcileID": "0d9bb0c2-42a6-4afe-89e4-d843e507c31b"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x167ddc5]

goroutine 160951 [running]:
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestScale(0xc00002c780, {0x40bf968, 0xc002df8e80}, 0xc001f3ca00, 0x0, 0x0)
        /workspace/pkg/scaling/executor/scale_scaledobjects.go:45 +0x225
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc000dfefc0, {0x40bf968, 0xc002df8e80}, {0x3873b80?, 0xc001f3ca00?}, {0x40accf8, 0xc004783c18})
        /workspace/pkg/scaling/scale_handler.go:361 +0x559
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0xc000dfefc0, {0x40bf968, 0xc002df8e80}, 0xc002b2b540, {0x3873b80, 0xc001f3ca00}, {0x40accf8, 0xc004783c18})
        /workspace/pkg/scaling/scale_handler.go:162 +0x32a
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject
        /workspace/pkg/scaling/scale_handler.go:118 +0x71d

KEDA Version

2.9.2

Kubernetes Version

1.24

Platform

Microsoft Azure

Scaler Details

Prometheus

Anything else?

Thanks!