sealed-secrets: Controller crashed: `Observed a panic: "assignment to entry in nil map"`
Which component:
controller version 0.23.0
Describe the bug
The only thing I know is that the controller crashed and the log looked like this:
kubectl -n kube-system logs sealed-secrets-controller-854f5fbffc-9g5xw
controller version: 0.23.0
2023/07/26 10:44:41 Starting sealed-secrets controller version: 0.23.0
Searching for existing private keys
----- sealed-secrets-keygkj4b
HTTP server serving on :8080
Updating trident/backend-ontap-nas
Updating argocd/cluster-config-repo
Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"trident", Name:"backend-ontap-nas", UID:"8012bc39-e03c-492e-8e62-b9c42d18c74a", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"31182844", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
Updating argocd/argocd-notifications-secret
Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"argocd", Name:"cluster-config-repo", UID:"c2f90054-82e2-42a5-bf07-b886967b396a", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"37273431", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
E0726 10:44:41.883408 1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 71 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x17ff680?, 0x1c87950})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0003ce020?})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:49 +0x75
panic({0x17ff680, 0x1c87950})
/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/panic.go:884 +0x213
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).unseal(0xc00015e150, {0x1ca2e10, 0xc000138000}, {0xc0003e61e0, 0x22})
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:354 +0x1545
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).processNextItem(0xc00015e150, {0x1ca2e10, 0xc000138000})
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:260 +0x119
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).runWorker(...)
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:248
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).Run.func1()
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:241 +0x45
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000070800?, {0x1c8ac80, 0xc00027b170}, 0x1, 0xc00051c300)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x1?, 0x0?)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:161
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).Run(0xc00015e150, 0xc00051c300)
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:240 +0x2b8
created by github.com/bitnami-labs/sealed-secrets/pkg/controller.Main
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/main.go:216 +0x48e
E0726 10:44:41.883494 1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 71 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x17ff680?, 0x1c87950})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000695730?})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:49 +0x75
panic({0x17ff680, 0x1c87950})
/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/panic.go:884 +0x213
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0003ce020?})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x17ff680, 0x1c87950})
/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/panic.go:884 +0x213
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).unseal(0xc00015e150, {0x1ca2e10, 0xc000138000}, {0xc0003e61e0, 0x22})
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:354 +0x1545
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).processNextItem(0xc00015e150, {0x1ca2e10, 0xc000138000})
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:260 +0x119
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).runWorker(...)
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:248
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).Run.func1()
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:241 +0x45
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000070800?, {0x1c8ac80, 0xc00027b170}, 0x1, 0xc00051c300)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x1?, 0x0?)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:161
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).Run(0xc00015e150, 0xc00051c300)
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:240 +0x2b8
created by github.com/bitnami-labs/sealed-secrets/pkg/controller.Main
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/main.go:216 +0x48e
panic: assignment to entry in nil map [recovered]
panic: assignment to entry in nil map [recovered]
panic: assignment to entry in nil map
goroutine 71 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000695730?})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x17ff680, 0x1c87950})
/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/panic.go:884 +0x213
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0003ce020?})
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x17ff680, 0x1c87950})
/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/panic.go:884 +0x213
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).unseal(0xc00015e150, {0x1ca2e10, 0xc000138000}, {0xc0003e61e0, 0x22})
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:354 +0x1545
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).processNextItem(0xc00015e150, {0x1ca2e10, 0xc000138000})
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:260 +0x119
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).runWorker(...)
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:248
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).Run.func1()
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:241 +0x45
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000070800?, {0x1c8ac80, 0xc00027b170}, 0x1, 0xc00051c300)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x1?, 0x0?)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.27.3/pkg/util/wait/backoff.go:161
github.com/bitnami-labs/sealed-secrets/pkg/controller.(*Controller).Run(0xc00015e150, 0xc00051c300)
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/controller.go:240 +0x2b8
created by github.com/bitnami-labs/sealed-secrets/pkg/controller.Main
/home/runner/work/sealed-secrets/sealed-secrets/pkg/controller/main.go:216 +0x48e
To Reproduce
I was switching between different versions of my deployment at the time, so I’m not 100% exactly what the order of events were. But I think I did this:
1. Create a good sealed secret.
2. Add a field to the sealed secret using
kubeseal --merge-into, but forget to set the correct namespace (rendering the new field (and possibly the whole secret?) useless).3. Realize the mistake and recreate the secret from scratch, this time using the correct namespace.
4. Crash. Maybe.
EDIT 1: No. I restarted the pod now, and it’s still crashing. If the above was my current state, a restart should have fixed the issue for me. I’ll dig deeper here. My current theory is that the issue is that I’m trying to patch an empty secret. That sounds reasonable considering the error message.
EDIT 2: Here are the real steps to reproduce this issue.
kubectl create secret generic foo
kubectl annotate secret foo 'sealedsecrets.bitnami.com/patch=true'
kubectl create secret generic foo --from-literal=bar=baz --dry-run=client -o json | kubeseal | kubectl apply -f -
Expected behavior
No crash. Maybe a failure state with a clear error message.
Version of Kubernetes:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6", GitCommit:"11902a838028edef305dfe2f96be929bc4d114d8", GitTreeState:"clean", BuildDate:"2023-06-14T09:56:58Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6+rke2r1", GitCommit:"11902a838028edef305dfe2f96be929bc4d114d8", GitTreeState:"clean", BuildDate:"2023-06-14T21:30:18Z", GoVersion:"go1.19.10 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
Additional context
Seems somewhat similar to #285, but that issue is so old that I doubt it is the same bug.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 19 (18 by maintainers)
Hi, @lindhe. We’ve identified the issue and have created a PR #1474 to address it. Thank you very much for reporting this and for your patience.
Thank you for fixing it! 👍
Hi @lindhe we’re going to take a look at the issue since it’s true that It can be easily reproduced.
cc: @alvneiayu @alemorcuq