tailscale: Kubernetes-operator - Cannot remove machine

What is the issue?

I have the k8s tailscale-operator setup in a cluster and am having trouble removing a machine and it’s ts-* pods from tailscale/k8s.

I annotated the default.kubernetes.svc resource in order to talk to the k8s cluster via kubectl over tailscale. This method worked, but I then removed the annotations so I could try the auth proxy method. Removing the annotations did not remove the machine from tailscale and did not remove the ts-* pods that were created. When I delete the ts-* pods they just get created again automatically.

Another thing I tried was adding the annotations back but setting tailscale.com/expose to "false" but that does not seem to work. In fact when I do that the operator pod throws this error:

{"level":"error","ts":"2023-05-23T18:42:16Z","msg":"Reconciler error","controller":"service","controllerGroup":"","controllerKind":"Service","Service":"name":"kubernetes","namespace":"default"},"namespace":"default","name":"kubernetes","reconcileID":"d5826b40-f63f-4c97-8f86-ae12f0f231ce","error":"getting statefulset: found multiple matching *v1.StatefulSet objects","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}

So, how do I correctly remove a k8s service from the tailscale-operator?

Thanks.

Steps to reproduce

• Deploy tailscale operator to k8s.
• Annotate the default.kubernetes.svc with tailscale.com/expose: "true".
• Remove the tailscale.com/expose annotation from default.kubernetes.svc.
• Witness that the machine is still in the tailscale admin interface and the ts-* pod still exists.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Linux (5.15.0-8.91.4.1.el8uek.x86_64)

Tailscale version

1.41.55

Other software

OKE, Oracle linux as workers.

Bug report

No response