pipeline: Old resource deletion hangs when `alpha` flag is off

Expected Behavior

Users should be able to delete any taskrun object using kubectl delete tr --all no matter if alpha feature flag is enabled.

Actual Behavior

After a new pipeline controller is deployed with alpha feature flag off, kubectl delete tr --all command hangs on the taskruns that worked fine under previous controller with alpha feature flag enabled.

Steps to Reproduce the Problem

Clone the pipeline repo. Change stable feature flag to alpha in this file.
Run ko apply -R -f config/
After the new controller is running, run the example yaml. You can see the pipelinerun will be successfully executed.
Back to step 1, change the flag back to stable
Rerun step 2
Run kubectl delete tr --all to delete all taskruns that were previously created. You will see the deletion is hanging.

Additional Info

Kubernetes version:

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12-gke.1500", GitCommit:"6c11aec6ce32cf0d66a2631eed2eb49dd65c89f8", GitTreeState:"clean", BuildDate:"2022-05-11T09:25:37Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Tekton Pipeline version:

Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'
```
latest main
```

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 24 (20 by maintainers)

Most upvoted comments

It’s on an UPDATE event, so it does validation and fails there. Chains tries to update the object, but cannot because some fields are behind alpha and stable is set. I am not sure why it would block deletion, except maybe if it fails to sign, put the object into the queue to retry, making it somehow un-deletable ?

^ @vdemeester this is correct. Chains uses a finalizer to make sure that Run objects don’t disappear before they can be attested. Normally, the finalizer is removed once chains has processed the object, but because the alpha flag was disabled, chains can’t complete its reconcile (probably when trying to add the chains.tekton.dev/signed annotation) so the finalizer is never removed.

I don’t think chains is unique here - any client that tries to update a beta object with alpha fields would fail in a similar way. This kind of unexpected behavior was what I was worried about in https://github.com/tektoncd/pipeline/pull/5005#issuecomment-1170484289 (cc @lbernick @afrittoli)

wlynch on Jul 28, 2022