argo-workflows: retry not working in argo workflow v3.5.0-rc2

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

hit retry on a failed workflow. got this error

{"code":6,"message":"Workflow already exists on cluster, use argo retry {name} instead"}: Workflow already exists on cluster, use argo retry {name} instead

Version

v3.5.0-rc2

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

N/A

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}



❯ kubectl logs -n argo deploy/workflow-controller | grep searchapi-integration-tests
Found 3 pods, using pod/workflow-controller-5f46cd5bd9-bb244
...
time="2023-09-29T16:11:08.475Z" level=info msg="Queueing Failed workflow backend/searchapi-integration-tests-27-g44kp for delete in 51h42m52s due to TTL"
time="2023-09-29T16:11:08.714Z" level=info msg="Queueing Failed workflow backend/searchapi-integration-tests-27-vnxbx for delete in 52h59m5s due to TTL"

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

N/A