argo-workflows: workflow metrics is not correct in v2.12.0-rc2 and latest master branch

Summary

What happened/what you expected to happen?

I have enabled argo metrics and scrape those metrics using prometheus, but find that the default argo metrics argo_workflows_count{status=“Pending”} is increasing continusly and argo_workflows_count{status=“Running”} is decreasing continusly.

Diagnostics

What Kubernetes provider are you using? v1.14

What version of Argo Workflows are you running?

1. v2.12.0-rc2
2. latest master branch

Paste a workflow that reproduces the bug, including status:
kubectl get wf -o yaml ${workflow} 

any workflow spec will be ok since it is a bug about argo metrics

Paste the logs from the workflow controller:
kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name) | grep ${workflow}

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.