argo-workflows: mutex ignored if using semaphore at same top level

Pre-requisites

I have double-checked my configuration
I can confirm the issues exists when I tested with :latest
I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

logs show nothing about mutex and i verified 2 workflows were able to run at same time

logs show semaphore is working.

then i tried removing semaphore and mutex worked. so seems they don’t work together?

https://github.com/argoproj/argo-workflows/blob/master/workflow/sync/sync_manager.go#L280-L283 seems like it doesn’t loop through both types of synchronization cc @sarabala1979

Version

3.4.11

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: redact-
spec:
  serviceAccountName: argo-workflows-server
  onExit: exit-handler
  entrypoint: flow
  synchronization:
    semaphore:
      configMapKeyRef:
        name: argowf-semaphores
        key: redact
    mutex:
      name: "{{= sprig.coalesce('something','catch') }}"
...can't share the rest



### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow’s wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

About this issue

Original URL
State: open
Created 9 months ago
Comments: 16 (14 by maintainers)

Most upvoted comments

why deadlock? it should only consume the mutex if there is room to fit under the semaphore limit too

tooptoop4 on Oct 1, 2023

the only thing with multi semaphores is it means i need to pre-define all the different tablenames in a configmap? whereas mutex accepts dynamic string that does not need to be pre-defined in configmap

tooptoop4 on Sep 23, 2023

@jakkubu @tooptoop4 Here is the workaround for having multiple locks

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: redact-
spec:
  serviceAccountName: argo-workflows-server
  onExit: exit-handler
  entrypoint: start
  synchronization:
    semaphore:
      configMapKeyRef:
        name: argowf-semaphores
        key: redact
  templates:
    - name: start
      synchronization:
        mutex:
          name: "{{= sprig.coalesce('something','catch') }}"
      steps:
        - name: flow
          template: flow

sarabala1979 on Dec 12, 2023

so i use parallelism to not scale up too many workers i use a lower semaphore on my sensor so that cronworkflows can have some slots to run (ie cronworkflows will be able to fill parallelism-semaphore) then i want mutex at a per table level lock

tooptoop4 on Sep 23, 2023