sos: Slowness deploying tasks

Not sure if it is related to recent changes in sos_targets but the production code seems a lot slower now particularly for task. Here is a MWE:

loci = path('gtex_loci.txt')

[1]

chunks = [x.strip().split() for x in open(f'{loci:a}').readlines() if not x.strip().startswith('#')]

input: for_each = 'chunks'
output: f"{_chunks[-1] if len(_chunks) > 3 else '%s_%s_%s' % (_chunks[0], _chunks[1], _chunks[2])}.pkl", group_with = 'chunks'
task:
bash: expand = True
   touch {_output}

Running it with -q none I can see at least it moves along, though from the output it does not feel like well paralleled. Without -q none it appears stuck.

The analyze involves 30K groups in gtex_loci.txt attached here:

gtex_loci.txt

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19 (19 by maintainers)

Commits related to this issue

Most upvoted comments

It could be something very simple to fix, or very hard if we are unlucky. I will try to fix it tomorrow if we are on the lucky side.