sos: Slowness deploying tasks
Not sure if it is related to recent changes in sos_targets but the production code seems a lot slower now particularly for task. Here is a MWE:
loci = path('gtex_loci.txt')
[1]
chunks = [x.strip().split() for x in open(f'{loci:a}').readlines() if not x.strip().startswith('#')]
input: for_each = 'chunks'
output: f"{_chunks[-1] if len(_chunks) > 3 else '%s_%s_%s' % (_chunks[0], _chunks[1], _chunks[2])}.pkl", group_with = 'chunks'
task:
bash: expand = True
touch {_output}
Running it with -q none I can see at least it moves along, though from the output it does not feel like well paralleled. Without -q none it appears stuck.
The analyze involves 30K groups in gtex_loci.txt attached here:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 19 (19 by maintainers)
Commits related to this issue
- Process returned substep result before all substeps are submitted #1139 — committed to vatlab/sos by deleted user 6 years ago
- Improve the substep worker allocation #1139 — committed to vatlab/sos by deleted user 6 years ago
- Fix again the logics of waiting for substeps #1139 — committed to vatlab/sos by deleted user 6 years ago
- Implement output group_with more efficiently #1139 — committed to vatlab/sos by deleted user 6 years ago
It could be something very simple to fix, or very hard if we are unlucky. I will try to fix it tomorrow if we are on the lucky side.