sos: Slowness preparing jobs
Now I can provide a MWE on this! Here is the SoS script to reproduce:
[1]
input: glob.glob('*.txt'), group_by = 1, concurrent = True
output: dynamic(glob.glob("*.out"))
bash: expand = True
touch {_input:n}.out
To see the problem, first go to an empty folder, type touch {1..50000}.txt in bash to create 50K test input files. Then run sos run test.sos. It takes forever to even get SoS started real processes, at least on my mechanical hard drive. This is a big problem now because previously I can analyze input of up to 50K scale and now I’m just trying to rerun my earlier analysis. Hopefully the bottleneck can be identified. It must be certainly signature related because I’ve not seen this before.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 30 (30 by maintainers)
Flight taking off. Later.