sos: Slowness preparing jobs

Now I can provide a MWE on this! Here is the SoS script to reproduce:

[1]
input: glob.glob('*.txt'), group_by = 1, concurrent = True
output: dynamic(glob.glob("*.out"))
bash: expand = True
  touch {_input:n}.out

To see the problem, first go to an empty folder, type touch {1..50000}.txt in bash to create 50K test input files. Then run sos run test.sos. It takes forever to even get SoS started real processes, at least on my mechanical hard drive. This is a big problem now because previously I can analyze input of up to 50K scale and now I’m just trying to rerun my earlier analysis. Hopefully the bottleneck can be identified. It must be certainly signature related because I’ve not seen this before.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 30 (30 by maintainers)

Most upvoted comments

Flight taking off. Later.