sos: Too many task files slows down SoS

This is separated from #796. There is slowness not only in job preparation but also at the end of every for loop in tasks – my task queue will always halt for about 10 seconds before submitting the next batch of jobs, ie, the overhead of submitting 1000 jobs is 500 seconds. Today I finally can no longer live with the slowness so I removed my ~/.sos/tasks. Out of curiosity I tar zcf before I delete it. It took quite a while to archive it, and generated for me a 1.9G tar.gz file. That might be a 10GB sqlite database if that’s what we will end up doing, and growing … that sounds disturbing.

This may be an important issue for heavy duty work. I’m really curious how other software deal with this problem.