sos: Rebuilding signature slows down in time

This is an issue we’ve discussed before a couple of times: I used to complain about slowness verifying signature and rebuilding them for large number of tasks. Lots of improvements have been made already. But still,

INFO: Step susie_bhat_1 (index=11945) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11946) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11947) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11948) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11949) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11950) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11951) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11952) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11955) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11956) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11957) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11959) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11960) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11962) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11963) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11965) is ignored with signature constructed
INFO: Step susie_bhat_1 (index=11966) is ignored with signature constructed

This started at perhaps ~10 signatuers per sec but currently it is going at about 2 signatures per sec. At this speed verifying 34K jobs will take quite a few hours to perhaps run only a handful of failed jobs. Then the next time even if I run without -s build, to maybe pick up a couple of jobs that still failed, the process might be another few hours.

This is not the most urgent issue compared to the others we are currently dealing with. I’m just posting it here for a record.