toil: Pickling error when running Cactus on larger input

I’m trying to run Cactus on a larger dataset on kubernetes, and this is the latest error PicklingError: Can't pickle <class 'toil.BotoCredentialAdapter'>: it's not found as toil.BotoCredentialAdapter

Is this a Toil problem?

86359de7-b94e-47bb-99e2-016fc4314469    Running command catchsegv 'cactus_blast_chunkSequences' 'INFO' '25000000' '10000' 'tsJ4ChA/tmp_DsPa0hrmcW' 'tmp4bFJbY.tmp'
86359de7-b94e-47bb-99e2-016fc4314469    Set log level to INFO
86359de7-b94e-47bb-99e2-016fc4314469    INFO:toil-rt:2019-12-05 23:45:54.502858: Successfully ran the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpdvB9f_/9f512bca-9683-4b6d-962a-dd060f2d5983:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpdvB9f_/9f512bca-9683-4b6d-962a-dd060f2d5983/tjMQD3l/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 tsJ4ChA/tmp_DsPa0hrmcW tmp4bFJbY.tmp" in 49.5165989399 seconds
86359de7-b94e-47bb-99e2-016fc4314469    INFO:toil.lib.bioio:Made the list of blasts
86359de7-b94e-47bb-99e2-016fc4314469    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/venv2/local/lib/python2.7/site-packages', name='cactus.blast.blast', fromVirtualEnv=True).
86359de7-b94e-47bb-99e2-016fc4314469    Traceback (most recent call last):
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 367, in workerScript
86359de7-b94e-47bb-99e2-016fc4314469        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
86359de7-b94e-47bb-99e2-016fc4314469      File "/tmp/tmpgbMfBk/3a536290e528005ce049de97a4aca1e6/cactus/shared/common.py", line 1201, in _runner
86359de7-b94e-47bb-99e2-016fc4314469        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1399, in _runner
86359de7-b94e-47bb-99e2-016fc4314469        self._serialiseExistingJob(jobGraph, jobStore, returnValues)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1308, in _serialiseExistingJob
86359de7-b94e-47bb-99e2-016fc4314469        self._serialiseJobGraph(jobGraph, jobStore, returnValues, False)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1277, in _serialiseJobGraph
86359de7-b94e-47bb-99e2-016fc4314469        job._serialiseJob(jobStore, jobsToJobGraphs, jobGraph)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1155, in _serialiseJob
86359de7-b94e-47bb-99e2-016fc4314469        pickle.dump(self, fileHandle, pickle.HIGHEST_PROTOCOL)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 1376, in dump
86359de7-b94e-47bb-99e2-016fc4314469        Pickler(file, protocol).dump(obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 224, in dump
86359de7-b94e-47bb-99e2-016fc4314469        self.save(obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_appends(iter(obj))
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
86359de7-b94e-47bb-99e2-016fc4314469        save(x)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_appends(iter(obj))
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
86359de7-b94e-47bb-99e2-016fc4314469        save(x)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_appends(iter(obj))
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
86359de7-b94e-47bb-99e2-016fc4314469        save(x)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(cls)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 1299, in save_type
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_global(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 754, in save_global
86359de7-b94e-47bb-99e2-016fc4314469        (obj, module, name))
86359de7-b94e-47bb-99e2-016fc4314469    PicklingError: Can't pickle <class 'toil.BotoCredentialAdapter'>: it's not found as toil.BotoCredentialAdapter
86359de7-b94e-47bb-99e2-016fc4314469    ERROR:toil.worker:Exiting the worker because of a failed job on host root-toil-d12ffe0b-ef0d-42cf-90f5-c2b0060670de-3188-s6m95
86359de7-b94e-47bb-99e2-016fc4314469    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'BlastSequencesAgainstEachOther' 86359de7-b94e-47bb-99e2-016fc4314469 with ID 86359de7-b94e-47bb-99e2-016fc4314469 to 2
Issued job 'BlastSequencesAgainstEachOther' 86359de7-b94e-47bb-99e2-016fc4314469 with job batch system ID: 3190 and cores: 1, disk: 85.8 G, and memory: 16.0 G
2019-12-06 00:07:33.322926: Running the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 tGGRcR4/tmp_b58alVaJnN tmpdrqoB4.tmp tmpFXOVtZ.tmp"
2019-12-06 00:09:06.558278: Successfully ran the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 tGGRcR4/tmp_b58alVaJnN tmpdrqoB4.tmp tmpFXOVtZ.tmp" in 92.7297759056 seconds
2019-12-06 00:09:06.559814: Running the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 txNK42h/tmp_Z4yHYyAACP tmpvqeGbA.tmp"
2019-12-06 00:09:54.259745: Successfully ran the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 txNK42h/tmp_Z4yHYyAACP tmpvqeGbA.tmp" in 47.1937949657 seconds
Reissued any over long jobs

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-466