law: HTCondor Jobs are failing when cache_task_completion = True
When the following is set in the luigi config:
[worker]
cache_task_completion: True
(This is e.g. recommended for running large local WFs.)
The following errors are appearing in (some) remote HTCondor jobs. I guess one single job per Worker Node is starting fine:
Process SyncManager-1:
Traceback (most recent call last):
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/managers.py", line 583, in _run_server
server = cls._Server(registry, address, authkey, serializer)
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/managers.py", line 156, in __init__
self.listener = Listener(address=address, backlog=16)
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 453, in __init__
self._listener = SocketListener(address, family, backlog)
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 596, in __init__
self._socket.bind(address)
OSError: [Errno 98] Address already in use
ERROR: luigi-interface - Uncaught exception in luigi
Traceback (most recent call last):
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/retcodes.py", line 75, in run_with_retcodes
worker = luigi.interface._run(argv).worker
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/interface.py", line 217, in _run
return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)
File "/srv/job_VG35uuMJ5QCe/repo/law/law/patches.py", line 94, in _schedule_and_run
return _schedule_and_run_orig(*args, **kwargs)
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/interface.py", line 168, in _schedule_and_run
worker = worker_scheduler_factory.create_worker(
File "/srv/job_VG35uuMJ5QCe/repo/law/law/patches.py", line 282, in create_worker
worker = luigi.worker.Worker(scheduler=scheduler, worker_processes=worker_processes,
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/worker.py", line 609, in __init__
self._task_completion_cache = multiprocessing.Manager().dict()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/context.py", line 57, in Manager
m.start()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/managers.py", line 558, in start
self._address = reader.recv()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError
ERROR: Uncaught exception in luigi
Traceback (most recent call last):
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/retcodes.py", line 75, in run_with_retcodes
worker = luigi.interface._run(argv).worker
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/interface.py", line 217, in _run
return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)
File "/srv/job_VG35uuMJ5QCe/repo/law/law/patches.py", line 94, in _schedule_and_run
return _schedule_and_run_orig(*args, **kwargs)
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/interface.py", line 168, in _schedule_and_run
worker = worker_scheduler_factory.create_worker(
File "/srv/job_VG35uuMJ5QCe/repo/law/law/patches.py", line 282, in create_worker
worker = luigi.worker.Worker(scheduler=scheduler, worker_processes=worker_processes,
File "/srv/job_VG35uuMJ5QCe/repo/luigi/luigi/worker.py", line 609, in __init__
self._task_completion_cache = multiprocessing.Manager().dict()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/context.py", line 57, in Manager
m.start()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/managers.py", line 558, in start
self._address = reader.recv()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc12-opt/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError
task exit code: 60
Disabling the task completion cache fixes this issue.
I’m using law version 0.1.20
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (15 by maintainers)
The same is true for me. I can’t really debug this, as I need a lot of free slots (>1k) in our batch system at the same time for this issue to start appearing. I might get the opportunity in the coming weeks to test submission to a new HPC.
Thank you for looking into it. If I find the time in the coming weeks to test, I will let you know. I’m quite busy with other stuff, atm. But I think that this commit will fix the issue.