wandb: [CLI]: BrokenPipeError: [Errno 32] Broken pipe

Bub description

Training and logging run fine; however, at the end of the process, the wandb outputs the error message below.

wandb: Waiting for W&B process to finish... (success).
wandb: \ 0.014 MB of 0.014 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb:               epoch ▁▁▁▁▁▁▁▁▃▃▃▃▃▃▃▃▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆████████
wandb:          train_loss █▄▃▂▂▃▄▃▁▂▂▄▄▂▃▂▁▂▁▁▁▁▁▁▃▁▁▁▁▂▁▁▁▁▁▅▁▁▁▃
wandb: trainer/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:          val_Mac-F1 ▁▄▆▅█
wandb:          val_Mic-F1 ▁▄▇▇█
wandb:          val_Wei-F1 ▁▅▇▇█
wandb:            val_loss █▂▁▄▆
wandb: 
wandb: Run summary:
wandb:               epoch 4
wandb:          train_loss 0.47728
wandb: trainer/global_step 5534
wandb:          val_Mac-F1 0.70413
wandb:          val_Mic-F1 0.88889
wandb:          val_Wei-F1 0.93459
wandb:            val_loss 0.46428
wandb: 
wandb: 🚀 View run BERT_WEBKB_0_exp at: https://wandb.ai/celsofranca/lightning_logs/runs/1qq5guxx
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: /tmp/wandb/run-20231012_134638-1qq5guxx/logs
Exception in thread Exception in thread IntMsgThrNetStatThr:
:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 299, in check_internal_messages
    self._target(*self._args, **self._kwargs)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 267, in check_network_status
    self._loop_check_status(
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 223, in _loop_check_status
    self._loop_check_status(
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 223, in _loop_check_status
    local_handle = request()
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 743, in deliver_internal_messages
    return self._deliver_internal_messages(internal_message)
    local_handle = request()
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 481, in _deliver_internal_messages
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 735, in deliver_network_status
    return self._deliver_record(record)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 428, in _deliver_record
    return self._deliver_network_status(status)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 475, in _deliver_network_status
    handle = mailbox._deliver_record(record, interface=self)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    return self._deliver_record(record)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 428, in _deliver_record
    interface._publish(record)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    handle = mailbox._deliver_record(record, interface=self)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    self._sock_client.send_record_publish(record)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    interface._publish(record)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self.send_server_request(server_req)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._sock_client.send_record_publish(record)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self._send_message(msg)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self.send_server_request(server_req)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._sendall_with_error_handle(header + data)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    self._send_message(msg)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
    self._sendall_with_error_handle(header + data)
  File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe



Additional Files

No response

Environment

  • WandB version: 0.15.12
  • OS: Ubuntu 20.04
  • Python version: Python 3.8.10
  • Versions of relevant libraries: pytorch-lightning==2.0.9

Additional Context

No response

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 5
  • Comments: 51 (1 by maintainers)

Most upvoted comments

I just hit a variant of this. In my case, another process was deleting the wandb dir during training.

Stepping back a bit, I would strongly prefer that wandb issues never cause my training process to die. I would prefer wandb include a blanket try/except around its work, with verbose terminal logging when something has gone wrong.

Then in the morning I can decide how I want to handle it, instead of discovering that I’ve had expensive GPUs sitting idle for 10 hours.

because of this error my progress of nearly 3days stopped in between. Now i have to start again. Is there any alternative for this or handler for this or should i just store the progress locally

Any updates @Carlo-Argel? This issue is killing joy of wandb, and it is just bizarre it takes so long to fix it.

How is this issue closed? This is not solved so should not be closed. I have a fresh install of wandb, and its happening.

Basically what I imagine is happening is that the server loses connection for a short time, and then wandb, does not have a local cache, and so it crashes.

Its happening on all of my nodes intermittently. My model will train for 3-5 days without issue then wandb will kill the training. Very uncool.

Just cache the data locally, and then sync when the server gets connectivity back!

I also encountered the same BrokenPipeError: [Errno 32] Broken pipe with sent = self._sock.send(data) when using sweep to do hyperparamter tuning. In my case, after carefully checked my code, I found that I have call wandb.init() twice, one was outside the function the wandb.agent was required to run (in my case, it is the main function) , and one was in the main function with with wandb.init():. And I deleted the first wandb.init(), it worked!

Specifically, here is what my code is like:

import wandb

wandb.login()
sweep_config = {
    """ here I defined my sweep config"""
}

sweep_id = wandb.sweep(sweep_config, project="project_name")
wandb.init() # I delete this line, and the BrokenPipeError fixed

def main():
      with wandb.init():
              """
                here is my code for training
              """

if __name__ == "__main__":
    wandb.agent(sweep_id, function=main, count=50)

Hi @celsofranssa,

I’ll be happy to assist you with this inquiry. We received this and we will investigate it and get back to you for updates.

Regards, Carlo Argel

I think I might have found the reason. This error occurs when the script is modified while the program is running.

I randomly get this error every now and then during training too, I assume it is related to networking issues. It would be great if any internal W&B issues wouldn’t result in the run crashing.

{'loss': 55842.3375, 'learning_rate': 0.00019748020497041964, 'epoch': 1.32}
{'loss': 55757.2188, 'learning_rate': 0.0001974556426587668, 'epoch': 1.33}
  9%|██████████▋                                                                                                             | 2587/29100 [1:22:29<526:27:27, 71.48s/it]
Exception in thread NetStatThr:
Traceback (most recent call last):
  File "/home/user/mambaforge/envs/slt/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/home/user/mambaforge/envs/slt/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 268, in check_network_status
    self._loop_check_status(
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    local_handle = request()
                   ^^^^^^^^^
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/interface/interface.py", line 792, in deliver_network_status
    return self._deliver_network_status(status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/interface/interface_shared.py", line 500, in _deliver_network_status
    return self._deliver_record(record)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/interface/interface_shared.py", line 449, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    interface._publish(record)
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/home/user/mambaforge/envs/slt/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
           ^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Killed```

This is still happening and not closed as reported by myself and others. Good that you guys almost fixed it but it you should permit to override how long you wait (more than just 2 minutes in case there is an hour of downtime or so) before throwing [Errno 32] Broken pipe and crashing the train loop. Can you enable that?

goroutine 1005219 [sleep]: time.Sleep(0x6fc15b067) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78

goroutine 1022219 [sleep]: time.Sleep(0x6fc1b5f67) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78

goroutine 1016776 [sleep]: time.Sleep(0x6fc216062) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78

goroutine 995293 [sleep, 2 minutes]: time.Sleep(0x6fc1f3e15) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78 ERROR!!!

Traceback (most recent call last): File “/workspace/strange-loop-networks/train_sln_v3.py”, line 576, in <module> wandb.log(message) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 452, in wrapper return func(self, *args, **kwargs) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 403, in wrapper_fn return func(self, *args, **kwargs) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 393, in wrapper return func(self, *args, **kwargs) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 1928, in log self._log(data=data, step=step, commit=commit)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 1647, in _log self._partial_history_callback(data, step, commit) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 1519, in _partial_history_callback self._backend.interface.publish_partial_history( File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface.py”, line 647, in publish_partial_history self._publish_partial_history(partial_history)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_shared.py”, line 89, in _publish_partial_history self._publish(rec) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_sock.py”, line 51, in _publish self._sock_client.send_record_publish(record)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 221, in send_record_publish self.send_server_request(server_req) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 155, in send_server_request self._send_message(msg) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 152, in _send_message self._sendall_with_error_handle(header + data)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 130, in _sendall_with_error_handle sent = self._sock.send(data) BrokenPipeError: [Errno 32] Broken pipe

在使用 sweep 进行 hyperparamter 调优时,我也遇到了同样的情况。就我而言,在仔细检查了我的代码后,我发现我调用了两次,一次是在需要运行 wandb.agent 的函数之外(在我的情况下,它是函数),一次是在带有 .我删除了第一个,它奏效了!BrokenPipeError: [Errno 32] Broken pipe``sent = self._sock.send(data)``wandb.init()``main``main`` with wandb.init():``wandb.init()

具体来说,这是我的代码是这样的:

import wandb

wandb.login()
sweep_config = {
    """ here I defined my sweep config"""
}

sweep_id = wandb.sweep(sweep_config, project="project_name")
wandb.init() # I delete this line, and the BrokenPipeError fixed

def main():
      with wandb.init():
              """
                here is my code for training
              """

if __name__ == "__main__":
    wandb.agent(sweep_id, function=main, count=50)

I tried and it really helps!thanks!

i meet the same question, wait for fixing it.

I found that when I launch a Weights & Biases (wandb) service with simulated data alone, there are no issues with the service communication. However, when I simultaneously load a model on the GPU, the wandb service immediately stops (with the same error as mentioned above). If I restart the wandb service at this point, I notice that it will automatically stop after a fixed period (about 1 minute). Could this be related to the load balancer?

Training /chenhui/zhangwuhan/stage2/trained_model/qwen1.5_7b_5_5e-5_2_1k_plugin 0
  0%|                                                                                                 | 0/14272 [00:06<?, ?it/s, train_loss=4.85]2024-04-27 17:18:41,774 - DEBUG - Successfully logged to WandB
  0%|                                                                                       | 1/14272 [00:11<25:10:01,  6.35s/it, train_loss=2.9]2024-04-27 17:18:47,054 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 2/14272 [00:16<22:40:19,  5.72s/it, train_loss=2.25]2024-04-27 17:18:52,202 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 3/14272 [00:22<21:38:07,  5.46s/it, train_loss=2.09]2024-04-27 17:18:57,456 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 4/14272 [00:27<21:18:49,  5.38s/it, train_loss=2.02]2024-04-27 17:19:02,601 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 5/14272 [00:32<20:58:47,  5.29s/it, train_loss=1.88]2024-04-27 17:19:07,853 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 6/14272 [00:38<25:34:01,  6.45s/it, train_loss=1.87]
Traceback (most recent call last):
  File "/chenhui/zhangwuhan/stage2/FastChat/fastchat/train/qwen1.5_7b_5_5e-5_2_1k_plugin.py", line 290, in <module>
    main()
  File "/chenhui/zhangwuhan/stage2/FastChat/fastchat/train/qwen1.5_7b_5_5e-5_2_1k_plugin.py", line 283, in main
    accelerator.log({"train_loss":  loss.item()}, step=batch_idx)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 602, in _inner
    return PartialState().on_main_process(function)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 2267, in log
    tracker.log(values, step=step, **log_kwargs.get(tracker.name, {}))
  File "/usr/local/lib/python3.10/dist-packages/accelerate/tracking.py", line 86, in execute_on_main_process
    return function(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/tracking.py", line 333, in log
    self.run.log(values, step=step, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 420, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 371, in wrapper_fn
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 361, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 1838, in log
    self._log(data=data, step=step, commit=commit)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 1602, in _log
    self._partial_history_callback(data, step, commit)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 1474, in _partial_history_callback
    self._backend.interface.publish_partial_history(
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface.py", line 602, in publish_partial_history
    self._publish_partial_history(partial_history)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_shared.py", line 89, in _publish_partial_history
    self._publish(rec)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
wandb: While tearing down the service manager. The following error has occurred: [Errno 32] Broken pipe
2024-04-27 17:19:14,144 - DEBUG - Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
2024-04-27 17:19:16,144 - DEBUG - Attempting to acquire lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,144 - DEBUG - Lock 140175619989824 acquired on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Attempting to release lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Lock 140175619989824 released on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Attempting to acquire lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Lock 140175619989824 acquired on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-04-27 17:19:16,146 - DEBUG - Attempting to release lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-04-27 17:19:16,146 - DEBUG - Lock 140175619989824 released on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock

I hope the official can resolve this issue as soon as possible.

Hi guys,

I think I found a temporary solution.

For me it wasn’t space issues.

but indeed, I believe it was the fact that the processes did not stop after the script has finished running. There seems to have been a way where wandb was automatically finishing those processes, but now that doesnt work properly.

What helped me was kiling the processes both on CPU obtained from top (straight forward killing with the PID)

but also the not so obvious GPU processes (something I only learned because of this issue 😃 )

  1. list the processes on gpu lsof /dev/nvidia*

  2. make sure that all the processes are yours and not some other users.

  3. if it s all yours and you dont need them (you want to kill them all) lsof /dev/nvidia* | awk '{print $2}' | xargs -I {} kill -9 {}

now reruning the script doesnt produce the error for me. I hope that helps.

there is enough space. it only happens with sweeps.

similar issue at the end of the process but it does not affect other things