wandb: [CLI]: BrokenPipeError: [Errno 32] Broken pipe
Bub description
Training and logging run fine; however, at the end of the process, the wandb outputs the error message below.
wandb: Waiting for W&B process to finish... (success).
wandb: \ 0.014 MB of 0.014 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: epoch ▁▁▁▁▁▁▁▁▃▃▃▃▃▃▃▃▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆████████
wandb: train_loss █▄▃▂▂▃▄▃▁▂▂▄▄▂▃▂▁▂▁▁▁▁▁▁▃▁▁▁▁▂▁▁▁▁▁▅▁▁▁▃
wandb: trainer/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb: val_Mac-F1 ▁▄▆▅█
wandb: val_Mic-F1 ▁▄▇▇█
wandb: val_Wei-F1 ▁▅▇▇█
wandb: val_loss █▂▁▄▆
wandb:
wandb: Run summary:
wandb: epoch 4
wandb: train_loss 0.47728
wandb: trainer/global_step 5534
wandb: val_Mac-F1 0.70413
wandb: val_Mic-F1 0.88889
wandb: val_Wei-F1 0.93459
wandb: val_loss 0.46428
wandb:
wandb: 🚀 View run BERT_WEBKB_0_exp at: https://wandb.ai/celsofranca/lightning_logs/runs/1qq5guxx
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: /tmp/wandb/run-20231012_134638-1qq5guxx/logs
Exception in thread Exception in thread IntMsgThrNetStatThr:
:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 299, in check_internal_messages
self._target(*self._args, **self._kwargs)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 267, in check_network_status
self._loop_check_status(
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 223, in _loop_check_status
self._loop_check_status(
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 223, in _loop_check_status
local_handle = request()
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 743, in deliver_internal_messages
return self._deliver_internal_messages(internal_message)
local_handle = request()
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 481, in _deliver_internal_messages
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 735, in deliver_network_status
return self._deliver_record(record)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 428, in _deliver_record
return self._deliver_network_status(status)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 475, in _deliver_network_status
handle = mailbox._deliver_record(record, interface=self)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
return self._deliver_record(record)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 428, in _deliver_record
interface._publish(record)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
handle = mailbox._deliver_record(record, interface=self)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
self._sock_client.send_record_publish(record)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
interface._publish(record)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
self.send_server_request(server_req)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
self._sock_client.send_record_publish(record)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
self._send_message(msg)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
self.send_server_request(server_req)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
self._sendall_with_error_handle(header + data)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
self._send_message(msg)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
self._sendall_with_error_handle(header + data)
File "/home/celso/projects/venvs/LightningPrototype/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
Additional Files
No response
Environment
- WandB version: 0.15.12
- OS: Ubuntu 20.04
- Python version: Python 3.8.10
- Versions of relevant libraries: pytorch-lightning==2.0.9
Additional Context
No response
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 5
- Comments: 51 (1 by maintainers)
I just hit a variant of this. In my case, another process was deleting the wandb dir during training.
Stepping back a bit, I would strongly prefer that wandb issues never cause my training process to die. I would prefer wandb include a blanket
try/exceptaround its work, with verbose terminal logging when something has gone wrong.Then in the morning I can decide how I want to handle it, instead of discovering that I’ve had expensive GPUs sitting idle for 10 hours.
because of this error my progress of nearly 3days stopped in between. Now i have to start again. Is there any alternative for this or handler for this or should i just store the progress locally
Any updates @Carlo-Argel? This issue is killing joy of wandb, and it is just bizarre it takes so long to fix it.
How is this issue closed? This is not solved so should not be closed. I have a fresh install of wandb, and its happening.
Basically what I imagine is happening is that the server loses connection for a short time, and then wandb, does not have a local cache, and so it crashes.
Its happening on all of my nodes intermittently. My model will train for 3-5 days without issue then wandb will kill the training. Very uncool.
Just cache the data locally, and then sync when the server gets connectivity back!
I also encountered the same
BrokenPipeError: [Errno 32] Broken pipewithsent = self._sock.send(data)when using sweep to do hyperparamter tuning. In my case, after carefully checked my code, I found that I have callwandb.init()twice, one was outside the function the wandb.agent was required to run (in my case, it is themainfunction) , and one was in themainfunction withwith wandb.init():. And I deleted the firstwandb.init(), it worked!Specifically, here is what my code is like:
Hi @celsofranssa,
I’ll be happy to assist you with this inquiry. We received this and we will investigate it and get back to you for updates.
Regards, Carlo Argel
I think I might have found the reason. This error occurs when the script is modified while the program is running.
I randomly get this error every now and then during training too, I assume it is related to networking issues. It would be great if any internal W&B issues wouldn’t result in the run crashing.
This is still happening and not closed as reported by myself and others. Good that you guys almost fixed it but it you should permit to override how long you wait (more than just 2 minutes in case there is an hour of downtime or so) before throwing [Errno 32] Broken pipe and crashing the train loop. Can you enable that?
goroutine 1005219 [sleep]: time.Sleep(0x6fc15b067) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78
goroutine 1022219 [sleep]: time.Sleep(0x6fc1b5f67) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78
goroutine 1016776 [sleep]: time.Sleep(0x6fc216062) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78
goroutine 995293 [sleep, 2 minutes]: time.Sleep(0x6fc1f3e15) /usr/local/go/src/runtime/time.go:285 +0xf2 github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait.func1() /project/core/internal/waiting/waiting.go:97 +0x109 created by github.com/wandb/wandb/core/internal/waiting.(*realStopwatch).Wait in goroutine 133 /project/core/internal/waiting/waiting.go:91 +0x78 ERROR!!!
Traceback (most recent call last): File “/workspace/strange-loop-networks/train_sln_v3.py”, line 576, in <module> wandb.log(message) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 452, in wrapper return func(self, *args, **kwargs) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 403, in wrapper_fn return func(self, *args, **kwargs) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 393, in wrapper return func(self, *args, **kwargs) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 1928, in log self._log(data=data, step=step, commit=commit)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 1647, in _log self._partial_history_callback(data, step, commit) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py”, line 1519, in _partial_history_callback self._backend.interface.publish_partial_history( File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface.py”, line 647, in publish_partial_history self._publish_partial_history(partial_history)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_shared.py”, line 89, in _publish_partial_history self._publish(rec) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_sock.py”, line 51, in _publish self._sock_client.send_record_publish(record)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 221, in send_record_publish self.send_server_request(server_req) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 155, in send_server_request self._send_message(msg) File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 152, in _send_message self._sendall_with_error_handle(header + data)
File “/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py”, line 130, in _sendall_with_error_handle sent = self._sock.send(data) BrokenPipeError: [Errno 32] Broken pipe
I tried and it really helps!thanks!
i meet the same question, wait for fixing it.
I found that when I launch a Weights & Biases (wandb) service with simulated data alone, there are no issues with the service communication. However, when I simultaneously load a model on the GPU, the wandb service immediately stops (with the same error as mentioned above). If I restart the wandb service at this point, I notice that it will automatically stop after a fixed period (about 1 minute). Could this be related to the load balancer?
I hope the official can resolve this issue as soon as possible.
Hi guys,
I think I found a temporary solution.
For me it wasn’t space issues.
but indeed, I believe it was the fact that the processes did not stop after the script has finished running. There seems to have been a way where wandb was automatically finishing those processes, but now that doesnt work properly.
What helped me was kiling the processes both on CPU obtained from
top(straight forward killing with the PID)but also the not so obvious GPU processes (something I only learned because of this issue 😃 )
list the processes on gpu
lsof /dev/nvidia*make sure that all the processes are yours and not some other users.
if it s all yours and you dont need them (you want to kill them all)
lsof /dev/nvidia* | awk '{print $2}' | xargs -I {} kill -9 {}now reruning the script doesnt produce the error for me. I hope that helps.
there is enough space. it only happens with sweeps.
similar issue at the end of the process but it does not affect other things