py-ipfs-http-client: Broken Connection on successive calls using add_bytes
I am attempting to debug an issue wherein I am pushing the binary image data represented in Python byte string into IPFS using ipfshttpclient.
I have been able to distill this down to an example with two successive calls of add_bytes
with large byte strings:
import ipfshttpclient
import os
bytes = bytearray(os.urandom(100000))
IPFS_API = ipfshttpclient.Client(f"/dns/localhost/tcp/5001/http")
res1 = IPFS_API.add_bytes(bytes)
res2 = IPFS_API.add_bytes(bytes)
…produces:
Traceback (most recent call last):
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/adapters.py", line 472, in send
low_conn.send(i)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 251, in _do_request
return requests.request(*args, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/requests_wrapper.py", line 244, in request
return session.request(method=method, url=url, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/requests_wrapper.py", line 223, in request
return super(Session, self).request(method, url, *args, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: [Errno 32] Broken pipe
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "pngpushtest.py", line 8, in <module>
res2 = IPFS_API.add_bytes(bytes)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/utils.py", line 142, in wrapper
res = cmd(*args, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/client/base.py", line 15, in wrapper
result = func(*args, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/client/__init__.py", line 223, in add_bytes
data=body, headers=headers, **kwargs)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 50, in wrapper
return func(self, *args, **merged)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 370, in request
files, headers, data, timeout=timeout)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 288, in _request
timeout=timeout)
File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 255, in _do_request
six.raise_from(exceptions.ConnectionError(error), error)
File "<string>", line 3, in raise_from
ipfshttpclient.exceptions.ConnectionError: ConnectionError: [Errno 32] Broken pipe
Changing the above line to bytes = bytearray(os.urandom(10))
, using 10
instead of 100000
does not produce this exception.
- Python 3.7.2
- ipfshttpclient 0.4.12
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (14 by maintainers)
Commits related to this issue
- Only add `Connection: close` header for go-IPFS v0.4.18- (cf #187) — committed to ntninja/py-ipfs-api-client by ntninja 5 years ago
- Only add `Connection: close` header for go-IPFS v0.4.18- (cf #187) — committed to ntninja/py-ipfs-api-client by ntninja 5 years ago
- Only add `Connection: close` header for go-IPFS v0.4.18- (cf #187) — committed to ntninja/py-ipfs-api-client by ntninja 5 years ago
- Only add `Connection: close` header for go-IPFS v0.4.18- (cf #187) — committed to ipfs-shipyard/py-ipfs-http-client by ntninja 5 years ago
- Only add `Connection: close` header for go-IPFS v0.4.18- (cf #187) — committed to ocelot/py-ipfs-api by ntninja 5 years ago
Thanks @machawk1, if the new version of this library works well with the new version of the go-ipfs then we can move on.
@machawk1: Thanks for your tremendously valuable research here! While, as @ibnesayeed found out, adding the header should not be a problem anymore, it’s still an unnecessary performance penalty and I’m glad to here that this can be removed with go-IPFS v0.4.19+.
I will create a patch that removes the
Connection: close
flag for v0.4.19+ and would welcome your feedback on this.The problem with 0.4.20 was that several API methods had returned broken or incomplete responses making it basically useless if you relied on any of these. Hence I decided to blacklist it altogether, rather than dealing with support queries that I can do nothing about anyways. (As such I don’t think it was related to this, but maybe played a part as well.)