requests: "OverflowError: string longer than 2147483647 bytes" when trying requests.put

Hi,

I’m trying to upload a file that weight about 3GB and I’m getting the following error: “OverflowError: string longer than 2147483647 bytes”

If I understand correctly it seems like there’s a 2GB limit? didnt manage to find any reference to such limiation or how to bypass it (if possible).

The code i’m using is:


datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    args = myfile.read()
resp = requests.put(url, data=args, verify=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 995, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1029, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 991, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 844, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 820, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/ssl.py", line 234, in sendall
    v = self.send(data[count:])
  File "/usr/local/lib/python2.7/ssl.py", line 203, in send
    v = self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

For smaller files this code works fine for me.

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 33 (15 by maintainers)

Commits related to this issue

Fix string limit issue in _request_put_file Requests can't handle a put call for very large data objects. However, it can accept the data as a file-like object instead, and the size issue will not sh... — committed to lfit/releng-lftools by MightyNerdEric 4 years ago
workaround to upload files >2GB (see psf/requests#2717) — committed to cstenkamp/python-seafile by cstenkamp 4 years ago
workaround to upload files >2GB (see psf/requests#2717) — committed to cstenkamp/python-seafile by cstenkamp 4 years ago

Most upvoted comments

I’m having the same basic issue as @gjedeer and see the same behavior as @cmbasnett (that wrapping in BytesIO is not a solution). I’m trying to use a file object to upload something larger than 2GB using a TLS encrypted post. Specifically I’m trying to use a presigned url to upload a file to S3. It appears that the underlying ssl library in python doesn’t like files over 2GB. Is there an accepted workaround to this? Stack trace:

Basic code:

 with open(self.path_to_data, 'rb') as f:
    fields = 'defined elsewhere...'
    files = {'file': f}
    request('post', url, data=fields, files=files)

Traceback (most recent call last):
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/bin/mds", line 11, in <module>
    load_entry_point('mdscli', 'console_scripts', 'mds')()
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/coltonhicks/dev/mds_cli/mdscli/cli.py", line 133, in upload_member_data
    uploader.main()
  File "/Users/coltonhicks/dev/mds_cli/mdscli/upload_member_data.py", line 30, in main
    self.client.upload_member_data(self.mida, self.data_type, f)
  File "/Users/coltonhicks/dev/mds_cli/mdscli/requests_client.py", line 300, in upload_member_data
    logger.info(
  File "/Users/coltonhicks/dev/mds_cli/mdscli/requests_client.py", line 186, in _request
    res = requests.request(method, url, data=data, files=files)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/Users/coltonhicks/.virtualenvs/mds_cli-AtYG3_5U/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1055, in _send_output
    self.send(chunk)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 977, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1015, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 984, in send
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

coltonbh on Aug 23, 2019

@SpoonMeiser Wrapping the file contents in a BytesIO does not fix the problem on Python 3.6, still raises the exact same error.

cmbasnett on Jul 8, 2019

Have you tried using the MultipartEncoder from the requests-toolbelt? That allows you to stream multipart/form-data uploads like doing an upload of a single file. It was written specifically for this use case and was added to our docs.

sigmavirus24 on Oct 11, 2016

This limitation is in httplib. You can easily avoid it by slightly changing your code:

datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    resp = requests.put(url, data=myfile, verify=False)

Lukasa on Aug 12, 2015