web3.py: Bug: OverflowError: Python int too large to convert to C ssize_t

Version: 5.8.0
Python: 3.7.6
OS: win
pip freeze output

attrs==19.3.0
base58==2.0.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Cython==0.29.14
cytoolz==0.10.1
eth-abi==2.1.0
eth-account==0.4.0
eth-hash==0.2.0
eth-keyfile==0.5.1
eth-keys==0.2.4
eth-rlp==0.1.2
eth-typing==2.2.1
eth-utils==1.8.4
gevent==1.4.0
greenlet==0.4.15
hexbytes==0.2.0
idna==2.8
importlib-metadata==1.5.0
ipfshttpclient==0.4.12
jsonschema==3.2.0
lru-dict==1.1.6
multiaddr==0.0.9
mypy-extensions==0.4.3
netaddr==0.7.19
oauthlib==3.1.0
parsimonious==0.8.1
protobuf==3.11.2
pycparser==2.19
pycryptodome==3.9.4
pypiwin32==223
pyrsistent==0.15.7
pywin32==227
requests==2.22.0
requests-oauthlib==1.3.0
rlp==1.2.0
six==1.13.0
toolz==0.10.0
typing-extensions==3.7.4.1
uniswap-python==0.3.4
urllib3==1.25.7
varint==1.0.2
web3==5.8.0
websocket==0.2.1
websocket-client==0.57.0
websockets==8.1
yappi==1.2.3
zipp==2.1.0

What was wrong?

I get a overflow error when calling a specific view function which returns a int256[]; When i call te same function online using a site like https://justsmartcontracts.dev the function works fine

  File "C:/Users/*/PycharmProjects/UltraDex/View.py", line 27, in <module>
    ab = az.getreturn()
  File "C:/Users/*/PycharmProjects/UltraDex/View.py", line 21, in getreturn
    return viewinstance.functions.buysellmultiple(self.addresa,self.addresb,self.am,self.parts,self.flags).call()
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\web3\contract.py", line 959, in call
    **self.kwargs
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\web3\contract.py", line 1498, in call_contract_function
    output_data = web3.codec.decode_abi(output_types, return_data)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\codec.py", line 181, in decode_abi
    return decoder(stream)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\decoding.py", line 127, in __call__
    return self.decode(stream)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_utils\functional.py", line 45, in inner
    return callback(fn(*args, **kwargs))
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\decoding.py", line 173, in decode
    yield decoder(stream)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\decoding.py", line 127, in __call__
    return self.decode(stream)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\decoding.py", line 144, in decode
    stream.push_frame(start_pos)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\decoding.py", line 95, in push_frame
    self.seek_in_frame(0)
  File "C:\Users\*\AppData\Local\Programs\Python\Python37\lib\site-packages\eth_abi\decoding.py", line 84, in seek_in_frame
    super().seek(self._total_offset + pos, *args, **kwargs)
OverflowError: Python int too large to convert to C ssize_t

I know this isn’t enough information. If you tell me what else you need i will try to provide is as quickly as posible

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 29 (19 by maintainers)

Commits related to this issue

tests: Added xfail to complex abi decoding test for struct NOTE: This works in reality (see https://github.com/ethereum/web3.py/issues/1634#issuecomment-650797252) however our test suite currently ca... — committed to fubuloubu/vyper by fubuloubu 4 years ago
tests: Added xfail to complex abi decoding test for struct NOTE: This works in reality (see https://github.com/ethereum/web3.py/issues/1634#issuecomment-650797252) however our test suite currently ca... — committed to fubuloubu/vyper by fubuloubu 4 years ago

Most upvoted comments

I’m having a similar sort of issue as well, it seems this comes from an issue with web3.codec.decode_abi (which web3.eth.contract uses) doing things differently from how it would if it were using eth_abi.decode_single directly. The former breaks with an error very similar to the above, the latter works perfectly:

>>> data = HexBytes('0x000000000000000000000000000000000000000000000000000000000000000b00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000280000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000005686f727365000000000000000000000000000000000000000000000000000000')
>>> decode_single('(uint256,int128[7],bytes,int128[3][3],uint256,uint256)', data)                                                    
(11, (0, 0, 13, 0, 0, 0, 0), b'horse', ((0, 0, 0), (0, 0, 0), (0, 0, 0)), 0, 0)

So, it seems part of the issue is that web3.eth.contract needs to be refactored to use the newer API from eth-abi: https://eth-abi.readthedocs.io/en/latest/decoding.html#decoding-abi-values

As a note, this is for decoding a function that returns a struct of the following format:

struct W:
    a: uint256
    b: int128[7]
    c: bytes[100]
    e: int128[3][3]
    f: uint256
    g: uint256

fubuloubu on Jun 28, 2020

@tshirtman The MKR contract uses a different (old?) version of the ERC20 spec where name() returns a bytes32 and not a string. You need to handle that case separately.

See this PR for how we handle it in uniswap-python: https://github.com/uniswap-python/uniswap-python/pull/191

ErikBjare on Jul 4, 2022

It appears this issue is not a problem with web3.py and I am now re-closing this. Happy to re-visit/re-open if someone feels differently or there is something that web3.py could be doing in this case to mitigate.

pipermerriam on Jul 25, 2020

As for what to do about methods that return singular tuple types with variable size elements, are we in agreement that the output encoding produced by vyper is missing the “wrapper” that other variable size return values exhibit when returned from the EVM?

Actually, I figured out that it is actually the opposite: if the value returned is a complex type (such as a struct or tuple), the convention is not to wrap it. I think we’re seeing that our ABI encoder did the right thing, but what we output for the ABI is wrong.

EDIT: Sorry I edited the previous comments one too many times lol

fubuloubu on Jul 25, 2020

Here is what I know so far…

In [35]: w3.codec.encode_single('uint256', 1234)
Out[35]: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\xd2'

In [36]: w3.codec.encode_abi(['uint256',], [1234,])
Out[36]: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\xd2'

Here we see the behavior for fixed size value encoding. The singular/plural encodings are identical.

Next lets look at variable length encodings.

In [41]: w3.codec.encode_single('uint256[]', [1234, 4321])
Out[41]: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\xd2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\xe1'

In [42]: w3.codec.encode_abi(['uint256[]',], [[1234, 4321]])
Out[42]: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\xd2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\xe1'

If we try to decode the output from the encode_single call with decode_abi things blow up with loud stacktraces.

In [45]: w3.codec.decode_abi(['uint256[]'], w3.codec.encode_single('uint256[]', [1234, 4321]))
---------------------------------------------------------------------------
InsufficientDataBytes                     Traceback (most recent call last)
... <STACKTRACE OMITTED HERE>

So now looking at the tuple example, what I believe we are seeing is an incorrectly encoded output from the smart contract.

If we take a look at a simple example, like the name() function found on most token contracts, I think we can extrapolate what the correct behavior here should be. The name() method on ERC contracts returns a string type. To decode this output we have to do w3.codec.decode_abi(['string'], data). That means our output is not just the string, but an ABI list with a single element of the encoded string.

Now looking at the provided tuple example: The contract ABI does not match the returned data.

What the ABI “claims” is being returned:
- ['(uint256,int128[7],bytes,int128[3][3],uint256,uint256)']
What is actually being returned:
- ['uint256', 'int128[7]', 'bytes', 'int128[3][3]', 'uint256', 'uint256']

So I think that what we are seeing is vyper not correctly encoding the return value for this function.

But maybe I’m incorrect

I did not look to see what solidity does for singular tuple type returns so I think that is the next step needed before a decision can be made on what the correct behavior here should be.

pipermerriam on Jul 24, 2020

If you can reproduce the above error in CI then dropping into a pdb might be the most informative way to try and figure out which value is exceeding the limit and then how eth-abi is encountering a that value.

pipermerriam on May 4, 2020