tendermint: Latest versions fill up temporary storage rapidly to the point of crashing

BUG REPORT

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source): 0.13.0 and 0.14.0

Environment: Windows

What happened: I recently updated from 0.10.3 to 0.13.0 and later to 0.14.0 when that was available, but both of these experience the same behavior. After I set up a single node and enter a bunch of data, my temporary storage drive starts filling up rapidly (3GB+ for around 20k tx’s, which have never taken up more than 130MB of data when looking at the .tendermint folder). Since I’m running this on a basic VM, I only have a bit of space for my temporary storage drive and after it fills up entirely Tendermint crashes with the following output.

What you expected to happen: When working with 0.10.3, I never experienced this. I even went back to check and it doesn’t even touch my temporary storage drive when inputting the exact same data. So I expected it to be the same for this version.

How to reproduce it (as minimally and precisely as possible): A single node, running on a basic VM receiving about 20k tx’s.

Anything else do we need to know: This VM has a C: drive of 126GB which doesn’t get used apart from the normal storage things you’d expect. However the (already half filled) D: drive “Temporary Storage” only contains a total of 7GB, which causes Tendermint to crash after it fills up all the way. As soon as I close the Tendermint application, the D: drive instantly clears up.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 23 (13 by maintainers)

Commits related to this issue

Most upvoted comments

^^ You’ll need to use the latest abci version to be compatible, sorry for not being clear

@ebuchman help me please… << Ethermint >>> ethermint --datadir ~/.ethermint --rpc --rpcaddr=0.0.0.0 --ws --wsaddr=0.0.0.0 --rpcapi eth,net,web3,personal,admin INFO [03-15|18:00:10] Starting peer-to-peer node instance=ethermint/v1.6.7-stable/linux-amd64/go1.8.3 INFO [03-15|18:00:10] Allocated cache and file handles database=/root/.ethermint/ethermint/chaindata cache=128 handles=1024 INFO [03-15|18:00:10] Initialised chain configuration config=“{ChainID: 15 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: <nil> EIP155: 0 EIP158: 0 Metropolis: <nil> Engine: unknown}” WARN [03-15|18:00:10] Ethash used in fake mode INFO [03-15|18:00:10] Initialising Ethereum protocol versions=“[63 62]” network=1 INFO [03-15|18:00:10] Loaded most recent local header number=0 hash=2b72f9…d468ac td=64 INFO [03-15|18:00:10] Loaded most recent local full block number=0 hash=2b72f9…d468ac td=64 INFO [03-15|18:00:10] Loaded most recent local fast block number=0 hash=2b72f9…d468ac td=64 INFO [03-15|18:00:10] Starting P2P networking INFO [03-15|18:00:10] RLPx listener up self=“enode://1c55fa69494b0fe1a2356195da71f925ff3d8ae1d94327ab72a8d32fc28de0031811fa1ae3d28f0a44e02fca616e17a35fc2aaf1fc231abdba98c91adb4127d6@[::]:30303?discport=0” INFO [03-15|18:00:10] Waiting for tendermint endpoint to start err=“Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused” INFO [03-15|18:00:10] HTTP endpoint opened: http://0.0.0.0:8545 INFO [03-15|18:00:10] IPC endpoint opened: /root/.ethermint/geth.ipc INFO [03-15|18:00:10] WebSocket endpoint opened: ws://0.0.0.0:8546 INFO [03-15|18:00:12] Starting ABCIServer module=abci-server impl=ABCIServer INFO [03-15|18:00:12] Waiting for new connection… module=abci-server INFO [03-15|18:00:13] Waiting for tendermint endpoint to start err=“Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused” INFO [03-15|18:00:13] Accepted a new connection module=abci-server INFO [03-15|18:00:13] Waiting for new connection… module=abci-server INFO [03-15|18:00:13] Accepted a new connection module=abci-server INFO [03-15|18:00:13] Waiting for new connection… module=abci-server INFO [03-15|18:00:13] Accepted a new connection module=abci-server INFO [03-15|18:00:13] Waiting for new connection… module=abci-server

ERROR[03-15|18:00:13] Connection error module=abci-server error=“Error reading message: Varint overflow”

INFO [03-15|18:00:16] Waiting for tendermint endpoint to start err=“Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused” INFO [03-15|18:00:19] Waiting for tendermint endpoint to start err=“Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused” INFO [03-15|18:00:22] Waiting for tendermint endpoint to start err=“Post http://localhost:46657/status: dial tcp [::1]:46657: getsockopt: connection refused” ^Ccaptured interrupt, exiting… INFO [03-15|18:00:24] Stopping ABCIServer module=abci-server impl=ABCIServer


<tendermint> tendermint --home ~/.ethermint/tendermint node I[03-15|09:00:13.963] Starting multiAppConn module=proxy impl=multiAppConn I[03-15|09:00:13.963] Starting socketClient module=abci-client connection=query impl=socketClient I[03-15|09:00:13.963] Starting socketClient module=abci-client connection=mempool impl=socketClient I[03-15|09:00:13.963] Starting socketClient module=abci-client connection=consensus impl=socketClient E[03-15|09:00:13.964] Stopping abci.socketClient for error: EOF module=abci-client connection=query I[03-15|09:00:13.964] Stopping socketClient module=abci-client connection=query impl=socketClient E[03-15|09:00:24.539] Stopping abci.socketClient for error: EOF module=abci-client connection=consensus I[03-15|09:00:24.539] Stopping socketClient module=abci-client connection=consensus impl=socketClient E[03-15|09:00:24.539] Stopping abci.socketClient for error: EOF module=abci-client connection=mempool I[03-15|09:00:24.539] Stopping socketClient module=abci-client connection=mempool impl=socketClient

You should be fine to use master across the board for tendermint repos. v0.16 should compile completely with master (ie. you should be able to just go get it 😃 )

@melekes @ebuchman Sorry for the delay, I’ve been trying to get the ABCI to work but I either keep running into the error I provided above or a ton of different other errors probably related to some library that has probably not been updated properly. I’ll try to do a wipe of my existing libraries at some point and update them all to the latest version to see if that fixes it. To be clear; do I still need to use any versions from the develop branch or is it alright to use the master branch now that version 0.16 has been released?

@bijlar thank you for trying. It looks very much like compatibility issue. What ABCI are you using? Tendermint 0.16-dev might not work with older versions (latest: https://github.com/tendermint/abci/tree/develop)

@ebuchman I tried to checkout and install the develop version (0.16.0-dev), but when I start the ABCI and run tendermint node it results in an error in the ABCI:

E[02-20|09:26:05.990] Connection error
module=abci-server error="Error reading message: Varint overflow"

And a corresponding error in the Tendermint node:

E[02-20|09:26:05.994] Stopping abci.socketClient for error: EOF
module=abci-client connection=query

After this both applications still seem to be doing something, but when I try to reach the blockchain through port 46657 like usual, it doesn’t work. Checking my ports also reveals this port is not in use by any application. Maybe I made some errors while installing the develop version, but I can’t seem to get it to work as it stands.

@melekes do you think this is fixed by the recent memleak fixes? Weird that they didn’t notice anything in 0.10.3 - I don’t think the memleak stuff was introduced after that ?

Anyways, this is hopefully fixed on develop. @bijlar can you try it out and let us know? Thanks!

That’s the odd thing, when I look at the D: drive, the only file there is one called DATALOSS_WARNING_README.txt which just contains some generic Microsoft warning about how the data stored on this drive is subject to loss. Even though that file is only 650 bytes and the drive is clearly a lot more full than that, I can’t find any other files or folders, no matter what tool I use to look at the drive.

Can you do the same thing (I mean du -h) for the whole D: drive?

Thank you!

Now we know that at least data inside .tendermint folder is not growing rapidly. Can it be /tmp analog for Windows?

restart Tendermint, it also cleans out the temporary storage drive.

looks very much like so.