tendermint: Node crashes with "Failed to write response"

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):

tendermint: ""
abci: 0.17.0
blockprotocol: 11
p2pprotocol: 8

ABCI app (name for built-in, URL for self-written if it’s publicly available): Desmos v0.15.1

Environment:

  • OS (e.g. from /etc/os-release):
    NAME="Ubuntu"
    VERSION="20.04.1 LTS (Focal Fossa)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 20.04.1 LTS"
    VERSION_ID="20.04"
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    VERSION_CODENAME=focal
    UBUNTU_CODENAME=focal
    
  • Install tools:
  • Others:

What happened: Yesterday, one of our chain nodes has stopped with error Failed to write response for no apparent reason. This is not the first time this happens, and we still have to identify why.

What you expected to happen:
The node should not crash

Have you tried the latest version:
No

How to reproduce it (as minimally and precisely as possible):
I have yet to know this

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):
https://pastebin.com/rbaw6FVH

Config (you can paste only the changes you’ve made):

pruning = "nothing"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (18 by maintainers)

Most upvoted comments

@RiccardoM Hey,

if you’d like to get the Tendermint version in here:

tendermint: ""
abci: 0.17.0
blockprotocol: 11
p2pprotocol: 8

add:

VERSION := $(shell go list -m github.com/tendermint/tendermint | sed 's:.* ::')
LD_FLAGS = -X github.com/tendermint/tendermint/version.TMCoreSemVer=$(VERSION)

to your makefile.

cool. so what was the stacktrace?

I could not get any stacktrace. I just used that command to kill the service as the tendermint debug kill was not working. But I was not able to get any stracktrace from it. The node just stopped without any error and the only way to solve was to kill it or restart it.

sorry, maybe I should’ve been more specific. I meant Linux kill command https://linux.die.net/man//kill

Yeah, I’ve run the kill -6 command. Then I restarted the node and tried with tendermint debug kill but I got that error

@melekes You right, it hangs. I’m now running tendermint debug kill but it also seems to be hanging 😅

do we have that documented anywhere?

added here: https://github.com/tendermint/tendermint/pull/6151