cosmos-sdk: 504 Gateway Time-out error when querying tx for some addresses

@clevinson I couldn’t reproduce this on a local machine with huge dataset as well, might be something to do with timeout as @nddeluca suggested. We’ll revisit if that doesn’t solve the issue for anyone in the future and we are safe to close this issue for now imo

We ran into the same error on our Kava nodes – believe it’s the write timeout set in the http server at https://github.com/tendermint/tendermint/blob/bc572217c07b90ad9cee851f193aaa8e9557cbc7/rpc/lib/server/http_server.go#L41, which is pulled from 'timeout_broadcast_tx_commit` in the config.

Raising the limit above our nginx proxy timeout resolved the panics for us. The rest-server also has a default of 10 seconds and can be raised by restarting with the --write-timeout flag.

Also, in this case, OP stated the following request was being used: /txs?transfer.recipient=cosmos1y6yvdel7zys8x60gz9067fjpcpygsn62ae9x46&limit=20&page=1. This request is paginated and results in 20 txs at most. There is no way that results in 1MB.

However, it could very well be since this RPC service is public, that the given request is not what causes this panic(s), but something else.

@alexanderbez its a t3a.large machine (2 vCPUs, 8GB memory), and it is using around 45% of CPU and 21% of memory. There is also about 25% disk space available.

I have restarted both gaiad and gaiacli services, but same panic happens.