Gravity-Bridge: panic when withdrawing rewards or commission from validator

On the current testnet (gravity-bridge-test{1..4}), my validator is not able to withdraw rewards or commission, but is able to successfully send other transactions like transferring tokens.

withdrawing:

./gravity tx distribution withdraw-rewards gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9 --commission --from validator --keyring-backend pass
{"body":{"messages":[{"@type":"/cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward","delegator_address":"gravity156n6x0hvfd2zxq5vhehp2e20mz6tdskhxtlf33","validator_address":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9"},{"@type":"/cosmos.distribution.v1beta1.MsgWithdrawValidatorCommission","validator_address":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9"}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"29485","txhash":"A612BAF0FB4B6A46D4DF51ADE93C746BF35366408A75A5CA2E0AEB2653F75180","codespace":"undefined","code":111222,"data":"","raw_log":"panic message redacted to hide potentially sensitive system info: panic","logs":[],"info":"","gas_wanted":"200000","gas_used":"97657","tx":null,"timestamp":""}

transferring:

./gravity tx bank send validator gravity15lpnf70ycjxyhhjysmlkm6k4uf0sm220ftqy79  1000000ugraviton --keyring-backend pass
{"body":{"messages":[{"@type":"/cosmos.bank.v1beta1.MsgSend","from_address":"gravity156n6x0hvfd2zxq5vhehp2e20mz6tdskhxtlf33","to_address":"gravity15lpnf70ycjxyhhjysmlkm6k4uf0sm220ftqy79","amount":[{"denom":"ugraviton","amount":"1000000"}]}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"14131","txhash":"64FD8BDD953639CA0CB15CFBCF6A61B3028279F4A552ABBC075221F0E0D68563","codespace":"","code":0,"data":"0A1E0A1C2F636F736D6F732E62616E6B2E763162657461312E4D736753656E64"...

I was able to change my commission and withdraw once (maybe during gravity-bridge-test1 or gravity-bridge-test2), but something happened and I’m not able to anymore.

I am able to withdraw as a different delegator to my validator:

./gravity tx distribution withdraw-rewards gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9 --from mine --keyring-backend pass
Default sign-mode 'direct' not supported by Ledger, using sign-mode 'amino-json'.
{"body":{"messages":[{"@type":"/cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward","delegator_address":"gravity1pqk05hdkyhlvj07tept96cwa3fj7axasq8cg0h","validator_address":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9"}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"35806","txhash":"65A5654C7D0F79AE6FB67DBE4D8F0B5226E71A81BFBB3C7A972AA50EE65E6381","codespace":"","code":0,"data":"0A390A372F636F736D6F732E646973747269627574696F6E2E763162657461312E4D7367576974686472617744656C656761746F72526577617264",

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 19 (7 by maintainers)

Most upvoted comments

Ok the results of today’s debugging is that this is a bug in the Gravity module.

Namely that ‘jailed’ validators must not be slashed again while jailed. This is implied by the slashing function contract which states that unbonding validators can not be slashed. The problem is that jailing a validator does put them into an unbonding state in some segments of the code.

As you can see here the Somm fork wraps the slashing code in a check if the validator is already jailed. Whereas the Gravity Bridge code does not.

This means solving this problem should be rather simple, but fixing the withdraw rewards so that those affected can withdraw is a major part of any fix.

It’s my intuition that when we perform gravity export to perform the upgrade the amount of unclaimed rewards will be exported. When re-imported after the computation in the cosmos-sdk distribution keeper should succeed as no slashing of jailed validators will ever occur on the upgraded chain. Allowing affected users to withdraw their full reward amount.

What remains for me to research tomorrow is to confirm that this is the case then we can simply plan the upgrade ASAP.

jkilpatr on Dec 15, 2021

This issue was also noticed in one of the early beta testnets of Umee: It’s a cosmos sdk issue afaik cosmos/cosmos-sdk#10091

It looks like Umee uses x/peggy, which x/gravity is derived from. Since it is only seen on these chains, the culprit is likely x/peggy and x/gravity, not cosmos-sdk.

I would not be terribly surprised by this. Cosmos-sdk has a bad habit of creating secret requirements for modules to interact with core functions. It’s really arguable where ‘fault’ lies, but discussing that isn’t productive.

I’m digging into this today. Will report back with a status update.

jkilpatr on Dec 15, 2021