hermes: Creating a channel fails trying to simulate the tx and ultimately fails out

Summary of Bug

When I try to create a channel with Hermes, it fails with a lengthy output. I was attempting this between Osmosis testnet and my own Testnet for Bluzelle. Interestingly, an earlier version of Curium chain works fine with Hermes. This deployment of Curium chain has some parametric differences and is a fork of an existing LaunchPad network, but I don’t see how the fork itself is the difference/cause. Obviously, something different explains this new failure.

The key aspect of the failure appears to be the call to “CreateClient”, where an attempt to estimate the gas, fails.

Here is the output (with much of the golang output purged for practicality – please ask if the rest is needed):

Hint: Consider using the default invocation

hermes create channel --port-a <PORT-ID> --port-b <PORT-ID> <CHAIN-A-ID> <CONNECTION-A-ID>

to re-use a pre-existing connection. yes
2022-07-06T11:08:58.034961Z  INFO ThreadId(01) Creating new clients, new connection, and a new channel with order ORDER_UNORDERED
2022-07-06T11:08:58.047592Z  INFO ThreadId(11) wait_for_block_commits: waiting for commit of tx hashes(s) 3A557833D22CF1EBE1DA762A7D421F3C55E52E501BC388850B6BD31E31B508B0 id=osmo-test-4
2022-07-06T11:09:00.761983Z  INFO ThreadId(01) 🍭 [curium -> osmo-test-4:07-tendermint-2065]  => CreateClient(
    CreateClient(
        Attributes {
            height: Height {
                revision: 4,
                height: 5447855,
            },
            client_id: ClientId(
                "07-tendermint-2065",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 8961,
            },
        },
    ),
)

2022-07-06T11:09:00.774013Z ERROR ThreadId(21) send_tx_with_account_sequence_retry{id=curium}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "recovered: UnmarshalJSON cannot decode empty bytes
stack:
goroutine 3429905 [running]:
runtime/debug.Stack()
        /snap/go/9759/src/runtime/debug/stack.go:24 +0x65
github.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x2fc0220, 0xc0102410a0})
        /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/baseapp/recovery.go:71 +0x27
github.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x2fc0220, 0xc0102410a0})
        /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/baseapp/recovery.go:39 +0x30
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2fc0220, 0xc0102410a0}, 0xc013397440)
        /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/baseapp/recovery.go:28 +0x37
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2fc0220, 0xc0102410a0}, 0x3c5ac40)
        /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/baseapp/recovery.go:33 +0x5e

Version

Hermes version:

hermes 0.15.0+4e83aae

Steps to Reproduce

Here is the command that was run and can be run again to reproduce:

hermes create channel osmo-test-4 -c curium --port-a transfer --port-b transfer --new-client-connection

You will need an adequately funded curium account. Here is the json you can use to add the key to Hermes:

":\"A8D7EWAqMuKRc/xADg6XHGVb4OcGIXn/+jGqQnQbnxeO\"}","mnemonic":"give ahead beyond together relax ahead clip execute stick valid leaf hybrid staff noise keen month snake purity clump jump climb present shrimp cement"}

Note: You will likely need to add the following parameter when adding the key above to Hermes:

--hd-path "m/44'/483'/0'/0/0"

You will need an adequately funded osmosis account. For simplicity, I provide here the json you can use for adding a key for osmosis:

{"name":"osmosis-ibc","type":"local","address":"osmo1hhv8mqfn7kv9gte7gw7whsueh4566w93k2elu9","pubkey":"{\"@type\":\"/cosmos.crypto.secp256k1.PubKey\",\"key\":\"Akz3ggVQWyVm7UGctCPMJ9gx7yG2OxGAGpn6sEev1G+9\"}","mnemonic":"shy scorpion portion believe outer frost pink crunch tennis satisfy dash oval trust duty wash mad result region area buzz safe frequent trend kitchen"}

Here is a relevant snippet of Hermes’ config.toml I use (modifed with the rpc and grpc endpoints working for the Internet):

[[chains]]
id = 'osmo-test-4'
rpc_addr = 'http://preview.bluzelle.com:26677'
grpc_addr = 'http://preview.bluzelle.com:9110'
websocket_addr = 'ws://preview.bluzelle.com:26677/websocket'
rpc_timeout = '10s'
account_prefix = 'osmo'
key_name = 'relayer'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 1402071
gas_price = { price = 0.0001, denom = 'uosmo' }
gas_adjustment = 0.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '10s'
trusting_period = '12days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }
memo_prefix = 'Interbloc for Osmosis'
[chains.packet_filter]
policy = 'allow'
list = []

[[chains]]
id = 'curium'
rpc_addr = 'http://preview.bluzelle.com:26657'
grpc_addr = 'http://preview.bluzelle.com:9090'
websocket_addr = 'ws://preview.bluzelle.com:26657/websocket'
rpc_timeout = '10s'
account_prefix = 'bluzelle'
key_name = 'relayer'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 800000
gas_price = { price = 0.0007, denom = 'ubnt' }
gas_adjustment = 0.2
max_msg_num = 30
max_tx_size = 200000
clock_drift = '5s'
max_block_time = '10s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }
memo_prefix = 'Interbloc for Bluzelle'
[chains.packet_filter]
policy = 'allow'
list = [['transfer', 'channel-0']]

The other sections of config.toml is the default if I recall.

Acceptance Criteria

I can run the command to create a channel, successfully.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned