cosmos-sdk: Cannot make block after skipped x/upgrade

Summary of Bug

We tried doing x/upgrade on musselnet-2 with --upgrade-time (previously only tested with --upgrade-height) and it failed. (@orkunkl will add another issue documenting this)

We then decided to just skip it, using --unsafe-skip-upgrade on our validators (to whom we had staked almost 75% of the voting power). The nodes then came up and starting participating in consensus, but never came to make a new block, even though we have > 67% voting power in pre-commits.

Version

wasmd 0.14.0 based on cosmos-sdk v0.40.1 (we were attempted to upgrade to wasmd 0.15.0 / cosmos-sdk v0.41.0)

Steps to Reproduce

Not sure. I am dumping some output of the rpc nodes in hope they clarify what is going on. Rounds 2 to 31 should have produced a block from my understanding. (Please rename from .json.txt to .json, github did not allow json uploads)

Consensus state: consensus_state.json.txt

Validators: validators.json.txt

Validators (previous block): validators-previous.json.txt

Key example from consensus state (height_vote_set):

        {
          "round": 2,
          "prevotes": [
            "Vote{0:6AF22951B169 445131/02/SIGNED_MSG_TYPE_PREVOTE(Prevote) C7152A70B514 7AEC1669862B @ 2021-02-08T09:56:17.004097353Z}",
            "Vote{1:07D76D2F9D17 445131/02/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000 E0700879FC9E @ 2021-02-08T09:56:17.096206306Z}",
            "Vote{2:AEF865B54EAC 445131/02/SIGNED_MSG_TYPE_PREVOTE(Prevote) C7152A70B514 AE743F755C5B @ 2021-02-08T09:56:17.452091526Z}",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote"
          ],
          "prevotes_bit_array": "BA{11:xxx________} 4034984861364/5380640551050 = 0.75",
          "precommits": [
            "Vote{0:6AF22951B169 445131/02/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000 176D5326E96C @ 2021-02-08T09:56:19.206038981Z}",
            "Vote{1:07D76D2F9D17 445131/02/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000 EB4DD3D144F8 @ 2021-02-08T09:56:19.10513201Z}",
            "Vote{2:AEF865B54EAC 445131/02/SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000 C6DBF37D6384 @ 2021-02-08T09:56:19.349238122Z}",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote",
            "nil-Vote"
          ],
          "precommits_bit_array": "BA{11:xxx________} 4034984861364/5380640551050 = 0.75"
        }

75% precommits, but it does not advance?

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned