cosmos-sdk: Post upgrade with x/upgrade queries not working

Summary of Bug

Today we ran an automatic upgrade using the x/upgrade module. After a first error in the upgrade during the de-serialization of the data from the old state, we pushed a fix and every validator was able to upgrade correctly.

After the upgrade, the chain continued to create blocks correctly. You can see it running from here.

Unluckily, we later found out that all the queries are now returning the same error:

From the REST APIs:

{
"error": "invalid request: failed to load state at height 516938; version does not exist (latest height: 516938)"
}

From the CLI:

invalid request: failed to load state at height 516938; version does not exist (latest height: 516938)

This happened on all the nodes that underwent the upgrade procedure, no matter what their pruning strategies was. We tested it with ones having pruning = "nothing, pruning = "everything" and pruning = "default". All with the same result.

Currently we’re testing if restarting a node with pruning = "nothing", and making sure the update works at the first try, solves the problem.

Version

0.39.1

Steps to Reproduce

Create an upgrade proposal
Have the upgrade proposal run
Try query some data

For Admin Use

Not duplicate issue
Appropriate labels applied
Appropriate contributors tagged
Contributor assigned/self-assigned

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 32 (28 by maintainers)

Commits related to this issue

post-upgrade with x/upgrade queries not working (#7531) Origin: https://github.com/cosmos/cosmos-sdk/pull/7415 Closes: https://github.com/cosmos/cosmos-sdk/issues/7385 Thanks: @RiccardoM for the ba... — committed to cosmos/cosmos-sdk by RiccardoM 4 years ago

Most upvoted comments

Ok, I ran a script on the database which inspected the latest IAVL version of each substore in the database, it found this:

s/k:params 532167
s/k:upgrade 532167
s/k:evidence 532167
s/k:magpie 532167
s/k:reports 532167
s/k:posts 532167
s/k:relationships 18596
s/k:slashing 532167
s/k:staking 532167
s/k:gov 532167
s/k:profiles 532167
s/k:main 532167
s/k:supply 532167
s/k:acc 532167
s/k:distribution 532167

Notice the s/k:relationships store which is only at version 18596. When loading the stores, it will fail on this since it doesn’t have version 532167.

Notice that 18596 is almost exactly the number of heights since the upgrade (at 513573). This is not a coincidence. This is probably a new store that was registered during the upgrade, but it starts from version 1 (as all IAVL stores do).

Either the SDK needs to handle this case during upgrades by using the initial height API currently included in the IAVL 0.15.x prereleases, or it should just use a single IAVL store for all modules (which would also fix the partial commit problem in #6370) - I’m heavily in favor of the latter.

erikgrinaker on Sep 25, 2020

Could you please check if this is better?

You need to store iterator.Key() in the slice, not iterator.Value(). You’re also not checking for iteration errors, you’ll need to check iterator.Error() after the for-loop.

erikgrinaker on Sep 25, 2020