gaia: Unable to query archival data with recent upgrade 14470501

Summary of Bug

Since the Gaia migration at block 14470501, historical data is screwed up for blocks below.

Version

gaiad-v9.0.1-linux-amd64

Steps to Reproduce

Blocks >= 14470501: OK

$ grpcurl --max-msg-sz 400194304 -H "x-cosmos-block-height: 14470501" -plaintext NODE:PORT  cosmos.gov.v1beta1.Query/Proposals

[...]

ok

Blocks < 14470501: KO

$ grpcurl --max-msg-sz 400194304 -H "x-cosmos-block-height: 14470500" -plaintext NODE:PORT cosmos.gov.v1beta1.Query/Proposals

ERROR:
  Code: Unknown
  Message: failed to load state at height 14470500; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 14504397): invalid request

This happens for all gRPC calls, no matter which module is used, using a fresh Quicksync full archival dump using gaiad-v9.0.1-linux-amd64 version.

Is this a regression?


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
  • Is a spike necessary to map out how the issue should be approached?

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 32 (15 by maintainers)

Commits related to this issue

Most upvoted comments

@Jacoblava I have updated my unofficial patch

https://github.com/Reecepbcups/cosmos-sdk/releases/tag/v0.45.16-ics-lsm-iavlpatch.1

replace github.com/cosmos/cosmos-sdk => github.com/reecepbcups/cosmos-sdk v0.45.16-ics-lsm-iavlpatch.1

also cc @mmulji-ic

This issue has been addressed in the SDK and backported for v0.46 and v0.47. There are a few temporary workarounds until the Hub upgrades to SDK v0.47 in the meantime which we’ll provide documentation for shortly.

@mmulji-ic Thank you for reply. As a user of cosmos node, I think this issue is critical. Once HF done, And Can’t get block data before HF. How could be verified data in the cosmos network?

Hope that this issue will raise in serious manner, fix in ASAP personally.

I’ve found a hacky fix by patching the cosmos-sdk as follows:

diff --git a/store/iavl/store.go b/store/iavl/store.go
index 50324c89e..619b722b7 100644
--- a/store/iavl/store.go
+++ b/store/iavl/store.go
@@ -108,7 +108,7 @@ func UnsafeNewStore(tree *iavl.MutableTree) *Store {
 // Any mutable operations executed will result in a panic.
 func (st *Store) GetImmutable(version int64) (*Store, error) {
        if !st.VersionExists(version) {
-               return &Store{tree: &immutableTree{&iavl.ImmutableTree{}}}, fmt.Errorf("version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height")
+               return &Store{tree: &immutableTree{&iavl.ImmutableTree{}}}, nil
        }
 
        iTree, err := st.tree.GetImmutable(version)

I’m not sure of all implications but I guess this is a regression that was introduced in the SDK bump during the Gaia v8 upgrade.

Thanks a lot, Will look at it now.

On Mon, Oct 2, 2023, 21:52 Reece Williams @.***> wrote:

oh and also @gilxgil https://github.com/gilxgil ^

— Reply to this email directly, view it on GitHub https://github.com/cosmos/gaia/issues/2313#issuecomment-1743582094, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4PBZQFH2YEEYNBUPEVCF73X5MEPFAVCNFSM6AAAAAAV65W7G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTGU4DEMBZGQ . You are receiving this because you were mentioned.Message ID: @.***>

Hi @Jacoblava @gilxgil we’ll take a look to see if we can adapt that fix or suggestion another mitigation.

I can confirm this resolved the issue temporarily for me.

Modify go.mod with

replace github.com/cosmos/cosmos-sdk => github.com/reecepbcups/cosmos-sdk v0.45.13-ics.archive.1

then

go mod tidy
make install

@Jacoblava @ScepticMatt please repoen if your forked forks do not work. Otherwise, closing as this issue seems temporarily resolved. We hope to move to sdk47 soon and hopefully this forking will not be needed. Thanks @Reecepbcups for updating.

We’re planing to upgrade Gaia to SDK 0.47 in the following months. In the meantime, please use the suggested fix.

The suggested fix doesn’t work anymore. Latest upgrade to v12.0.0 broke it. No more support for archive nodes. @mpoke can you please re-open the issue?

  1. Reopening, this is still a major issue for archive nodes.
  2. The fix is now broken with v12.0.0. Any suggestions from the team?

This affected Juno as well, quoting Reece:

Reece | Juno-Network — Today at 16:09 I did this back for Juno’s v13 upgrade as well. Is because of a past issue with a chain upgrade which did not add a new modules storeKey

4-5 chains have had issues with this since the change (implemented in 45.12 I believe) 😔

So adding the new storeKey is the way forward, this is backwards compatible. I’ll try to come up with a cleaner patch when I find some time if that sounds good (unless it’s solved before 😃).