core: Postgres sync corrupts blocks from 98493

Hey all, just an update here.

@lazynina put out a fix in this PR:

• https://github.com/deso-protocol/core/pull/212

We are now re-running a full sync from scratch to make sure there aren’t any more surprises, but feel free to give it a shot yourselves in the meantime.

I also wanted to take a moment to apologize for the pain you guys felt on this one. Across the core team, we have treated Postgres as a secondary data store, not used in consensus or our production nodes, and not yet integrated into our extremely thorough test suite.

The reason for this is that we have some fear that Postgres will not scale well as DeSo usage increases. And so, while it’s useful for apps getting off the ground, we have yet to make a go or no-go decision with regard to whether we want to offer full long-term support for it.

The above being said, given that serious apps like NFTz are now relying on it, I think we are leaning toward working around Postgres’s limitations. In particular, we have made a few breakthroughs with a new syncing mechanism we call Hypersync that, if applied to our Postgres path, should allow it to scale reasonably well.

All this to say: We’re trying really hard to support Postgres at the level that we support all core code, and we appreciate your patience with us as we work out the kinks. I know how useful it is to have a SQL interface over the DeSo data, and we’re doing everything we can to keep this a viable path for you guys long-term.

My resync just hit another issue FYI. We’re digging into it. At this point it may be faster to just bite the bullet and hook Postgres into our test suite.

I0206 13:38:56.471424  784421 server.go:1147] Server._handleBlock: Received block ( 98599 / 101808 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=178 ]                                                                                                                     
E0206 13:38:56.492515  784421 server.go:1141] Server._handleBlock: Encountered an error processing block <Header: <98599, 0000000000004d02262295ffe7f63fe770ce4442d095927113cd25a6938d11f1, 1 >, Signer Key: BC1YLh768bVj2R3QpSiduxcvn7ipxF3L3XHsabZYtCGtsinUnNrZvNN>. Disconnecting 
from peer [ Remote Address: 34.123.41.111:17000 PeerID=178 ]: Error while processing block: : ConnectBlock: : ConnectTransaction: : _connectAcceptNFTBid: _helpConnectNFTSold: Profile missing for additional coin royalty for pub key: BC1YLjSCiP67Zq3v7fwc613Akd8hvQrs2ChuPPqRFTfFQGdmtomNR1J tBCKYZSjZN4yvxqraWvY3dQZ2SjJjNdxHSyK7WLzU34TDYjL26XoRq                                                                  

I will try and do this today, which should allow us to catch all issues like this in the future.

sync complete @diamondhands0

Just an update here: Still hunting down the issue. I believe it was due to the encoding of the royalty maps, which I just fixed in this commit:

• https://github.com/deso-protocol/core/pull/212/commits/4809196253d78a6dee8493d8b18a7e6e39c1b2c6

My sync runs pretty quickly, so my hope is that this works, and we can defer integrating Postgres into our test suite, which is a bigger effort than I expected.

@mvanhalen I see the issue. We need to initialize the make with a make(map[string]uint64) before we start filling it in.

@mvanhalen - thanks for flagging this. I’m out of pocket this weekend, but will investigate first thing on Monday.

Just hit the same error myself. Should be able to fix it tonight. Luckily this one shouldn’t require a resync after it’s fixed.

Yep looks like Messaging v3 does not support postgres yet

The metadata needs to be added:

https://github.com/deso-protocol/core/blob/8107b9be5883f117a676bb72b1fcf4707404461c/lib/postgres.go#L107

and then used here

https://github.com/deso-protocol/core/blob/4809196253d78a6dee8493d8b18a7e6e39c1b2c6/lib/postgres.go#L1125

and a migration added to create the required tables:

https://github.com/deso-protocol/core/tree/main/migrate

cc @AeonSw4n

k great ive just rebuild with those changes and sync was at 49k so lets see if that manages to remain ok.

Working with @tijno on this @lazynina .

Added AdditionalNFTRoyaltiesToCoinsBasisPoints: make(map[string]uint64), AdditionalNFTRoyaltiesToCreatorsBasisPoints: make(map[string]uint64),

below extra data on https://github.com/deso-protocol/core/blob/main/lib/postgres.go#L1447

It compiled and running that for a sync. Will know the result tomorrow.

@diamondhands0 @maebeam backend images have not been updated yet

Green with envy here 🤣

@diamondhands0 @lazynina as per @tijno’s comment above, if you could update the docker image that’d be awesome 👍

Fully synced now as well! nftz.zone back in sync as well. Now full focus on version 2 again.

Looks like image for backend has not yet been regenerated ?

stable and latest still 4 days old.

usually its pretty instant.

I think it’s finally fixed. I just finished syncing a Postgres node, and it made it all the way to the end. This was the final commit that was needed to fix all the issues:

• https://github.com/deso-protocol/core/commit/1b9d4d61499d66f3c3f2911496785aa6b95dd5d5

I cut a new release v2.0.5 so just rebooting your nodes should get you to the end if you had the MessagingGroup issue.

Please let me know once your syncs complete!

Just synced past 98599 till 98714. Got a new error. @diamondhands0 @tijno Which is progress.

` Server._handleBlock: Encountered an error processing block <Header: < 98714, 0000000000005428375f6ee0666ae8bde2cc3 432da490d53f8588341c542b4a6, 1 >, Signer Key: BC1YLh768bVj2R3QpSiduxcvn7ipxF3L3XHsabZYtCGtsinUnNrZvNN>. Disconnecting from peer [ Remote Address: 35.232.92.5:17000 PeerID=9 ]:

Error while processing block: : ProcessBlock: Problem upserting block and transactions: InsertTransactionTx: Unimplemented txn type MESSAGING_GROUP `

Seems group messaging transactions have an issue with Postgres

Cool, just rebuild and syncing from 94500 so will let all know.

Thank you so much for the alert on this - I’ll pause my sync (~80k) in the hope we can fix this without another resync.

Confirmed on my end too. Also mentioned by kanshi on discord.