graph-node: Encountering a consistent indexing error on Matic "Found no transaction for event"

Hey @miguelmota and others here: We cleaned our db and apparently the new Bor version should fix our problem. Your subgraph is syncing. We hope that we now finally have a stable situation. Let us know if there are any problems.

Sync error - block #16797184

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16797184 (0x1d93…909a), transaction df952e2b5bb59beb5acbced5be788277312ec838142a808fcdbe437cff8ff12f: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

Sync error - block #16785216

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16785216 (0xdc6c…85c5), transaction 8d63b25cbc8e3fa4a62b4513d70151f8485c1fe1d5da044a83789a47219a6b55: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

Sync error - block #16776192

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16776192 (0x072c…c74c), transaction e2897b360222e76480388b4c55a020250717910c87c175a4e00c7230758ab4ce: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

Sync error - block: #16766208

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16766208 (0xcf8f…3600), transaction 448b929a92f44df4f7e28de7144be8229485f27fa7f89c0ef165885d5f78d133: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

Hypothesis that this is caused by a race condition relating to use of block number instead of block hash for Bor transactions, which should be fixed in this PR: https://github.com/maticnetwork/bor/pull/160

Still experiencing this issue with zkSync Era mainnet:

Error: failed to process trigger: block #9376171 (0x01fa…404a), transaction 0000000000000000000000000000000000000000000000000000000000000000: Found no transaction for event

@tilacog : which table are you referring to?

Currently, to resync a faulty block you must delete the block record from its database table and restart graph-node afterwards.

@tilacog could you please help us out with https://thegraph.com/legacy-explorer/subgraph/lemonadesocial/lemonade-marketplace?version=pending

failed to process trigger: block #18223680 (0xd0aa…7fee), transaction 7651a8dca5c6ece8758e46aabf29daddc3fdd504dded7d6cde0f0c9f29ff5d26: Found no transaction for event

Thank you

Sync error - block #16800192

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16800192 (0x131d…5b30), transaction 18ad7854383bfb6bf2712aa95b710ffc254ce91e882c3ed6306f929322dcdd53: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog

Hey @tilacog

Messages on Polygon that are sent from L1 appear as events that are emitted from the 0x0 address. For example, here is a “transaction” that has an event from one of our users that sent a message from L1 -> L2. This message emitted an event (topic 0x320958176930804eb66c2343c7343fc0367dc16249590c0f195783bee199d094) on L2 that you can see in this transaction.

On any other chain, I would expect this event to be emitted on the transaction that actually updated state. However, on Polygon, the transaction that emits the event is not the transaction that updates state. Might this be the issue that the indexer is running into?

@shanefontaine this will definitely work consistently in the future.

It is still unclear why blocks appear without those transactions at ingestion time, but that error is being actively studied by our and our providers’ teams.

I’ve refetched that block for your subgraph, so It should get back to sync soon.

Sync error - block #16735872

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16735872 (0xbd03…dbf3), transaction e37d065810059bbba193706ee2f09f7751949f260b13ee63079efbafe09cdf01: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

@tilacog It seems to fail to sync the block every few thousand blocks and the subgraph still has 1-2M blocks to index to catch up to head.

{"data":{"indexingStatusForPendingVersion":{"fatalError":{"message":"failed to process trigger: block #16710656 (0xc5c3…6a47), transaction 78e1ceac8041c2046f732c9ed6680fc6e5ed96c6d351f873945e2af8d2c7d67a: Found no transaction for event"},"nonFatalErrors":[],"subgraph":"QmfDg1oHAPb7TPuy882BfBFZrJKMPNEuib98Em7rFds2nb"}}}

Any way to automate the re-sync of the block?

We’ve been trying to fully sync this subgraph for some time now 😅

@tilacog it was making good progress but just got stuck again recently, if you don’t mind taking a look. Thanks!

@tilacog another block sync failure on subgraph. Thanks!

Hey @miguelmota, I’m following that subgraph closely and re-fetching faulty blocks from our storage. As of now, it has resumed syncing, but I’ll keep an eye on it.

Unfortunately, we’ll have to do this manually until this issue persists.

@nachomazzara the issue seems pretty specifically tied to 0x000 -> 0x000 transactions, rather than a more general re-org issue (if it was as you described, I think we would see it for other transactions, unless I have misunderstood your suggestion?)

Thanks for flagging @miguelmota, @tilacog can you take a look?

@nachomazzara we get this also for reasonably old blocks, so I don’t think so.

We cleared the cache, which has resolved this issue for the subgraph we were testing with - @TravelingTechGuy you should be able to restart this subgraph before the problematic block (15167232), and it should index successfully. I will close this issue, but please do re-open if you see a recurrence!

We have identified the issue, which relates to our Matic RPC not including all transactions in eth_getBlockByHash at some point, which was then cached in the database, resulting in the ongoing issue. This may relate to the new Bor requirement to specifically return bor logs. We are planning to clear the cache which should resolve the issue, as Matic RPCs are now returning the correct number of transactions for these blocks cc @leoyvens @tilacog