solana: Duplicate Signature Being Returned When Using `getSignaturesForAddress` method on web3

I think that I’m running into something similar to this. Sometimes, getSignaturesForAddress will return signatures that are actually before my until parameter. This makes pagination pretty difficult.

Anecdotally, this happens when I’m fetching transactions that are “near the tip” eg…polling for new transactions. I haven’t seen this happen when I’m backfilling older transactions.

I’m using the "confirmed" finality parameter. Maybe I should be using the “finalized” parameter to avoid this? Also, I’m using a GenesysGo endpoint.

Hmm yeah it must be an RPC problem, can you bring it up with them to debug?

Happy to help diagnose the issue, I’ve created this issue to come up with a solution to avoid this problem: https://github.com/solana-labs/solana/issues/24620

For “until” requests, the RPC node will collect signatures until it hits the “until” signature. When your script gets the following response…

1000 new sigs from 8:21:40 p.m.:130360512 to 8:20:16 p.m.:130360362

The issue seems to be that the “until” signature is not found at all so the RPC node will just return the 1000 most recent signatures for an address. Note that the most recent block time for that response was “8:21:40 p.m.:130360512” which is actually not as recent as the block time from the previous response. So this leads me to believe that the your requests are being handled by two different RPC nodes and sometimes your request is handled by a node which hasn’t finalized as recent of a block as the other node.

I think the best recommendation I have for you is to ignore responses for data older than what you have already retrieved and retry later. Otherwise using a single RPC node should work ok

I’ve been able to reduce repro this in a small snippet. Here is a little bit of code that polls Raydium’s address for new transactions.

  const address = new PublicKey("675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8");
  const mostRecentSignature = await connection.getSignaturesForAddress(address, { limit: 1 }, "finalized");
  const fetchOptions: SignaturesForAddressOptions = { until: mostRecentSignature[0].signature };

  while(true) {
    let signatures = await connection.getSignaturesForAddress(address, fetchOptions, "finalized");

    if (signatures.length > 0) {
      const firstTime = (new Date(signatures[0].blockTime!*1000.0)).toLocaleTimeString();
      const lastTime = (new Date(signatures[signatures.length-1].blockTime!*1000.0)).toLocaleTimeString();
      const firstSlot = signatures[0].slot;
      const lastSlot = signatures[signatures.length-1].slot;
      console.log(`${signatures.length} new sigs from ${firstTime}:${firstSlot} to ${lastTime}:${lastSlot}`);

      fetchOptions.until = signatures[0].signature;
    }

    await new Promise(r => setTimeout(r, 1000));
  }

This loop will happily plug away and find new transactions as they come in and log the following output as expected:

55 new sigs from 8:15:30 p.m.:130359880 to 8:15:28 p.m.:130359877
31 new sigs from 8:15:31 p.m.:130359882 to 8:15:31 p.m.:130359881
64 new sigs from 8:15:35 p.m.:130359885 to 8:15:32 p.m.:130359883
64 new sigs from 8:15:36 p.m.:130359887 to 8:15:35 p.m.:130359886
14 new sigs from 8:15:37 p.m.:130359888 to 8:15:37 p.m.:130359888
32 new sigs from 8:15:37 p.m.:130359889 to 8:15:37 p.m.:130359889
20 new sigs from 8:15:38 p.m.:130359890 to 8:15:38 p.m.:130359890
77 new sigs from 8:15:43 p.m.:130359895 to 8:15:40 p.m.:130359891

However, about once every 30 minutes, there’s an anomaly, where a bunch of older transactions will get returned as follows. Notice the 1000 transaction results that all appear to be the same.

46 new sigs from 8:21:47 p.m.:130360521 to 8:21:42 p.m.:130360513
8 new sigs from 8:21:48 p.m.:130360522 to 8:21:48 p.m.:130360522
1000 new sigs from 8:21:40 p.m.:130360512 to 8:20:16 p.m.:130360362
90 new sigs from 8:21:51 p.m.:130360529 to 8:21:42 p.m.:130360513
1000 new sigs from 8:21:40 p.m.:130360512 to 8:20:16 p.m.:130360362
263 new sigs from 8:22:18 p.m.:130360576 to 8:21:42 p.m.:130360513
1000 new sigs from 8:21:40 p.m.:130360512 to 8:20:16 p.m.:130360362
276 new sigs from 8:22:20 p.m.:130360581 to 8:21:42 p.m.:130360513

Real time follow up: I still experience the same thing using “finalized”.

Ok, if you are able to reproduce again, please sure the results and we can take a look