forest: Send FIL failed

Describe the bug

The wallet test failed, and the FIL was not sent. The target wallet didn’t receive anything and the transaction is not visible in the source wallet. https://calibration.filscan.io/address/general?address=t1ac6ndwj6nghqbmtbovvnwcqo577p6ox2pt52q2y (it should’ve happened around 2023-03-23 17:47:17, target t1fye76pfymru4y2fqbgdxswpwqt3ukeems2nzfna)

This means that either we have a bug in the test setup or the send command doesn’t always send the funds, which would be a significant issue.

To Reproduce

It happened in this PR https://github.com/ChainSafe/forest/pull/2709 (which didn’t introduce any logic changes)

The job failed: https://github.com/ChainSafe/forest/actions/runs/4503553182/jobs/7926889785

Log output

Wallet tests
Importing preloaded wallet key
t1ac6ndwj6nghqbmtbovvnwcqo577p6ox2pt52q2y
Exporting key
Fetching metrics
--2023-03-23 17:47:17--  http://localhost:6116/metrics
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:6116... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:6116... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28751 (28K) [text/plain]
Saving to: ‘metrics.log’

     0K .......... .......... ........                        100%  404M=0s

2023-03-23 17:47:17 (404 MB/s) - ‘metrics.log’ saved [28751/28751]

Listing wallet balances
Address                                   Default Balance
t1ac6ndwj6nghqbmtbovvnwcqo577p6ox2pt52q2y           ~100 FIL
Creating a new address to send FIL to
t1fye76pfymru4y2fqbgdxswpwqt3ukeems2nzfna
Listing wallet balances
Address                                   Default Balance
t1ac6ndwj6nghqbmtbovvnwcqo577p6ox2pt52q2y  X        ~100 FIL
t1fye76pfymru4y2fqbgdxswpwqt3ukeems2nzfna           0 atto FIL
Sending FIL to the above address
Checking balance of t1fye76pfymru4y2fqbgdxswpwqt3ukeems2nzfna...
Checking balance 1/10
Checking balance 2/10
Checking balance 3/10
Checking balance 4/10
Checking balance 5/10
Checking balance 6/10
Checking balance 7/10
Checking balance 8/10
Checking balance 9/10
Checking balance 10/10
Address                                   Default Balance
t1ac6ndwj6nghqbmtbovvnwcqo577p6ox2pt52q2y  X        ~100 FIL
t1fye76pfymru4y2fqbgdxswpwqt3ukeems2nzfna           0 atto FIL
FIL amount should match
Shutting down Forest node

Expected behaviour

The test passes.

Screenshots

Environment (please complete the following information):

OS: buildjet runner, Ubuntu 20.04
Rust version(e.g. rustc --version)
Branch/commit 2dc3442

Other information and links

About this issue

Original URL
State: open
Created a year ago
Comments: 18 (18 by maintainers)

Most upvoted comments

@elmattic Would you like to contact Lotus devs to validate that this is the solution to the same problem?

LesnyRumcajs on Sep 29, 2023

This is a great finding, unfortunately it’s not the same error than the one in the CI where the message was successfully added to the pool and its CID was printed.

elmattic on Jun 8, 2023

I think that a bad message Nonce could cause the transaction to fail.

In Forest node I can see: WARN forest_message_pool::msg_chain: encountered message from actor t1ekkzekiozleakm4jauekqqc6uhn3dcwzcrrqrry with nonce 53 less than the current nonce 54

Let’s sum up what is happening here: The sending actor has its Nonce with a value of 53.

We’re sending to t1mljyv5pzvu7jgzuvzfog2ycsd5fdh4y7c2z6t6q 500 afil. Forest creates the message using the Nonce from sending actor Nonce (53).

Transaction is mined successfully in block bafy2bzaceacr42ctk3z7i7vuqkdyfzydscvcv47dgzzdiunbh3qmcc6dp65su.

Sending actor Nonce is somewhat being updated in the blockchain to 54 (todo: find who does that).

We’re sending a new transaction (500 afill to t1i63uuuqrgqvntdekae4wmomz5abgnwltm3buyia). Nonce used here is still 53 (this could be our issue):

{
  "Message": {
    "Version": 0,
    "To": "t1i63uuuqrgqvntdekae4wmomz5abgnwltm3buyia",
    "From": "t1ekkzekiozleakm4jauekqqc6uhn3dcwzcrrqrry",
    "Nonce": 53,
    "Value": "500",
    "GasLimit": 5866637,
    "GasFeeCap": "100374",
    "GasPremium": "99320",
    "Method": 0,
    "Params": "",
    "CID": {
      "/": "bafy2bzacedw3qgd4dy3qyizg3ian44i5yefkjtzknetdksqqwi7khc2cuvpbq"
    }
  },
  "Signature": {
    "Type": 1,
    "Data": "1drZdRSXUCMEv0niS/3XSCd8MhE3UZkFei4EYpjh+ismkBpVDGqNwAyXRxccVy8Uqhoj2mFkmOMI6GliekqC0wA="
  },
  "CID": {
    "/": "bafy2bzacecmeugxrjtaaxkp5nci7oqblt2bccsqcrc57dqx3do6kfvymhgqa6"
  }
}

Transaction is rejected because it would expect a Nonce of 54.

It’s not even reaching the Lotus node running the explorer (probably because detected as invalid by network nodes during p2p message broadcast).

Generally speaking, even if we didn’t had this issue here, there could be a race between two CI jobs since we use the same sending actor.

elmattic on Jun 6, 2023