bitcoin: Intermittent failures in interface_usdt_mempool.py

https://cirrus-ci.com/task/5779522121891840

 node0 2023-03-31T11:45:58.812384Z (mocktime: 2023-04-14T11:46:01Z) [http] [httpserver.cpp:257] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:38238 
 node0 2023-03-31T11:45:58.812543Z (mocktime: 2023-04-14T11:46:01Z) [httpworker.2] [rpc/request.cpp:179] [parse] [rpc] ThreadRPCServer method=getpeerinfo user=__cookie__ 
 test  2023-03-31T11:45:58.814000Z TestFramework (INFO): Hooking into mempool:rejected tracepoint... 
 test  2023-03-31T11:45:59.798000Z TestFramework (INFO): Sending invalid transaction... 
 test  2023-03-31T11:45:59.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:15944: msg_tx(tx=CTransaction(nVersion=2 vin=[CTxIn(prevout=COutPoint(hash=76b2ca9a8c3ffa3c454db9591dfaf5c4979926154e33a5bef67412ea0eaa16c5 n=0) scriptSig= nSequence=0)] vout=[CTxOut(nValue=49.99968800 scriptPubKey=51202913b252fe537830f843bfdc5fa7d20ba48639a87c86ff837b92d083c55ad7c1)] wit=CTxWitness(CScriptWitness(51,c00000000000000000000000000000000000000000000000000000000000000001)) nLockTime=0)) 
 test  2023-03-31T11:45:59.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:15944: msg_ping(nonce=00000002) 
 node0 2023-03-31T11:45:59.799697Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:3169] [ProcessMessage] [net] received: tx (133 bytes) peer=0 
 node0 2023-03-31T11:45:59.800472Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [txmempool.cpp:658] [check] [mempool] Checking mempool with 2 transactions and 2 inputs 
 node0 2023-03-31T11:45:59.800746Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:4174] [ProcessMessage] [mempoolrej] c917a71e43b825756c1d45985fcc87f8c245db0654f5eeb000a92dc2cccb1d6c from peer=0 was not accepted: min relay fee not met, 0 < 104 
 node0 2023-03-31T11:45:59.800913Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:3169] [ProcessMessage] [net] received: ping (8 bytes) peer=0 
 node0 2023-03-31T11:45:59.800953Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net.cpp:2841] [PushMessage] [net] sending pong (8 bytes) peer=0 
 test  2023-03-31T11:45:59.801000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:15944: msg_pong(nonce=00000002) 
 node0 2023-03-31T11:45:59.849969Z (mocktime: 2023-04-14T11:46:01Z) [http] [httpserver.cpp:257] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:38238 
 test  2023-03-31T11:45:59.850000Z TestFramework (INFO): Polling buffer... 
 node0 2023-03-31T11:45:59.850174Z (mocktime: 2023-04-14T11:46:01Z) [httpworker.3] [rpc/request.cpp:179] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2023-03-31T11:45:59.898000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2023-03-31T11:45:59.898000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 339, in run_test
                                       self.rejected_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 323, in rejected_test
                                       assert_equal(EXPECTED_REJECTED_EVENTS, handled_rejected_events)
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 56, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not(1 == 0)
 test  2023-03-31T11:45:59.899000Z TestFramework (DEBUG): Closing down network thread 
 test  2023-03-31T11:45:59.949000Z TestFramework (INFO): Stopping nodes 
 test  2023-03-31T11:45:59.950000Z TestFramework.node0 (DEBUG): Stopping node 

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 44 (44 by maintainers)

Commits related to this issue

Most upvoted comments

Maybe it will ā€œfix itselfā€ after #27360 ?

Unfortunately, it didn’t (https://github.com/bitcoin/bitcoin/runs/13206824577 from master after #27360 was merged)

Happy to give this another spin.

To check that this is still an issue, I ran https://github.com/bitcoin/bitcoin/compare/master...0xB10C:bitcoin:refs/heads/2024-11-27380-intermitting-usdt-mempool-failures in CI 20 times. It failed 6 times. I then ran it with https://github.com/0xB10C/bitcoin/commit/8a3eeb48482955e9d0cd26632a03457e35a10568 and it didn’t fail a single time. It seems that getting rid of bpf_usdt_readarg_p indeed fixes this.

I’ll work on a PR to get rid of our bpf_usdt_readarg_p usage in all tests.

Maybe copying the string into a static or global variable would help rule out a lifetime issue?

diff --git a/src/validation.cpp b/src/validation.cpp
index 9126d8e928..683d13e0e5 100644
--- a/src/validation.cpp
+++ b/src/validation.cpp
@@ -1863,6 +1863,8 @@ MempoolAcceptResult AcceptToMemoryPool(Chainstate& active_chainstate, const CTra
     CTxMemPool& pool{*active_chainstate.GetMempool()};
 
     std::vector<COutPoint> coins_to_uncache;
+    static char string_buffer[120];
+
     auto args = MemPoolAccept::ATMPArgs::SingleAccept(chainparams, accept_time, bypass_limits, coins_to_uncache, test_accept);
     MempoolAcceptResult result = MemPoolAccept(pool, active_chainstate).AcceptSingleTransaction(tx, args);
     if (result.m_result_type != MempoolAcceptResult::ResultType::VALID) {
@@ -1873,9 +1875,10 @@ MempoolAcceptResult AcceptToMemoryPool(Chainstate& active_chainstate, const CTra
 
         for (const COutPoint& hashTx : coins_to_uncache)
             active_chainstate.CoinsTip().Uncache(hashTx);
+        strncpy(string_buffer, result.m_state.GetRejectReason().c_str(), 120 - 1);
         TRACE2(mempool, rejected,
                 tx->GetHash().data(),
-                result.m_state.GetRejectReason().c_str()
+                string_buffer
         );
     }
     // After we've (potentially) uncached entries, ensure our coins cache is still within its size limits

I guess we can close this for now, and can reopen if #30559 is reverted?

Have seen this a couple times by now.

cc @0xB10C @virtu

Thanks @mzumsande - all 11 CI runs with your patch succeed.

This should confirm it’s indeed a lifetime issue.

Given that you can reproduce on 50% of your new action runs, does the following diff work around the bug?

diff --git a/src/validation.cpp b/src/validation.cpp
index cf4dce3762..d2b6969370 100644
--- a/src/validation.cpp
+++ b/src/validation.cpp
@@ -1873,10 +1873,13 @@ MempoolAcceptResult AcceptToMemoryPool(Chainstate& active_chainstate, const CTra
 
         for (const COutPoint& hashTx : coins_to_uncache)
             active_chainstate.CoinsTip().Uncache(hashTx);
+        std::string copy{result.m_state.GetRejectReason()};
+        auto c_str{copy.c_str()};
         TRACE2(mempool, rejected,
                 tx->GetHash().data(),
-                result.m_state.GetRejectReason().c_str()
+                c_str
         );
+        UninterruptibleSleep(1ms);
     }
     // After we've (potentially) uncached entries, ensure our coins cache is still within its size limits
     BlockValidationState state_dummy;

https://github.com/bitcoin/bitcoin/actions/runs/10214494976/job/28262011108?pr=29641#step:7:9088

test  2024-08-02T11:52:52.096000Z TestFramework (INFO): Hooking into mempool:rejected tracepoint... 
 test  2024-08-02T11:52:53.880000Z TestFramework (INFO): Sending invalid transaction... 
 node0 2024-08-02T11:52:53.880644Z (mocktime: 2024-08-16T11:52:53Z) [http] [httpserver.cpp:305] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:46704 
 node0 2024-08-02T11:52:53.880818Z (mocktime: 2024-08-16T11:52:53Z) [httpworker.2] [rpc/request.cpp:232] [parse] [rpc] ThreadRPCServer method=getblockchaininfo user=__cookie__ 
 test  2024-08-02T11:52:53.883000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:13268: msg_tx(tx=CTransaction(version=2 vin=[CTxIn(prevout=COutPoint(hash=6e95b1706527bac148b682dad0df924b578a2b92fa24de08d5eb6e846b06c07e n=0) scriptSig= nSequence=0)] vout=[CTxOut(nValue=50.00000000 scriptPubKey=51202913b252fe537830f843bfdc5fa7d20ba48639a87c86ff837b92d083c55ad7c1)] wit=CTxWitness(CScriptWitness(51,c00000000000000000000000000000000000000000000000000000000000000001)) nLockTime=0)) 
 test  2024-08-02T11:52:53.884000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:13268: msg_ping(nonce=00000000) 
 test  2024-08-02T11:52:53.884000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:13268: msg_ping(nonce=00000002) 
 node0 2024-08-02T11:52:53.886782Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3710] [ProcessMessage] [net] received: tx (133 bytes) peer=0 
 node0 2024-08-02T11:52:53.887424Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [txmempool.cpp:664] [check] [mempool] Checking mempool with 0 transactions and 0 inputs 
 node0 2024-08-02T11:52:53.887636Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3209] [ProcessInvalidTx] [mempoolrej] 1ca65439fa59f1a719bc6cfd2dd34fb9c2dc6a8c699ece5b4d5397b084f37f95 (wtxid=15bb5f2afba6c18e6ba825160ebcf8fb735e2755045400439d1242cb6499a4e9) from peer=0 was not accepted: min relay fee not met, 0 < 104 
 node0 2024-08-02T11:52:53.887802Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3710] [ProcessMessage] [net] received: ping (8 bytes) peer=0 
 node0 2024-08-02T11:52:53.887841Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net.cpp:3808] [PushMessage] [net] sending pong (8 bytes) peer=0 
 node0 2024-08-02T11:52:53.887991Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3710] [ProcessMessage] [net] received: ping (8 bytes) peer=0 
 test  2024-08-02T11:52:53.888000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:13268: msg_pong(nonce=00000000) 
 test  2024-08-02T11:52:53.888000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:13268: msg_pong(nonce=00000002) 
 node0 2024-08-02T11:52:53.888038Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net.cpp:3808] [PushMessage] [net] sending pong (8 bytes) peer=0 
 node0 2024-08-02T11:52:53.935933Z (mocktime: 2024-08-16T11:52:53Z) [http] [httpserver.cpp:305] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:46704 
 test  2024-08-02T11:52:53.936000Z TestFramework (INFO): Polling buffer... 
 node0 2024-08-02T11:52:53.936089Z (mocktime: 2024-08-16T11:52:53Z) [httpworker.0] [rpc/request.cpp:232] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2024-08-02T11:52:53.937000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2024-08-02T11:52:53.937000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
                                       self.run_test()
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 318, in run_test
                                       self.rejected_test()
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 299, in rejected_test
                                       assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 75, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not( == min relay fee not met)

neither tx nor result go out of scope before it’s called

I don’t think the lifetime of result is relevant here. GetRejectReason() gives birth to a fresh temporary copy of the string, having its own lifetime.

All that said, it’s obviously still going wrong somehow…

Yeah, I am just guessing that this is a lifetime issue. It would be good to rule this out.

AFAIU the macros themselves don’t copy any data, rather just define a static probe point. Any arguments passed to them just get incorporated into the assembly code. The tracing program (BPF) if loaded into the kernel can run code at the trace points. Here it can choose to, and AFAIU does in our case with BPF, copy the data into a different kernel space map when it hits a tracepoint, which can then be read by the USDT probe in our python test.

So the data does get copied but not by the macro, rather by the BPF program.

https://cirrus-ci.com/task/4626519738286080?logs=ci#L3262

 node0 2023-06-29T19:45:37.525879Z (mocktime: 2023-07-13T19:45:38Z) [httpworker.2] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2023-06-29T19:45:37.526000Z TestFramework (INFO): Polling buffer... 
 test  2023-06-29T19:45:37.527000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2023-06-29T19:45:37.527000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 329, in run_test
                                       self.rejected_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 310, in rejected_test
                                       assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 57, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not( == min relay fee not met)

https://github.com/bitcoin/bitcoin/pull/27831 improves the logging, probably.

Can it be forced to ā€œreproduceā€ on master by adding a typo to the error string in assert_equal?

see https://github.com/0xB10C/bitcoin/commit/4091d621f16b6316eef5551a56c00bf0e2c0e541 and this CI task https://cirrus-ci.com/task/4909169740873728

@0xB10C and I had a look at this today. Here’s what we found so far:

  1. We believe the issue is caused by one or more of the assertions failing in the handle_rejected_event callback (although we are not sure why they fail).
  2. The test framework’s assert_equal() communicates failed assertions via exceptions. Unfortunately exceptions raised in bcc callback functions are not properly propagated (Exception ignored on calling ctypes callback function: <function PerfEventArray._open_perf_buffer.<locals>.raw_cb_ at 0xffff83490e50>), so they only appear on stderr, not in the logs. As a consequence, the logs only show that the test case failed, not why it failed.
  3. Because (2) is a general problem with functional tests for tracepoints, I’ll create a PR that fixes the issue for all of them. The best solution we came up with so far is wrapping all assert_equals() in bcc callback functions in try blocks and logging any AssertionError exceptions using self.log(). Once this (or a better solution, preferable one that involves not ignoring exceptions in ctype callback functions, if that’s possible) is merged, the logs should tell us why the mempool:rejected test fails sporadically.

master @ e0a70c5b4f2c691e8d6b507c8ce879f0b0424254. https://cirrus-ci.com/task/5247512164958208:

 node0 2023-05-10T17:29:06.449483Z (mocktime: 2023-05-24T17:29:09Z) [httpworker.1] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2023-05-10T17:29:06.450000Z TestFramework (INFO): Polling buffer... 
 test  2023-05-10T17:29:06.486000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2023-05-10T17:29:06.486000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 339, in run_test
                                       self.rejected_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 323, in rejected_test
                                       assert_equal(EXPECTED_REJECTED_EVENTS, handled_rejected_events)
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 55, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not(1 == 0)
 test  2023-05-10T17:29:06.487000Z TestFramework (DEBUG): Closing down network thread
...
interface_usdt_mempool.py                              | āœ– Failed  | 5 s