bitcoin: Intermittent failures in interface_usdt_mempool.py
https://cirrus-ci.com/task/5779522121891840
[0;34m node0 2023-03-31T11:45:58.812384Z (mocktime: 2023-04-14T11:46:01Z) [http] [httpserver.cpp:257] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:38238 [0m
[0;34m node0 2023-03-31T11:45:58.812543Z (mocktime: 2023-04-14T11:46:01Z) [httpworker.2] [rpc/request.cpp:179] [parse] [rpc] ThreadRPCServer method=getpeerinfo user=__cookie__ [0m
[0;36m test 2023-03-31T11:45:58.814000Z TestFramework (INFO): Hooking into mempool:rejected tracepoint... [0m
[0;36m test 2023-03-31T11:45:59.798000Z TestFramework (INFO): Sending invalid transaction... [0m
[0;36m test 2023-03-31T11:45:59.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:15944: msg_tx(tx=CTransaction(nVersion=2 vin=[CTxIn(prevout=COutPoint(hash=76b2ca9a8c3ffa3c454db9591dfaf5c4979926154e33a5bef67412ea0eaa16c5 n=0) scriptSig= nSequence=0)] vout=[CTxOut(nValue=49.99968800 scriptPubKey=51202913b252fe537830f843bfdc5fa7d20ba48639a87c86ff837b92d083c55ad7c1)] wit=CTxWitness(CScriptWitness(51,c00000000000000000000000000000000000000000000000000000000000000001)) nLockTime=0)) [0m
[0;36m test 2023-03-31T11:45:59.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:15944: msg_ping(nonce=00000002) [0m
[0;34m node0 2023-03-31T11:45:59.799697Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:3169] [ProcessMessage] [net] received: tx (133 bytes) peer=0 [0m
[0;34m node0 2023-03-31T11:45:59.800472Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [txmempool.cpp:658] [check] [mempool] Checking mempool with 2 transactions and 2 inputs [0m
[0;34m node0 2023-03-31T11:45:59.800746Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:4174] [ProcessMessage] [mempoolrej] c917a71e43b825756c1d45985fcc87f8c245db0654f5eeb000a92dc2cccb1d6c from peer=0 was not accepted: min relay fee not met, 0 < 104 [0m
[0;34m node0 2023-03-31T11:45:59.800913Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:3169] [ProcessMessage] [net] received: ping (8 bytes) peer=0 [0m
[0;34m node0 2023-03-31T11:45:59.800953Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net.cpp:2841] [PushMessage] [net] sending pong (8 bytes) peer=0 [0m
[0;36m test 2023-03-31T11:45:59.801000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:15944: msg_pong(nonce=00000002) [0m
[0;34m node0 2023-03-31T11:45:59.849969Z (mocktime: 2023-04-14T11:46:01Z) [http] [httpserver.cpp:257] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:38238 [0m
[0;36m test 2023-03-31T11:45:59.850000Z TestFramework (INFO): Polling buffer... [0m
[0;34m node0 2023-03-31T11:45:59.850174Z (mocktime: 2023-04-14T11:46:01Z) [httpworker.3] [rpc/request.cpp:179] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ [0m
[0;36m test 2023-03-31T11:45:59.898000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... [0m
[0;36m test 2023-03-31T11:45:59.898000Z TestFramework (ERROR): Assertion failed [0m
[0;36m Traceback (most recent call last):[0m
[0;36m File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main[0m
[0;36m self.run_test()[0m
[0;36m File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 339, in run_test[0m
[0;36m self.rejected_test()[0m
[0;36m File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 323, in rejected_test[0m
[0;36m assert_equal(EXPECTED_REJECTED_EVENTS, handled_rejected_events)[0m
[0;36m File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 56, in assert_equal[0m
[0;36m raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))[0m
[0;36m AssertionError: not(1 == 0)[0m
[0;36m test 2023-03-31T11:45:59.899000Z TestFramework (DEBUG): Closing down network thread [0m
[0;36m test 2023-03-31T11:45:59.949000Z TestFramework (INFO): Stopping nodes [0m
[0;36m test 2023-03-31T11:45:59.950000Z TestFramework.node0 (DEBUG): Stopping node [0m
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 44 (44 by maintainers)
Commits related to this issue
- ci: fix asan task name Pointed out in https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-1536434598. — committed to fanquake/bitcoin by fanquake 4 years ago
- Merge bitcoin/bitcoin#27584: ci: fix asan task name bf07e3a47e4a8677077e0508558bb11227989ab2 ci: fix asan task name (fanquake) Pull request description: Pointed out in https://github.com/bitcoin/... — committed to bitcoin-core/gui by fanquake 4 years ago
- Merge bitcoin/bitcoin#27584: ci: fix asan task name bf07e3a47e4a8677077e0508558bb11227989ab2 ci: fix asan task name (fanquake) Pull request description: Pointed out in https://github.com/bitcoin/... — committed to syscoin/syscoin by fanquake 4 years ago
- ci: fix asan task name Pointed out in https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-1536434598. — committed to 0xB10C/bitcoin by fanquake 4 years ago
- ci: fix asan task name Pointed out in https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-1536434598. — committed to RandyMcMillan/bitcoin by fanquake 4 years ago
- no-merge: introduce typo to demonstrate assert_equal issue https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-1578265398 — committed to 0xB10C/bitcoin by 0xB10C 3 years ago
- Merge bitcoin/bitcoin#27831: test: handle failed `assert_equal()` assertions in bcc callback functions 61f4b9b7ad6e992a9dbbbb091e9b7ba9abe529ac Manage exceptions in bcc callback functions (virtu) Pu... — committed to bitcoin/bitcoin by fanquake 3 years ago
- Merge bitcoin/bitcoin#27831: test: handle failed `assert_equal()` assertions in bcc callback functions 61f4b9b7ad6e992a9dbbbb091e9b7ba9abe529ac Manage exceptions in bcc callback functions (virtu) Pu... — committed to syscoin/syscoin by fanquake 3 years ago
- Merge bitcoin/bitcoin#27846: [coinselection] Increase SRD target by change_fee 1771daa815ec014276cfcb30c934b0eaff4d72bf [fuzz] Show that SRD budgets for non-dust change (Murch) 941b8c6539d72890fd4e36... — committed to bitcoin-core/gui by achow101 3 years ago
- Merge bitcoin/bitcoin#27846: [coinselection] Increase SRD target by change_fee 1771daa815ec014276cfcb30c934b0eaff4d72bf [fuzz] Show that SRD budgets for non-dust change (Murch) 941b8c6539d72890fd4e36... — committed to syscoin/syscoin by achow101 3 years ago
- Merge bitcoin/bitcoin#28088: test: Disable known broken USDT test faf8be7c32be00f660eba90d3f07313fb25d5d1c test: Disable known broken USDT test (MarcoFalke) Pull request description: The failure ... — committed to bitcoin-core/gui by fanquake 3 years ago
- Merge bitcoin/bitcoin#28088: test: Disable known broken USDT test faf8be7c32be00f660eba90d3f07313fb25d5d1c test: Disable known broken USDT test (MarcoFalke) Pull request description: The failure ... — committed to syscoin/syscoin by fanquake 3 years ago
- ci: fix asan task name Pointed out in https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-1536434598. — committed to BitgesellOfficial/bitgesell by fanquake 4 years ago
- Merge bitcoin/bitcoin#30559: test: Try previously intermittently broken USDT test again fa2269dd65bac09f22c7cee8a2dfc46694e9a986 test: Try previously intermittently broken USDT test again (MarcoFalke... — committed to bitcoin/bitcoin by fanquake 2 years ago
- Merge bitcoin/bitcoin#30574: test: Disable known broken USDT test for now faed5337435f025811caeb5f782ecbf9683a24b3 test: Disable known broken USDT test for now (MarcoFalke) Pull request description:... — committed to bitcoin/bitcoin by fanquake 2 years ago
- no-merge: try maflcko's recommendation https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-2277655269 — committed to 0xB10C/bitcoin by 0xB10C 2 years ago
- no-merge: try maflcko's recommendation https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-2277655269 — committed to 0xB10C/bitcoin by 0xB10C 2 years ago
- no-merge: try mzumsande's suggestion https://github.com/bitcoin/bitcoin/issues/27380#issuecomment-2284894942 — committed to 0xB10C/bitcoin by 0xB10C 2 years ago
- test: don't use bpf_usdt_readarg_p Instead of using the undocumented bcc helper bpf_usdt_readarg_p(), use bpf_usdt_readarg() [1] and bpf_probe_read_user{_str}() [2, 3] as documented in the bcc USDT r... — committed to 0xB10C/bitcoin by 0xB10C 2 years ago
- Revert "test: Disable known broken USDT test for now" This reverts commit faed5337435f025811caeb5f782ecbf9683a24b3. This commit worked around a lifetime issue likely caused by using bpf_usdt_readarg... — committed to 0xB10C/bitcoin by 0xB10C 2 years ago
https://cirrus-ci.com/task/6053265947754496 ? š¢
Unfortunately, it didnāt (https://github.com/bitcoin/bitcoin/runs/13206824577 from master after #27360 was merged)
To check that this is still an issue, I ran https://github.com/bitcoin/bitcoin/compare/master...0xB10C:bitcoin:refs/heads/2024-11-27380-intermitting-usdt-mempool-failures in CI 20 times. It failed 6 times. I then ran it with https://github.com/0xB10C/bitcoin/commit/8a3eeb48482955e9d0cd26632a03457e35a10568 and it didnāt fail a single time. It seems that getting rid of
bpf_usdt_readarg_pindeed fixes this.Iāll work on a PR to get rid of our
bpf_usdt_readarg_pusage in all tests.Maybe copying the string into a static or global variable would help rule out a lifetime issue?
I guess we can close this for now, and can reopen if #30559 is reverted?
Have seen this a couple times by now.
cc @0xB10C @virtu
Thanks @mzumsande - all 11 CI runs with your patch succeed.
This should confirm itās indeed a lifetime issue.
Thanks! Did a few CI runs of https://github.com/0xB10C/bitcoin/commit/3d806eafa04415791b95ed20525eed2fd54f5b38. Your patch doesnāt seem to improve the situation.
Given that you can reproduce on 50% of your new action runs, does the following diff work around the bug?
https://github.com/bitcoin/bitcoin/actions/runs/10214494976/job/28262011108?pr=29641#step:7:9088
I donāt think the lifetime of
resultis relevant here.GetRejectReason()gives birth to a fresh temporary copy of the string, having its own lifetime.Yeah, I am just guessing that this is a lifetime issue. It would be good to rule this out.
AFAIU the macros themselves donāt copy any data, rather just define a static probe point. Any arguments passed to them just get incorporated into the assembly code. The tracing program (BPF) if loaded into the kernel can run code at the trace points. Here it can choose to, and AFAIU does in our case with BPF, copy the data into a different kernel space map when it hits a tracepoint, which can then be read by the USDT probe in our python test.
So the data does get copied but not by the macro, rather by the BPF program.
https://cirrus-ci.com/task/4626519738286080?logs=ci#L3262
https://github.com/bitcoin/bitcoin/pull/27831 improves the logging, probably.
see https://github.com/0xB10C/bitcoin/commit/4091d621f16b6316eef5551a56c00bf0e2c0e541 and this CI task https://cirrus-ci.com/task/4909169740873728
@0xB10C and I had a look at this today. Hereās what we found so far:
handle_rejected_eventcallback (although we are not sure why they fail).assert_equal()communicates failed assertions via exceptions. Unfortunately exceptions raised in bcc callback functions are not properly propagated (Exception ignored on calling ctypes callback function: <function PerfEventArray._open_perf_buffer.<locals>.raw_cb_ at 0xffff83490e50>), so they only appear on stderr, not in the logs. As a consequence, the logs only show that the test case failed, not why it failed.assert_equals()in bcc callback functions intryblocks and logging anyAssertionErrorexceptions usingself.log(). Once this (or a better solution, preferable one that involves not ignoring exceptions in ctype callback functions, if thatās possible) is merged, the logs should tell us why themempool:rejectedtest fails sporadically.master @ e0a70c5b4f2c691e8d6b507c8ce879f0b0424254. https://cirrus-ci.com/task/5247512164958208:
Again here https://cirrus-ci.com/task/5250935128064000?logs=ci#L3275