bitcoin: ci: failure in `wallet_multiwallet.py --legacy-wallet` - (`void wallet::UnloadWallet(std::shared_ptr &&): Assertion 'it.second' failed.`)

https://cirrus-ci.com/task/5692425725804544?logs=ci#L3370:

 node0 2023-12-13T12:44:27.733356Z [httpworker.1] [rpc/request.cpp:187] [parse] [rpc] ThreadRPCServer method=unloadwallet user=__cookie__ 
 test  2023-12-13T12:44:27.782000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
                                       self.run_test()
                                     File "/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/wallet_multiwallet.py", line 292, in run_test
                                       assert_equal(got_loading_error, True)
                                     File "/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/util.py", line 57, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not(False == True)
 test  2023-12-13T12:44:27.784000Z TestFramework (DEBUG): Closing down network thread 
 test  2023-12-13T12:44:27.834000Z TestFramework (INFO): Stopping nodes 
 test  2023-12-13T12:44:27.834000Z TestFramework.node0 (DEBUG): Stopping node 
 node0 stderr Error: Specified -walletdir "wallets" is a relative path 
 node0 stderr bitcoin-node: wallet/wallet.cpp:246: void wallet::UnloadWallet(std::shared_ptr<CWallet> &&): Assertion `it.second' failed. 
 node0 stderr Warning: Ignoring duplicate -wallet w1. 
 node0 stderr Error: Specified -walletdir "wallets" does not exist 
 node0 stderr Error: Specified -walletdir "/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20231213_124140/wallet_multiwallet_255/node0/regtest/wallets/notadir" is not a directory 
 node0 stderr Error: BerkeleyDatabase: Can't open database w8_copy (duplicates fileid 378185002c000000dd7bcafdc6f3010000000000 from w8) 
 node0 stderr Error: Error parsing command line arguments: Invalid parameter -upgradewallet 
 node0 stderr Error: Specified -walletdir "bad" does not exist 
 node0 stderr Error: Invalid -wallet path 'w8_symlink'. -wallet path should point to a directory where wallet.dat and database/log.?????????? files can be stored, a location where such a directory could be created, or (for backwards compatibility) the name of an existing data file in -walletdir ("/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20231213_124140/wallet_multiwallet_255/node0/regtest/wallets") 
 node0 stderr Error: Specified -walletdir "debug.log" is not a directory 
 node1 stderr Error: Error initializing wallet database environment "/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20231213_124140/wallet_multiwallet_255/competing_walletdir"! 

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (24 by maintainers)

Commits related to this issue

Most upvoted comments

This seems like a bug in the UnloadWallet function, where it just crashes if it is called by more than one thread at the same time for the same wallet. (For example if there are multiple unloadwallet RPC calls at the same time, or if there is an unloadwallet RPC call at the same time as a shutdown.) Possible fix would be:

--- a/src/wallet/wallet.cpp
+++ b/src/wallet/wallet.cpp
@@ -244,15 +244,19 @@ void UnloadWallet(std::shared_ptr<CWallet>&& wallet)
 {
     // Mark wallet for unloading.
     const std::string name = wallet->GetName();
+    bool notify_unload;
     {
         LOCK(g_wallet_release_mutex);
         auto it = g_unloading_wallet_set.insert(name);
-        assert(it.second);
+        // Multiple threads could try to unload the wallet at the same time, but
+        // if this is the first thread, notify the GUI that it should release
+        // any pointers to the wallet.
+        notify_unload = it.second;
     }
     // The wallet can be in use so it's not possible to explicitly unload here.
     // Notify the unload intent so that all remaining shared pointers are
     // released.
-    wallet->NotifyUnload();
+    if (notify_unload) wallet->NotifyUnload();
 
     // Time to ditch our shared_ptr and wait for ReleaseWallet call.
     wallet.reset();

This wallet unloading code is messy in general, and there’s an old comment about cleaning up https://github.com/bitcoin/bitcoin/pull/18338#issuecomment-605060348 that still applies.

Give me your thumbs up and will push the PR. And will probably also rework the shutdown flow too.

Definitely feel free! I think the fix you suggested https://github.com/bitcoin/bitcoin/issues/29073#issuecomment-2278066850 will prevent the crash, but it has drawbacks compared to the change suggested in https://github.com/bitcoin/bitcoin/issues/29073#issuecomment-2274237242:

  • The main drawback is that allows the UnloadWallets call to return early before all wallets have actually unloaded. This was not how UnloadWallets was intended to work and is probably not how someone would expect it to work. I think it could just lead to problems shutting down the node and gui if they try to shut down while wallets are still loaded, though I’m not sure.
  • Secondarily that fix leaves code in a more fragile state. It adds a new warning that a wallet was unable to be unloaded without any indication of what users or developers are supposed to about the warning. Ignore it? debug it? report it? And instead of making it safe to call UnloadWallet from multiple threads and simply wait for the wallet to be unloaded, it will still crash the program if called concurrently, even though on the surface it seems like it is written be a thread-safe function.

So I’d recommend using the fix in https://github.com/bitcoin/bitcoin/issues/29073#issuecomment-2274237242 or something similar, and if you want to clean this code up more I think all the ideas in https://github.com/bitcoin/bitcoin/pull/18338#issuecomment-605060348 are still applicable and would be improvements.