bitcoin: Creating too many wallets exhausts file descriptor limit and leads to crash

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

This behavior was first noted by @EthanHeilman here: https://github.com/bitcoin/bitcoin/issues/11368#issuecomment-751574945 but it was conflated with the RPC issue there while I now think this is a separate issue.

When a lot of wallets are created in quick succession, sqlite will exhaust all the available file descriptors and the node crashes the next time it tries to create file for unrelated reasons (blk, rev etc.).

Expected behaviour

The node shouldn’t crash.

Steps to reproduce

for i in $(seq 1 100); do src/bitcoin-cli createwallet "fd-wallet$i"; done

Relevant log output

https://0bin.net/paste/swIJVCaU#CuCCWv7cC4ZscxjCEiauqEhB1icPiMHkYo+D5dC8mSr

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

master

Operating system and version

macOS

Machine specifications

No response

About this issue

Original URL
State: open
Created a year ago
Comments: 15 (15 by maintainers)

Most upvoted comments

Thanks, seems like the relevant short excerpt is:

2023-05-23T22:32:53Z SQLite Error. Code: 14. Message: os_unix.c:46922: (24) open(/Users/FJ/Library/Application Support/Bitcoin/testnet3/wallets/fd-wallet99/wallet.dat-journal) - Too many open files
2023-05-23T22:32:53Z SQLite Error. Code: 14. Message: statement aborts at 1: [BEGIN EXCLUSIVE TRANSACTION] unable to open database file
2023-05-23T22:32:53Z Using SQLite Version 3.39.5
2023-05-23T22:32:53Z Using wallet /Users/FJ/Library/Application Support/Bitcoin/testnet3/wallets/fd-wallet100
2023-05-23T22:32:53Z SQLite Error. Code: 14. Message: cannot open file at line 46922 of [554764a6e7]
2023-05-23T22:32:53Z SQLite Error. Code: 14. Message: os_unix.c:46922: (24) open(/Users/FJ/Library/Application Support/Bitcoin/testnet3/wallets/fd-wallet100/wallet.dat-journal) - Too many open files
2023-05-23T22:32:53Z SQLite Error. Code: 14. Message: statement aborts at 1: [BEGIN EXCLUSIVE TRANSACTION] unable to open database file
2023-05-23T22:33:38Z UpdateTip: new best=00000000607ba1efbd078e5a305b832ae14a6c444b5aa9b557db4982cca74aed height=1148054 version=0x20000000 log2_work=69.251056 tx=14198802 date='2017-06-24T09:42:34Z' progress=0.216198 cache=45.4MiB(394627txo)
2023-05-23T22:33:38Z UpdateTip: new best=00000000b7e5ec444dfcf9cff739783eb0550c692ff1c66cfe37af6374fbaad6 height=1148055 version=0x20000000 log2_work=69.251056 tx=14199082 date='2017-06-24T10:02:36Z' progress=0.216202 cache=50.5MiB(414075txo)
2023-05-23T22:33:39Z UpdateTip: new best=0000000013869d488791f4961dcf1b1f93f050b0a787616359b6355a70701e86 height=1148056 version=0x20000000 log2_work=69.251056 tx=14199702 date='2017-06-24T10:29:53Z' progress=0.216212 cache=50.8MiB(414298txo)
2023-05-23T22:33:39Z New outbound peer connected: version: 70015, blocks=2435012, peer=6 (outbound-full-relay)
2023-05-23T22:33:39Z UpdateTip: new best=0000000086afcb0260af2c4f609535db645a0f765484d389834c0309030eabcb height=1148057 version=0x20000000 log2_work=69.251056 tx=14200140 date='2017-06-24T10:52:00Z' progress=0.216219 cache=50.8MiB(414672txo)
2023-05-23T22:33:39Z UpdateTip: new best=000000009be200143cc507115d9a57504241f80536b0e9fa0a304fc9f13a7f14 height=1148058 version=0x00000004 log2_work=69.251056 tx=14200141 date='2017-06-24T11:15:47Z' progress=0.216219 cache=50.8MiB(414674txo)
2023-05-23T22:33:39Z New outbound peer connected: version: 70016, blocks=2435012, peer=5 (outbound-full-relay)
2023-05-23T22:33:39Z UpdateTip: new best=000000000e0e1b8cc908edb67e8a9329a256b162ab45a8bbfe83a9773fe32bbd height=1148059 version=0x20000000 log2_work=69.251056 tx=14200255 date='2017-06-24T11:39:07Z' progress=0.216220 cache=50.8MiB(414513txo)
2023-05-23T22:34:26Z UpdateTip: new best=000000003aae175dee028b660af7116bf8fa66ad2934845e961b9839a18821d8 height=1148060 version=0x20000000 log2_work=69.251056 tx=14200492 date='2017-06-24T12:00:07Z' progress=0.216224 cache=50.8MiB(414378txo)
2023-05-23T22:34:26Z UpdateTip: new best=00000000f7bd2dbd40619d3a20755603f010012af19f5aafbbab46ae374160db height=1148061 version=0x20000000 log2_work=69.251056 tx=14201162 date='2017-06-24T12:20:08Z' progress=0.216234 cache=50.8MiB(414715txo)
2023-05-23T22:34:26Z UpdateTip: new best=0000000003110e54d36f4113fc94f46a4463e5da18eb356e4cc0d67bb2fa79c5 height=1148062 version=0x20000000 log2_work=69.251056 tx=14201706 date='2017-06-24T12:57:58Z' progress=0.216242 cache=50.8MiB(414666txo)
2023-05-23T22:34:26Z UpdateTip: new best=000000003b71aa64a460b2f7ebe40a959ed3e9c2fe02af8df4a99fda44efe906 height=1148063 version=0x20000000 log2_work=69.251056 tx=14202041 date='2017-06-24T13:19:45Z' progress=0.216247 cache=50.8MiB(414737txo)
2023-05-23T22:34:26Z New outbound peer connected: version: 70016, blocks=2435012, peer=7 (outbound-full-relay)
2023-05-23T22:34:26Z New outbound peer connected: version: 70016, blocks=2435012, peer=8 (outbound-full-relay)
2023-05-23T22:34:26Z Synchronizing blockheaders, height: 2435012 (~100.00%)
2023-05-23T22:34:26Z UpdateTip: new best=00000000a41344a056a034e823dd4108d6186f9d61449354c29d617f5222bad3 height=1148064 version=0x20000000 log2_work=69.251056 tx=14202449 date='2017-06-24T13:56:46Z' progress=0.216254 cache=50.8MiB(414869txo)
2023-05-23T22:34:26Z UpdateTip: new best=00000000823daf27e85da59baa4c6a6b90d98283d45f425c84351592dc7655df height=1148065 version=0x20000000 log2_work=69.251056 tx=14202845 date='2017-06-24T14:20:57Z' progress=0.216260 cache=52.3MiB(428380txo)
2023-05-23T22:34:51Z Unable to open file /Users/FJ/Library/Application Support/Bitcoin/testnet3/blocks/rev00076.dat
2023-05-23T22:34:51Z ERROR: UndoWriteToDisk: OpenUndoFile failed
2023-05-23T22:34:51Z *** Failed to write undo data
2023-05-23T22:34:51Z Error: A fatal internal error occurred, see debug.log for details
2023-05-23T22:34:51Z tor: Thread interrupt
2023-05-23T22:34:51Z addcon thread exit
2023-05-23T22:34:51Z Shutdown: In progress...
2023-05-23T22:34:51Z opencon thread exit
2023-05-23T22:34:51Z torcontrol thread exit
2023-05-23T22:34:51Z ERROR: ConnectTip: ConnectBlock 00000000c101ef80a98b3e6e11d925578c02512f37731838a911591e7e64b858 failed, Failed to write undo data
2023-05-23T22:34:51Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to write undo data)
2023-05-23T22:34:51Z msghand thread exit
2023-05-23T22:34:51Z net thread exit
2023-05-23T22:34:51Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started
2023-05-23T22:34:51Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2023-05-23T22:34:52Z scheduler thread exit
2023-05-23T22:34:52Z Writing 0 unbroadcast transactions to disk.
2023-05-23T22:34:52Z Dumped mempool: 4.75e-05s to copy, 0.00576754s to dump

maflcko on Jun 14, 2023

After a deeper look, I think the crashing may be related to how macOS handles file descriptors when you try to use more than the limit.

It’s not how macOS handles it, any OS will cause bitcoind to crash if you use more fd than the limit. It’s just that among all the OS macOS has the lowest standard limit and so such issue are usually observed there first.

Bitcoin Core loads the wallet immediately after you create it and it keeps that file descriptor open for easy access. You can try running bitcoin-cli listwallets and you should see all of the wallets created up until the point where you go over the file descriptor limit.

Yes. But this doesn’t change that bitcoind can crash and ideally it shouldn’t, which is why I opened this issue.

I’m able to trigger this on macOS with a slightly bigger for loop (the default fd limit for macOS is 256, so anything around that ballpark should trigger). However, looks like this is not making bitcoind crash anymore, at least not for me.

[...]
2023-06-22T14:38:41Z [rpc] ThreadRPCServer method=createwallet user=user
2023-06-22T14:38:41Z Using SQLite Version 3.39.5
2023-06-22T14:38:41Z Using wallet /Users/sergi/Library/Application Support/Bitcoin/regtest/wallets/fd-wallet132
2023-06-22T14:38:41Z SQLite Error. Code: 14. Message: cannot open file at line 46922 of [554764a6e7]
2023-06-22T14:38:41Z SQLite Error. Code: 14. Message: os_unix.c:46922: (24) open(/Users/sergi/.bitcoin/regtest/wallets/fd-wallet132/wallet.dat-journal) - Too many open files
2023-06-22T14:38:41Z SQLite Error. Code: 14. Message: statement aborts at 1: [BEGIN EXCLUSIVE TRANSACTION] unable to open database file
2023-06-22T14:38:41Z [http] Received a POST request for / from 127.0.0.1:49976
2023-06-22T14:38:41Z [rpc] ThreadRPCServer method=createwallet user=user
2023-06-22T14:38:41Z Using SQLite Version 3.39.5
2023-06-22T14:38:41Z Using wallet /Users/sergi/Library/Application Support/Bitcoin/regtest/wallets/fd-wallet133
2023-06-22T14:38:41Z SQLite Error. Code: 14. Message: cannot open file at line 46922 of [554764a6e7]
2023-06-22T14:38:41Z SQLite Error. Code: 14. Message: os_unix.c:46922: (24) open(/Users/sergi/.bitcoin/regtest/wallets/fd-wallet133/wallet.dat-journal) - Too many open files
2023-06-22T14:38:41Z SQLite Error. Code: 14. Message: statement aborts at 1: [BEGIN EXCLUSIVE TRANSACTION] unable to open database file
2023-06-22T14:38:41Z [http] Received a POST request for / from 127.0.0.1:49977
[...]

sr-gi on Jun 22, 2023

After a deeper look, I think the crashing may be related to how macOS handles file descriptors when you try to use more than the limit.

It’s not how macOS handles it, any OS will cause bitcoind to crash if you use more fd than the limit. It’s just that among all the OS macOS has the lowest standard limit and so such issue are usually observed there first.

Bitcoin Core loads the wallet immediately after you create it and it keeps that file descriptor open for easy access. You can try running bitcoin-cli listwallets and you should see all of the wallets created up until the point where you go over the file descriptor limit.

Yes. But this doesn’t change that bitcoind can crash and ideally it shouldn’t, which is why I opened this issue.

fjahr on Jun 18, 2023

Unfortunately I was not able to reproduce this after multiple tests. Do you maybe have a more specific scenario where the issue would occur?

I don’t really have any more specifics to give. This was the latest master at the time, compiled only with standard flags and no other wallets present etc. I reproduced the issue 4-5 times consistently with script given above. What OS are you on? If you are not on macOS maybe you have to lower the limit of allowed file descriptors.

Okay, I was able to reproduce it when lowering the file descriptors and will look further into it. Interesting enough, in my case it doesn’t crash the node but the file descriptors seem to be locked, and can’t load the already created wallets.

2023-06-14T19:22:21Z [fd-wallet1] Releasing wallet
2023-06-14T19:22:22Z Using SQLite Version 3.40.1
2023-06-14T19:22:22Z Using wallet /home/dimitaracev/.bitcoin/regtest/wallets/fd-wallet1
2023-06-14T19:22:22Z init message: Loading wallet…
2023-06-14T19:22:22Z [fd-wallet1] Wallet file version = 10500, last client version = 259900
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = 5e306353541637e7a8a5544e5b257ba8f1d3c488f2e9fd7cbf8b050742e3ec8d, type = legacy, internal = false
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = 46059f1ecfbe2666420840110eb03da53c829a2a1296bb63c64d5daf562d070b, type = p2sh-segwit, internal = false
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = e60db0e290b20b41cccd1c62b47120159af0e747b5829eb611118c6f053df157, type = bech32, internal = false
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = f706629e5c33dfe527c2f5ec6e2c0f0d36f3cc75f6e56b711554198e9120d2ca, type = bech32m, internal = false
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = 6c17557bdb8f060982bd9478d20c33989fec840f5fca6104f5c9b77ae2000425, type = legacy, internal = true
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = 6120ab62678492d7f653409175de0ac0bfbab67f7501b2387533477c8a14ee17, type = p2sh-segwit, internal = true
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = c9bb633eaf2982beb1711e977e8e679374160a898cf392a886f81ac41829c5e7, type = bech32, internal = true
2023-06-14T19:22:22Z [fd-wallet1] Setting spkMan to active: id = 15400e0ed8a94b5e6b245b991e78dc2f467873cea42ee508d4a3563bdb6456f1, type = bech32m, internal = true
2023-06-14T19:22:22Z [fd-wallet1] Keys: 8 plaintext, 0 encrypted, 0 w/ metadata, 8 total. Unknown wallet records: 0
2023-06-14T19:22:22Z SQLite Error. Code: 14. Message: cannot open file at line 42907 of [df5c253c0b]
2023-06-14T19:22:22Z SQLite Error. Code: 14. Message: os_unix.c:42907: (24) open(/home/dimitaracev/.bitcoin/regtest/wallets/fd-wallet1/wallet.dat-journal) - 
2023-06-14T19:22:22Z SQLite Error. Code: 14. Message: statement aborts at 14: [INSERT or REPLACE into main values(?, ?)] unable to open database file
2023-06-14T19:22:22Z WriteKey: Unable to execute statement: unable to open database file
2023-06-14T19:22:22Z [fd-wallet1] Releasing wallet
2023-06-14T19:27:36Z CreateNewBlock(): block weight: 892 txs: 0 fees: 0 sigops 400
2023-06-14T19:27:36Z Saw new header hash=1786ef9d4d7625c83d48e6e706c2d2cd53cb73f2023ad369aa9200a5269bf87f height=405
2023-06-14T19:27:36Z UpdateTip: new best=1786ef9d4d7625c83d48e6e706c2d2cd53cb73f2023ad369aa9200a5269bf87f height=405 version=0x30000000 log2_work=9.665336 tx=406 date='2023-06-14T19:27:36Z' progress=1.000000 cache=0.3MiB(304txo)

dimitaracev on Jun 14, 2023

Unfortunately I was not able to reproduce this after multiple tests. Do you maybe have a more specific scenario where the issue would occur?

I don’t really have any more specifics to give. This was the latest master at the time, compiled only with standard flags and no other wallets present etc. I reproduced the issue 4-5 times consistently with script given above. What OS are you on? If you are not on macOS maybe you have to lower the limit of allowed file descriptors.

fjahr on Jun 14, 2023

I can take this if still available.

Feel free to investigate, i am not working on this currently.

fjahr on Jun 13, 2023