bitcoin: Blocks directory on macOS uses more disk space than expected

Checking the file content

The file rev00337.dat has a reported size by du of 212MB before copying and 21MB after. However, the content of the files is exactly the same.

cd ~/Library/Application\ Support/Bitcoin/blocks

du -ch ./rev00337.dat
 212M	./rev00337.dat
 212M	total

hexdump -C rev00337.dat | tail -4
0141a770  e0 92 b0 c3 00 00 00 00  00 00 00 00 00 00 00 00  |................|
0141a780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01500000

cp rev00337.dat ~/Desktop

cd ~/Desktop

hexdump -C rev00337.dat | tail -4
0141a770  e0 92 b0 c3 00 00 00 00  00 00 00 00 00 00 00 00  |................|
0141a780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01500000

Checking that the files content is the same after copy

I have modified the previously posted script to show the files hashes before and after copy.

for file in ./rev*.dat
do
    if [[ -f $file ]]; then
	size_before=$(du -h $file | cut -f1)
	hash_before=$(shasum -a 256 $file | cut -d" " -f1)
        cp $file $file.tmp
        rm $file
	mv $file.tmp $file
	size_after=$(du -h $file | cut -f1)
        hash_after=$(shasum -a 256 $file | cut -d" " -f1)
	echo $file $size_before " -> " $size_after " hash_before: " $hash_before "hash_after: " $hash_after
    fi
done

Here is a sample output:

./rev01590.dat 3.9M  ->  4.0M  hash_before:  9fa0f20cfc1d9593b873399130cfce323aa574473f3a6b7caa58a970e41dc45b hash_after:  9fa0f20cfc1d9593b873399130cfce323aa574473f3a6b7caa58a970e41dc45b
./rev01591.dat 844K  ->  1.0M  hash_before:  ab8811a65806921492c63c85c3d05643a0a7cba9fb5b3a677cb16fabb1167918 hash_after:  ab8811a65806921492c63c85c3d05643a0a7cba9fb5b3a677cb16fabb1167918

All the hashes are the same before and after copying the files.

Difference in file size reporting

The file size reported by du is different from the one reported by ls and stat.

cd ~/Library/Application\ Support/Bitcoin/blocks

du -ch ./rev01668.dat
 90M	./rev01668.dat
 90M	total

ls -lah | grep rev01668.dat 
-rw-------     1 something  something    17M Jan  6 11:18 rev01668.dat

stat -x rev01668.dat          
  File: "rev01668.dat"
  Size: 17825792     FileType: Regular File
  Mode: (0600/-rw-------)         Uid: (  502/  something)  Gid: (   20/   something)
Device: 1,4   Inode: 65189970    Links: 1
Access: Mon Jan  6 11:53:20 2020
Modify: Mon Jan  6 11:18:58 2020
Change: Mon Jan  6 11:18:58 2020

Diagnosis

du is reporting the size in used blocks, ls the size of the files content.

diskutil info / | grep "Block Size"
   Device Block Size:         4096 Bytes
   Allocation Block Size:     4096 Bytes

cd ~/Library/Application\ Support/Bitcoin/blocks

du -ch ./rev01668.dat
 90M	./rev01668.dat
 90M	total

BLOCKSIZE=4096 du -c ./rev01668.dat
22916	./rev01668.dat
22916	total

BLOCKSIZE=4096 ls -lahs | grep rev01668.dat
 22916 -rw-------     1 something  something    17M Jan  6 11:18 rev01668.dat

cp rev01668.dat ~/Desktop

cd ~/Desktop

du -ch ./rev01668.dat
 17M	./rev01668.dat
 17M	total

BLOCKSIZE=4096 du -c ./rev01668.dat
4352	./rev01668.dat
4352	total

BLOCKSIZE=4096 ls -lahs | grep rev01668.dat
  4352 -rw-------   1 something  something    17M Jan  6 12:08 rev01668.dat

It can be seen that, after copying the file, the number of blocks used is reduced drastically and the file size is as expected.

Hypothesis and troubleshooting

ftruncate function is not working as expected when used in an APFS formated volume.

I have tried to reproduce the problem in an isolated environment wirhout any success. The files sizes were always correct.

Possible fix

If the function AllocateFileRange is modified to use the fallback method in macOS the file sizes are as expected and the problem dissapear.

As a workaround based on a tip from the IRC discussions, I did a small bash script to slim down the rev files one by one, by copying them once, allowing me to continue from 83% and finish the download on the mac mini. These are the steps I took, below reproduced on my second machine that was on 23% when I stopped downloading, hence the total size is lower, but you can see there is already an extra 40G of space being consumed:

Thanks for the workaround of copying them out. I was able to reduce disk usage by 237G. My before & after:

$ du -h
 92M	./index
538G	.

$ du -h
 92M	./index
301G	

I kinda need to retract what I said before. Today I woke up to a bitcoin core message “Error: Disk space is too low!”. This was at 91.20% of IBD. I checked the drive and About this Mac, Storage says only 80MB is left on the 500GB dedicated partition. Du and df commands and disk utility report the whole partition space used. Finder also reports whole disk space used when checking the whole drive but when checking the bitcoin folder only it says it takes only 273GB (there is really nothing else on this partition). Upon investigating further, I can see that Get Info option on a folder returns: 273 165 546 724 bytes (499,43 GB on disk) for 5 369 items similarly, all rev files in the blocks folder are visible in Finder as around 19MB files but when using Get info option the true size is revealed, e.g. 19 922 944 bytes (199,3 MB on disk) Same goes for ncdu, it reports true size of files on the disk and it seems that a lot of rev files actually take much more space than they should (e.g. 200MB).

I checked the debug.log and noticed that the rev files that are the biggest have the most Pre-allocating up to position, messages, and the smallest ones have the least:

• rev01037, 350.2 MiB, pre-allocating goes up to 0x1a00000
• rev01767, 8.6MiB, pre-allocating goes up to 0x500000 only

FYI, the Time Machine is disabled for this drive, later I disabled it completely, restarted the machine, this did not free up any space.

I guess that some dev needs to investigate closer how this pre-allocating mechanism behaves on APFS drives. Perhaps it reserves more space than it needs and later on it does not release it properly?

No, this is not intended behavior at all. The rev*.dat files shouldn’t go above 15-20 MB usually.

I currently have a fully synced node running on macOS (10.14) that uses nearly 500GB on disk:

475G	total
422G	blocks
 40G	indexes
8.4G	testnet3
4.1G	chainstate
356M	regtest
9.8M	debug.log
4.0M	wallet.dat
4.0M	peers.dat
4.0M	mempool.dat
896K	some_wallet
256K	fee_estimates.dat
4.0K	bitcoin.conf
4.0K	banlist.dat
  0B	db.log
  0B	.walletlock
  0B	.lock

Largest rev*.dat files:

206M	rev00352.dat
193M	rev00331.dat
190M	rev00461.dat
190M	rev00447.dat
190M	rev00390.dat
190M	rev00389.dat
190M	rev00386.dat
190M	rev00374.dat