chia-blockchain: Slow response time with portable plots 1.2.0 for MacOS exFAT

I am running into issues where my portable plots are finding proofs (partials) but the response time is super long and they are considered invalid by the pool I joined.

Chia version 1.2.0
Mac OS Catalina 10.15.7
iMac Pro, 3.2 Ghz Xeon, 32 gb ram
The computer is connected to my router via ethernet
Plotting with a fast high tbw Sabrent NVME connected via a thunderbolt 3 external enclosure.
Farming drives are mostly Seagate external Backup Plus 8TB drives, and a Western Digital 14 tb drive. I believe these are using 5200 rpm drives inside. I have formatted them all exFat since I was planning on having a raspberry Pi 4 do farming and the file format should work on linux.
The drives are connected directly to the iMac Pro.
I’m connected to Space Pool and it says pooling. My wallet and full node are sync’d. Seeing plenty of connections to other nodes using port 8444.

I’ve started to create plots using the NFT generated when I joined Space Pool.

What’s odd is that most of the logs are fine except for when finding a partial:

2021-07-09T20:04:40.260 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 38c6cf796d... Found 0 proofs. Time: 0.01266 s. Total 348 plots
2021-07-09T20:04:49.449 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 38c6cf796d... Found 0 proofs. Time: 0.01871 s. Total 348 plots
2021-07-09T20:04:54.369 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 417995711b... Found 1 proofs. Time: 134.45386 s. Total 348 plots
2021-07-09T20:05:00.679 harvester chia.harvester.harvester: INFO     2 plots were eligible for farming 38c6cf796d... Found 0 proofs. Time: 1.67033 s. Total 348 plots
2021-07-09T20:05:09.085 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 38c6cf796d... Found 0 proofs. Time: 0.01356 s. Total 348 plots
2021-07-09T20:05:19.127 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 38c6cf796d... Found 0 proofs. Time: 0.01275 s. Total 348 plots
2021-07-09T20:05:29.085 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 38c6cf796d... Found 0 proofs. Time: 0.01329 s. Total 348 plots

These are taking around 100-200 seconds. In the case that the drives might be going to sleep, I created a cronjob to touch a dummy file every minute on the 2 drives with plots and that does not seem to make a difference.

I’m not sure what to do besides try to move to a faster storage medium or try a different format for my drives, but that seems odd. I also tried a different pool and the issue was still there.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 31 (4 by maintainers)

Links to this issue

Anyone won a proof on OS X with exFAT drives? Or exFat drives at all?

Most upvoted comments

Release 1.2.3 has introduced the parallel_read option in config.yaml to solve this issue. If you upgrade, you will need to manually add the setting under harvester.

harvester:
  parallel_read: False

emlowe on Jul 27, 2021

Great idea @AndyRPH to use the plots check function! Waiting for a proof takes a long time for most of us.

I only noticed this problem when I was playing around with some of the Chia forks, since I can find proofs much more frequently there. I didn’t see any of these extremely long times until I started running forks based on Chia 1.2.x, so I decided to compare official Chia 1.1.7 to 1.2.2 using a check on the same solo plot file.

The following tests were performed after a reboot with no other applications running:

git clone -b 1.1.7 https://github.com/Chia-Network/chia-blockchain.git
mv chia-blockchain chia-blockchain_1.1.7
cd chia-blockchain_1.1.7
sh install.sh
. ./activate
chia plots add -d /Volumes/SkyPlots/chia_plots
chia plots check -g 00af496a23ca09c

git clone -b 1.2.2 https://github.com/Chia-Network/chia-blockchain.git
mv chia-blockchain chia-blockchain_1.2.2
cd chia-blockchain_1.2.2
sh install.sh
. ./activate
chia plots add -d /Volumes/SkyPlots/chia_plots
chia plots check -g 00af496a23ca09c

Results were as follows:

Version 1.1.7: 57 seconds Version 1.2.2: 17 minutes, 58 seconds

I would rate the impact of this issue as fairly high. Yes, ExFAT has poor performance and reliability, but nonetheless it is commonly used on Macs, as it is the only natively available file system with large file support and cross-platform compatibility. Whether ExFAT, macOS, Python, Chia, or something else is at the heart of the drastic change in performance between Chia v1.1 and v1.2, the consequences for Mac Chia farmers are enormous. They will have no indication that anything is wrong until they hit a proof. If they do hit a proof, they are certain to miss the reward. If they aren’t scanning logs, they’ll never know they should have won.

wondras on Jul 16, 2021

Adding in some data: chia plots check on a solo plot, on a exfat usb drive, 10 minutes, chia plots check on a nft plot, same exfat usb drive, 8 mins. chia plots check on the same solo plot on M1 internal ssd APFS filesystem: <1 second chia plots check on the same nft plot on M1 internal ssd APFS filesystem: <1 second

Is chia plots check a good representation of how long it would take to report back a valid found solution if I got lucky? or a valid partial proof to a pool?

It seems that at least on OS X exFAT filesystems are a terrible idea for Chia. And this is unrelated to solo vs NFT plots.

AndyRPH on Jul 16, 2021

Anyone with the issue can pull main and set parallel_read to False in config.yaml for their harvesters - this will use the sequential option that was in 1.1.7.

emlowe on Jul 21, 2021

@wondras Yeah I just did my own test with Chia 1.2.2 on Big Sur 11.4. I used a 1TB WD SSDs formatted as ExFat. Copied a plot to it, added it as a chia directory, and ran the plot check (default 30 challenges). I then did the same check for a plot on one of my APFS formatted 14TB drives…

Results: APFS HDD: 19 seconds ExFat SSD: (…still running as of 30 minutes ago!)

During the ExFat test, LaunchD process began spiking at 100%. After terminating the ExFat test, LaunchD went into the same ~0% to 100% cycle as per the original report #4396 back in May, until I ejected the volume (which takes a while when in this state.)

Also my python version is 3.9.6, so this rules out any difference between Python versions being the cause.

I was going to do a test with an SSD formatted with APFS for an apples to apples comparison, but obviously there’s no point, given the above result. ExFat is still buggy as heck with Chia and/or Python on Mac.

But, I did do a plots check test with an ExFat formatted SSD connected via USB on my Ubuntu Linux plotting system…

Result: ExFat SSD on Ubuntu: 2 seconds

So clearly not something Chia or Python is doing that doesn’t jive with ExFat as a format in general. It’s something about the interaction between Chia and Python on the Mac with Apple’s ExFat implementation.

smiletracer on Jul 16, 2021

I’ve traced the issue down to the combination of Mac OS + Chia with exFat formatted drives. Reformatting to APFS solved the issue and now the proof/partial retrieval is under 2 seconds usually. This should be noted somewhere (or is there a way to speed it up via the application?). The exFat drive works just fine on my raspberry pi 4 for farming.

jonnybutler on Jul 13, 2021