circuitpython: Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance

CircuitPython version

All CP versions are affected.

Code/REPL

n/a

Behavior

Writes to the CIRCUITPY filesystem when mounted by USB MSC on MacOS Sonoma 14.4 beta 2 and above are 400 to 800 times slower than with previous releases of MacOS.

Description

Previously, MacOS Sonoma 14.0 introduced a regression in its MSDOS filesystem support that caused a number of issues that are documented in issue #8449. The root cause of the issue was in Sonoma’s new MSDOS filesystem user-space support that replaced the earlier kernel extension support. The new user-space support delays writing filesystem metadata (in this case FAT table and root directory) for tens of seconds after writes to a file, causing a variety of undesirable outcomes.

Apple subsequently fixed this behavior in 14.4 beta 2 by completely re-writing all metadata (the equivalent of a sync) after every few blocks written. This has the undesirable effects of slowing writes by 400 to 800 times and introducing unnecessary wear to the flash memory.

It is possible to reproduce poor write performance using a USB stick. All of these tests were performed with a 16G stick. The test writes 512 blocks of 512 bytes of zeroes.

dd if=/dev/zero of=/Volumes/NO\ NAME/test.dat bs=512 count=512

Performance jumps ~400x to 800x when the filesystem size exceeds 1G.

Filesystem Size Filesystem Time(Sec.)
4M FAT12 26.981
8M FAT12 28.780
15M FAT16 29.019
16M FAT16 29.139
32M FAT16 29.669
64M FAT16 29.414
128M FAT16 50.138
256M FAT32 39.641
512M FAT32 40.786
1G FAT32 42.938
2G FAT32 0.050
4G FAT32 0.046
8G FAT32 0.101
16G FAT32 0.109

For convenience I made the test sticks on Linux. It should be possible to do the same with Disk Utilities on MacOS.

Partitioned with fdisk creating a primary partition of the desired size, for example:

rabeles@ub2004:~$ sudo fdisk /dev/sdd

Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-30892031, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-30892031, default 30892031): +1026M

Created a new partition 1 of type 'Linux' and of size 1 GiB.
Partition #1 contains a vfat signature.

Do you want to remove the signature? [Y]es/[N]o: yes

The signature will be removed by a write command.

Command (m for help): t
Selected partition 1
Hex code or alias (type L to list all): 06
Changed type of partition 'Linux' to 'FAT16'.

Command (m for help): p
Disk /dev/sdd: 14.73 GiB, 15816720384 bytes, 30892032 sectors
Disk model: Cruzer Blade    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start     End Sectors Size Id Type
/dev/sdd1        2048 2103295 2101248   1G  6 FAT16

Filesystem/RAID signature on partition 1 will be wiped.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Then create the filesystem:

rabeles@ub2004:~$ sudo mkfs.vfat -F32 /dev/sdd1
mkfs.fat 4.2 (2021-01-31)

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created 5 months ago
  • Reactions: 2
  • Comments: 24

Most upvoted comments

The general public release of macOS 14.4 is out today. As expected, the slow writes are still an issue.

The good news is that it still fixes the original issue and it’s still possible to write to smaller filesystems without using any workarounds. So @dhalbert it’s safe to update the blog post to tell people to update to 14.4.

Hopefully 14.5 beta will be out soon and we can begin the testing/optimism/disappointment/surprise cycle anew.

The code attached will automatically remount a CIRCUITPY device as synchronous, likely minimizing risk of data corruption #8449 and some of the perceived performance of delayed writes. It is human readable XML, uncommented. Install once, runs unattended, persists across Mac reboots and unlimited insertions/removals on any number of devices.

This is a launch daemon property list configured to watch /Volumes for change events without polling. An embedded bash script examines all mounted CIRCUITPY devices on change. Devices mounted synchronous or read-only are not touched. Any other CIRCUITPY device is unmounted and mounted “synchronous,noasync”

To use:

  1. Download the file and rename it, removing the .txt extension.
  2. $ sudo cp tech.auli.cato.plist /Library/LaunchDaemons/tech.auli.cato.plist
  3. $ sudo launchctl load -w /Library/LaunchDaemons/tech.auli.cato.plist

Monitor insertions using Apple’s log utility:

  1. $ log stream --predicate ‘process == “logger”’

tech.auli.cato.plist.txt

Yeah, I finally updated my M2Max Studio to Sonoma today because of the 14.4 release - been on Ventura since I bought it specifically because of this CP issue.

@romkey Thanks for checking. I’m not going to hold me breath about a fix coming in 14.4 😉

The excellent workaround for #8449 by @sperok will work with Sonoma 14.3 and earlier. The reason that the workaround worked was that re-mounting the device caused the older MSDOS filesystem kext to be loaded.

Unfortunately, Apple has changed mount behavior in Sonoma 14.4 beta 4 so that it continues to use the faulty user space MSDOS filesystem driver across a re-mount.