circuitpython: File system problems and soft reboot hanging

I was using a variant of CircuitPython 4.0.0 beta 3 (4.0.0-beta.3-25-g2169a6240-dirty) which initially looked good and then started seriously misbehaving with a lot of things disappearing from the file system and it failing to soft reboot, i.e. control-D from the REPL over serial over USB.

The first thing I noticed from a Win8.1 host was that the application on the Circuit Playground Express board I was using (it reads MIDI over USB and changes PWMOut properties) which had worked earlier no longer worked. I then wanted to check its health so tried to connect over COM9 but surprisingly that would not let me connect. I thought I’d check the CIRCUITPY drive associated with it and that was misbehaving too.

The following is from my notes which I initially put into comments in #1626.

The drive (on Windows) was no longer showing up with the typical spaced used but just marked as FAT and opening it in explorer showed it as empty. It responded to reset button but that’s left me with an CIRCUITPY drive that’s missing my code! I’ve only got some wavs, a README, .fseventsd and .Trashes. I did a power cycle and file system still looks the same, no sign of my code.py or massive lib directory etc.

I think we can rule out user error here because the whole lib directory has gone AWOL and that takes ages to delete so I would have noticed if I’d done this accidentally.

After that the wavs disappeared and a .metadata_never_index has appeared. (Minor chance this is related to Patch Tuesday but that’s even more frightening.)

My boot_out currently has the contents:

Adafruit CircuitPython 4.0.0-beta.3-25-g2169a6240-dirty on 2019-03-12; Adafruit CircuitPlayground Express with samd21g18

but earlier it had a large hex number with some hypens with braces on either end.

I’ve also noted that if i leave the CPX doing not much in REPL (I had a for loop printing the time every 10 seconds) then if I control-c that ten minute later then control-D for a soft reset the soft reset doesn’t complete.

On going back to 4.0.0 beta 3, this popped out on the serial console (over USB):

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.

You are running in safe mode which means something unanticipated happened.
Looks like our core CircuitPython code crashed hard. Whoops!
Please file an issue at https://github.com/adafruit/circuitpython/issues
 with the contents of your CIRCUITPY drive and this message:
Crash into the HardFault_Handler.

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 4.0.0-beta.3 on 2019-03-07; Adafruit CircuitPlayground Express with samd21g18
>>>

boot_out currently contains:

Adafruit CircuitPython 4.0.0-beta.3-25-g2169a6240-dirty on 2019-03-12; Adafruit CircuitPlayground Express with samd21g18

and has a date of 1-Jan-2000 which is a bit odd (date may not be off, these things dont have battery backed clocks on them, of course). I’m going to storage.erase_filesystem() to ensure I’ve got a clean slate.