circuitpython: Delayed writes and auto-reset problems when writing from Windows

I have had problems writing to the onboard filesystem from Windows. I sometimes see complete filesystem corruption, and sometimes just problems with one file. See https://forums.adafruit.com/viewtopic.php?f=60&t=109687 for background.

Specific scenario, showing a file that CPy has trouble reading:

The file in question is here: main.py.txt. (Renamed from main.py to main.py.txt so that GitHub will take it as an attachment.) I wrote this Python code while doing some CPy I/O testing, and the specific code probably doesn’t have anthing to do with this problem. However, this file is 556 bytes long, so it’s more than one 512-byte block, which does seem to be important.

If I write this file to D:\CIRCUITPY\main.py using NOTEPAD.EXE, it runs just fine. It prints “5” and the button-reading loop works.

If I write this exact same file using Notepad++ (a very common lightweight editor used on Windows) it does not work. The serial port shows:

Auto-soft reset is on. Simply save files over USB to run them.
main.py output:
Traceback (most recent call last):
  File "main.py", line 26
SyntaxError: invalid syntax

Line 26 is right around the 512-byte boundary in the file.

In the REPL, I can read the file I wrote with NOTEPAD.EXE:

>>> import uos
>>> uos.listdir()
['System Volume Information', 'main.py']
>>> f = open('main.py', 'rb')
>>> chars = f.read()
>>> chars
b'import nativeio as io\r\nfrom board import *\r\nimport time\r\n\r\nled = io.DigitalInOut(D13)\r\nled.switch_to_output()\r\n\r\nswitches = [io.DigitalInOut(pin) for pin in (D6, D10, D11, D12)]\r\nfor switch in switches:\r\n    switch.switch_to_input(pull=io.DigitalInOut.Pull.UP)\r\n\r\n\r\ndef blink(n, interval=0.2):\r\n    for i in range(n):\r\n        led.value = 1\r\n        time.sleep(interval)\r\n        led.value = 0\r\n        time.sleep(interval)\r\n\r\nprint(2+3)\r\n\r\n\r\nwhile(True):\r\n    for i, switch in enumerate(switches):\r\n        if not switch.value:\r\n            blink(i+1)\r\n\r\npass\r\n'
>>> f.close()

But if I write the same file again with Notepad++, I get an OSError when trying to read it in the REPL:

>>> f = open('main.py', 'rb')
>>> chars = f.read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: 5

I can TYPE either file from CMD.EXE, and if I use od (from GnuWin32) to look at the characters in the file, they are identical.

I have repeated the cycle of writing with NOTEPAD.EXE and then Notepad++ and I consistently have the error only with Notepad++. Once or twice TYPE complained it did not have access to the bad version of the file, but I cannot reproduce that problem consistently. If I look at the file properties in Windows Explorer, the two versions have identical properties and sizes.

I’ve looked at the Notepad++ source code where it writes files. It looks innocuous: it uses ::fwrite(). It does support UTF-8, but the file in question is all ASCII, and I set up Notepad++ to write it as ANSI.

This appears to be some oddity or corruption about how Windows is writing to the CPy filesystem. I’ve inquired on the Notepad++ forum about whether it does anything unusual when writing files, and will report back if I hear anything.

The workaround is not to use Notepad++, but it seems important to figure out what’s going wrong so other users will not have the same problem. I did some websearching and haven’t turned up any similar reports about Notepad++, FatFs, or MicroPython.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 1
  • Comments: 45 (14 by maintainers)

Most upvoted comments

Hi, thank you for your perfect analysis, I have the same problem in a different context, and I spent days to find a solution. I’m not sure to find one now, but I know what to try. If my case can help to motivate MS to fix this trouble: we use a custom ioT, which has 1MB of flash with USB port. It is used to configurate it, and to copy some data. People tend to forget to eject properly USB mass storage (because it is synced now), and it is difficult to explain that the filesystem will be corrupted, because it is smaller than 16MB… So, I agree that floppy disks don’t exist anymore, but ioT is a new case MS should consider.

@tannewt I didn’t expect anything to actually happen here but it turns out it might be fixed: https://twitter.com/zooba/status/1188954487924260864

I’m going to close this issue for now, since we’ve documented the problem thoroughly in the Learn Guides and described and implemented mitigations such as editor plugins that force immediate writes. If/when there’s some movement on the Windows side or we figure out an alternate filesystem that works on all the platforms, I’ll reopen or create a new issue.

I found this thread on how to unmount a disk using windows:

https://support.microsoft.com/en-us/help/165721/how-to-ejecting-removable-media-in-windows-nt-windows-2000-windows-xp

The issues seems to be solved with OpenMV IDE using this to reset the OpenMV Cam.

And on linux the syncfs() function can be used.

@dhalbert and me have report it in Windows insiders feedback hub and if it get more votes it get fixed faster

Uwe Sieber has some nice command-line friendly tools that we can integrate if necessary http://www.uwe-sieber.de/drivetools_e.html in particular “Eject Media”

I’d go with @ladyada 's second option. It shouldn’t be too hard to only autoreset after writes to a specific block or two. I don’t think I’d have it on by default though. I’d just have it as a setting you can set in boot.py like turning off autoreset.