platformio-core: Problem with Python 3.8+ on Windows when network drive is used

The cause of the issue: https://bugs.python.org/issue37993

Solution 1

Open Windows Registry regedit.exe
Create a new key named Command Processor at HKEY_CURRENT_USER\Software\Microsoft
Create a new DWORD (32-bit) Value named DisableUNCCheck
Set the value of DisableUNCCheck to 1 (Hex)
Restart your Windows machine.

Solution 2

Download our portable Python 3.7
Extract and replace contents in %HOMEPATH%/.platformio/python3.
Remove %HOMEPATH%/.platformio/penv folder.
Restart VSCode.

Configuration

Operating system: Windows_NT x64 10.0.17763

PlatformIO Version (platformio --version): 4.2.2b1 and 4.2.1

Python Version 3.8 32bit

Description of problem

During compilation I get the Python Traceback UnicodeDecodeError: ‘charmap’ codec can’t decode byte.

Full Message:

> Executing task in folder IR_Sender_WB: C:\Users\***\.platformio\penv\Scripts\platformio.exe run <

Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
- framework-arduinoespressif8266 3.20603.200130 (2.6.3)
- tool-esptool 1.413.0 (4.13)
- tool-esptoolpy 1.20800.0 (2.8.0)
- toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 31 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4
Building in release mode
Compiling .pio\build\nodemcuv2\src\main.cpp.o
Generating LD script .pio\build\nodemcuv2\ld\local.eagle.app.v6.common.ld
Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
    self.do_reading()
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 167: character maps to <undefined>
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRac.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRrecv.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRsend.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRtext.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRtimer.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRutils.cpp.o
======================================================================================== [FAILED] Took 5.42 seconds ========================================================================================
The terminal process terminated with exit code: 1

Terminal will be reused by tasks, press any key to close it.

If problems with PlatformIO Build System:

The content of platformio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
monitor_speed = 115200

Additional info

On another PC with Python version 3.7 there are no problems compiling. Here are the data of the PC where there is no problem:

Version: 1.43.0 (user setup) Commit: 78a4c91400152c0f27ba4d363eb56d2835f9903a Date: 2020-03-09T19:47:57.235Z Electron: 7.1.11 Chrome: 78.0.3904.130 Node.js: 12.8.1 V8: 7.8.279.23-electron.0 OS: Windows_NT x64 10.0.17763

PlatformIO: Core 4.2.1, Home 3.1.1

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 75 (26 by maintainers)

Commits related to this issue

Force PIPE reader to UTF-8 on Windows // Issue #3417 — committed to platformio/platformio-core by ivankravets 4 years ago
Print warning about an issue with mapped network drives on Windows // Issue #3417 Starting with Python 3.8 paths to mapped network drives are resolved to their real path in the system, e.g.: "Z:\path... — committed to platformio/platformio-core by valeros 4 years ago
Provide solution for issue #3417 — committed to platformio/platformio-core by ivankravets 3 years ago
Improved support for projects located on Windows network drives // Issue #3417 — committed to platformio/platformio-core by ivankravets a year ago
Revert "Improved support for projects located on Windows network drives // Issue #3417" This reverts commit a0c959be2877d36ba188ef5eb05329446061bbdb. — committed to platformio/platformio-core by ivankravets a year ago

Most upvoted comments

After long fiddeling around with different python versions i give up… with older python verisons the PIO home screen will not work!! Looking further in the problem that cmd.exe is not handling UNC names i found a very old KB

https://web.archive.org/web/20150518102450/https://support.microsoft.com/en-us/kb/156276

It was written for windows XP and NT but never the less its WORKING! under win10! HKEY_CURRENT_USER\Software\Microsoft\Command Processor <— create this key “Command Processor” add the value “DisableUNCCheck” REG_DWORD and set the value to 0 x 1 (Hex).

@kbfifi , please let me know if its working for you as well

Rotma on Mar 27, 2021

Neither fix worked for me. The first one made it end up making the path within the mapped network drive, Seems it thinks that the space in the path that is mapped to the drive letter is a seperator because everything after that was created in the projects folder and had some things put into it as it was failing.

the second one seemed to have all my work undone by downloading python over what I replaced it with when I pressed build.

Begrudgingly moved my project onto the local drive just so I could build it. This is a showstopper for the use of platformio for me as I rely on my files being on a nas so they are accessible on all computers

richms on Apr 1, 2022

@ivankravets Yeah, I’ve done that which helped I think, but I still got compiler errors. And, I got the “There is a known issue with Python 3.8+” warning despite not having Python 3.8+ installed. Here are the relevant logs:

Warning! `env_default` configuration option in section [platformio] is deprecated and will be removed in the next release! Please use `default_envs` instead
Processing melzi (platform: atmelavr; framework: arduino; board: sanguino_atmega1284p)
------------------------------------------------------------------------------------------------------------------------
There is a known issue with Python 3.8+ and mapped network drives on Windows.
Please downgrade Python to the latest 3.7. More details at:
https://github.com/platformio/platformio-core/issues/3417
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/sanguino_atmega1284p.html
PLATFORM: Atmel AVR (3.3.0) > Sanguino ATmega1284p (16MHz)
HARDWARE: ATMEGA1284P 16MHz, 16KB RAM, 127KB Flash
DEBUG: Current (simavr) On-board (simavr)
PACKAGES:
 - framework-arduino-avr 5.1.0
 - toolchain-atmelavr 1.70300.191015 (7.3.0)
Converting Marlin.ino
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
FileNotFoundError: [WinError 161] The specified path is invalid: '\\\\networkdrive\\code\\Marlin\\https:\\github.com\\MarlinFirmware\\U8glib-HAL\\archive\\dev.zip':
  File "C:\Users\User\.platformio\penv\Lib\site-packages\platformio\builder\main.py", line 177:
    env.SConscript("$BUILD_SCRIPT")
    ...

I ran into the same problems when compiling marlin from a network drive:

[WinError 161] The specified path is invalid

The solution that worked for me was:

Install python 3.7.9
Delete the content of the .platformio\python3 folder
Copy the content from AppData\Local\Programs\Python\Python37 into .platformio\python3

Marlin compiled properly with no errors. Hope this helps, its a very down and dirty solution but it works.

Bear in mind if you do anything to trigger a reinstall of the platformio extension, you will need to redo the steps above. It clearly proves the problem is with python.

VB5000 on May 6, 2021

@Rotma thanks for finding that! Adding that registry key works great for me, as well. @ivankravets thanks for updating first post and the warning message in PIO.

claudobahn on Mar 31, 2021

did a quick test, seems to work, thanks!

stisa on Jan 21, 2021

Please re-test with the latest development version of PIO Core

pio upgrade --dev

ivankravets on Jan 20, 2021

i had this issue on windows (local disk). I solved this by changing the following line:

https://github.com/platformio/platformio-core/blob/develop/platformio/proc.py#L33

from :

self._pipe_reader = os.fdopen(self._fd_read)

to :

self._pipe_reader = os.fdopen(self._fd_read, encoding='utf-8')

there are a couple of stackoverflow on this.

draeron on Mar 21, 2020

Hello, all Python versions were downloaded from the official website. (https://www.python.org/downloads/)

What did I do:

uninstalled Python 3.8 and installed the last release with version 3.8.2 and deleted the penv folder and restarted VS Code Result: Problem still exists
platformIO completely deleted from VS Code and reinstalled (Version 4.2.1) Result Problem still exists
uninstall Python 3.8.2 and install Python 3.7.3, so that the versions are exactly the same as the working PC. delete the penv folder and restart VS Code. Result: Compilation works!

Conclusion: Python 3.8 causes the error posted above In Python 3.7 these problems do not yet exist

Translated with www.DeepL.com/Translator (free version)

Hofei90 on Mar 14, 2020