conan: Windows: non-latin symbols in USERPROFILE or PATH break conan

The following script in 866 code page does nothing, but must install cmake files to generate the project:

call git clone https://github.com/memsharded/example-poco-timer.git 
cd example-poco-timer
set CONAN_USER_HOME=
set USERNAME=Профиль
set USERPROFILE=F:\Профиль
mkdir build 
cd build
call conan install ..

Double check the output, as you see conan says and does nothing:

F:\Work\conan>call git clone https://github.com/memsharded/example-poco-timer.git
Cloning into 'example-poco-timer'...
remote: Counting objects: 40, done.
remote: Total 40 (delta 0), reused 0 (delta 0), pack-reused 40
Unpacking objects: 100% (40/40), done.

F:\Work\conan>cd example-poco-timer

F:\Work\conan\example-poco-timer>set CONAN_USER_HOME=

F:\Work\conan\example-poco-timer>set USERNAME=Профиль

F:\Work\conan\example-poco-timer>set USERPROFILE=F:\Профиль

F:\Work\conan\example-poco-timer>mkdir build

F:\Work\conan\example-poco-timer>cd build

F:\Work\conan\example-poco-timer\build>call conan install ..

F:\Work\conan\example-poco-timer\build>dir
 Том в устройстве F имеет метку WORK
 Серийный номер тома: 4E18-2D78

 Содержимое папки F:\Work\conan\example-poco-timer\build

16.12.2017  18:02    <DIR>          .
16.12.2017  18:02    <DIR>          ..
               0 файлов              0 байт
               2 папок  29 097 922 560 байт свободно

If I assign to the USERPROFILE environment variable a latin string or set CONAN_USER_HOME to latin path everything is fine.

Also I observe a strange error when PATH contains non-latine symbols:

git clone https://github.com/anatoly-spb/conan-libqrencode
cd conan-libqrencode
set PATH=F:\Профиль;%PATH%
conan create bincrafters/stable

failed with

libqrencode/4.0.0@bincrafters/stable: Package '9b1a62505a16cd151a66fc2504680042cb2cf4f2' built
libqrencode/4.0.0@bincrafters/stable: Build folder F:\Conan\build\.conan\data\libqrencode\4.0.0\bincrafters\stable\build\9b1a62505a16cd151a66fc2504680042cb2
cf4f2
libqrencode/4.0.0@bincrafters/stable: Generated conaninfo.txt
libqrencode/4.0.0@bincrafters/stable: Generated conanbuildinfo.txt
libqrencode/4.0.0@bincrafters/stable: Generating the package
libqrencode/4.0.0@bincrafters/stable: Package folder F:\Conan\build\.conan\data\libqrencode\4.0.0\bincrafters\stable\package\9b1a62505a16cd151a66fc250468004
2cb2cf4f2
libqrencode/4.0.0@bincrafters/stable: Calling package()
libqrencode/4.0.0@bincrafters/stable package(): Copied 1 '.h' files: qrencode.h
libqrencode/4.0.0@bincrafters/stable package(): Copied 1 '.lib' files: qrencode.lib
libqrencode/4.0.0@bincrafters/stable: Package '9b1a62505a16cd151a66fc2504680042cb2cf4f2' created
libqrencode/4.0.0@bincrafters/stable test package: Generator cmake created conanbuildinfo.cmake
libqrencode/4.0.0@bincrafters/stable test package: Generator txt created conanbuildinfo.txt
libqrencode/4.0.0@bincrafters/stable test package: Generated conaninfo.txt
ERROR: Unable to build it successfully
  File "conan\conans\client\tools\env.py", line 34, in environment_append
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd2 in position 1792: ordinal not in range(128)

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 2
Comments: 26 (21 by maintainers)

Commits related to this issue

Add support for a unicode conan data directory. Tested so far with `conan download` on Py2.7. A unicode compatibility module has been started to contain elements that need to differ between Py2 and Py... — committed to mpdelbuono/conan by mpdelbuono 7 years ago
Add support for a unicode conan data directory. Tested so far with `conan download` on Py2.7. A unicode compatibility module has been started to contain elements that need to differ between Py2 and Py... — committed to mpdelbuono/conan by mpdelbuono 7 years ago
Add support for a unicode conan data directory. Supports including unicode characters within the conan.conf file. A unicode compatibility module has been started to contain elements that need to diffe... — committed to mpdelbuono/conan by mpdelbuono 7 years ago

Most upvoted comments

I have managed to solve this, now hopefully fully solved, in https://github.com/conan-io/conan/pull/12890, it will be released in next 1.58 (you can test it now running Conan from source develop branch), closing the ticket, thanks again for the feedback.

memsharded on Jan 17, 2023

My two cents in this, is if it’s not feasible to solve. A compile time error that states the issue that the path contains UTF-8 encoded characters, and that you have to relocate your Conan home folder will save lots of hair tearing for users that don’t think about that their username contains UTF-8 characters (my was created by Windows by azure-AD domain, so didn’t think about it)

I wish it was that simple, but that is also a bit risky. First, it might not fail depending on the build system integration (for example, cmake with Visual Studio generator, will not fail, because it doesn’t need the environment). It also depends on the system, if the user has chcp 65001 configured in their terminal, it might even work fine, so raising an error for these users will be a blocker and very annoying. I know it is not very likely, because that is not the default, but sooner or later with the Conan scale and so many thousands of users, it will also block some users.

We will keep exploring possibilities for the next release, lets see what we can improve here, thanks very much for the suggestion!

memsharded on Jan 12, 2023

The path to deprecate python2 has been defined (https://blog.conan.io/2018/08/13/Its-Time-To-Deprecate-Python-2.html) and many efforts towards handling unicode in a compatible way for py2 and py3 have been done without a clear success (thanks to all!! 🙌), so I agree with the “if you want unicode support, use Python 3”, don’t you?

Said that, this issue should be retargeted to an up-to-date version of Conan running in python 3, is it still an issue, @anatoly-spb?

jgsogo on Sep 12, 2018