WSL: Unknown encoding: 65001

I am attempting to run docker-compose.exe from Windows in BashOnWindows. Specifically I am running docker-compose.exe --version to verify that it can correctly output data. When doing this I am receiving the following error:

~ # docker-compose.exe --version                                                                                                                                                                                                  
Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose\cli\main.py", line 66, in main
  File "compose\cli\main.py", line 98, in dispatch
  File "compose\cli\docopt_command.py", line 25, in parse
  File "compose\cli\docopt_command.py", line 12, in docopt_full_help
  File "site-packages\docopt.py", line 575, in docopt
  File "site-packages\docopt.py", line 484, in extras
LookupError: unknown encoding: cp65001
Failed to execute script docker-compose
------------------------------------------------------------

I can see that codepage 65001 is the Windows codepage for UTF-8 encoding, but that this is not supported in Unix systems. A solution to this is to run the command chcp.com 437 to change to a different codepage. This solution is however not good as it removes the support for UTF-8 in the terminal. The issue should be persistent across different Windows applications that attempt to output any text in encoding 65001.

Windows build number: Microsoft Windows [Version 10.0.15063]

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Reactions: 11
  • Comments: 25

Most upvoted comments

Nevermind, its bash so I just made a function to pipe the output to cat. Turns out as long as stdout and stderr go to something else before they go to the console, python doesn’t complain.

docker-compose() {
  command docker-compose.exe "$@" 2>&1 | cat
}

This issue also causes display issues with Windows programs run from within bash.

See missing/incorrect umlauts below:

image

My codepage is 850, but the inner cmd.exe’s (ran from bash) is 65001.

Running chcp 850 in the inner cmd fixes it.

What is strange though is the following, why is it like that?

  • Open cmd
    • Run chcp - output: 850
    • Run bash
      • Run cmd.exe /c chcp - output: 65001 (expected because of this issue, albeit not good)
      • Run cmd.exe
        • Run chcp - output: 65001 (expected)
        • Run chcp 437
        • Run chcp - output: 437 (expected)
        • Run exit (back to bash)
      • Run cmd.exe /c chcp - output: 437 (why did the chcp inside the Windows child process have an effect on future child processes spawned by its Linux parent?)
      • Run exit (back to outer cmd)
    • Run chcp - output: 850 (looking at the previous 437 output, why is it not changed here as well?)

My workaround for this: Create a file at /usr/bin/docker-compose and put this in:

#!/usr/bin/env bash

trap "/mnt/c/Windows/System32/cmd.exe /C chcp 65001" INT

/mnt/c/Windows/System32/cmd.exe /C chcp 850

/mnt/c/Program\ Files/Docker/Docker/resources/bin/docker-compose.exe "$@"

/mnt/c/Windows/System32/cmd.exe /C chcp 65001

now I can run docker-compose from Bash on Ubuntu on Windows without any encoding issues.

I run such thing using custom linux executables that contain code like this: cmd.exe /c "chcp 850 >> nul && docker-compose $@"

The same thing goed for multipe python based apps

I would really recommend using Linux native clients for docker and docker-compose, they are easy to install and it will keep you from many problems with executing windows natives in WSL.

OS: MS Windows 10 Pro - Version: 10.0.15063 Build 15063 - Bash on Ubuntu on Windows (WSL) Docker CE for Windows - Version 17.06.0-ce-win19 (12801) - Channel: stable c98c1c2 ConEmu console emulator: 161206 [64] {Stable}

To determine what ‘active code page’ the standard MS Windows CMD console is using, open an MS Windows CMD prompt, then type:

chcp

My machine (British English) returned the code for Multilingual (Latin I):

Active code page: 850

After discovering the current ‘active code page’ is 850 in the ConEmu console emulator type:

cmd.exe /c chcp

This verified the current ‘active code page’ the ConEmu console emulator is using to be:

Active code page: 65001

To set the ‘active code page’ to be 850 in the ConEmu console emulator type;

cmd.exe /c chcp 850

Which responds with;

Active code page: 850

Finally the command…

docker-compose -version

…responds as expected with;

docker-compose version 1.14.0, build c7bdf9e3

Although, I haven’t yet worked out how to make this permanent as I expect the next time I restart I will have to repeat the previous steps?

@sunjoong correct the change was not permanent.

Permanently apply this setting to all future bash sessions by adding the following to your ~/.bashrc file

# Set Multilingual (Latin I) aka Code Page 850 (Workaround issue with docker-compose and utf8 incompatibility)

chcp.com 850

It seems this causes other problems such as the output of the npm install dependency tree, so it appears the safest way is to change the code page to 850, before running docker-compose and then returning the code page back to the original value.

Does anyone know how to automate this?

OK, inspired by this GitHub post: Unicode Display Error (discovered via npm STDOUT) #3819 a workaround is to pipe the output to a string, eg;

docker-compose version | while read line; do echo $line; done

will output…

docker-compose version 1.14.0, build c7bdf9e3
docker-py version: 2.3.0
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.2j 26 Sep 2016

This is a known bug: GitHub post: Unknown encoding error in Windows with cp65001 #2775.

Hopefully it will be fixed soon.

@sgtoj From my investigations the issue is that the Python installed in WSL does not support the code page 65001. You can see this if you dig around a bit in /usr/lib/python2.7/encodings/.

I am using cmder and the program can be run in the same terminal if I set it to use powershell. It is thus not isolated to docker-compose and I suspect it will be like this for any program that is compiled using codepage 65001. That leaves it to whether it is a thing that should be supported in WSL. My opinion on this matter is that it should as one of the features of WSL is running Windows programs in bash.