unit: unit 1.29 unexpected returns getfilesystemencoding is ascii

debian 11, all locale settings *.uft-8

system (as before):

$ python
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getfilesystemencoding()
'utf-8'

but in application (django) launched on unit + unit-python3.9 sys.getfilesystemencoding() returns after update (from 1.28 to 1.29) ‘ascii’ (with appropriate ancient spell “UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position” etc)

maybe it"s related to this?:

Feature: support per-application cgroups on Linux.

(nothing more suitable)

after updates-rollback is everything works. i read docs, issues, maybe i missed something?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19 (12 by maintainers)

Commits related to this issue

Python: Fix enabling of UTF-8 in some situations. There was a couple of reports of Python applications failing due to the following type of error File "/opt/netbox/netbox/netbox/configuration.py", l... — committed to nginx/unit by ac000 a year ago
Python: Fix enabling of UTF-8 in some situations. There was a couple of reports of Python applications failing due to the following type of error File "/opt/netbox/netbox/netbox/configuration.py", l... — committed to nginx/unit by ac000 a year ago

Most upvoted comments

Just compiled unit with this patch. No longer see any ascii related issues logged at all Hope all others will have same result 🫡

hawiliali on Jan 7, 2023

This patch seems to do the right thing and will enable UTF-8 depending on the setting of the LC_CTYPE environment variable (this is the default behaviour when using the non-isolated python config).

So if LC_CTYPE is either: C, POSIX or some specific UTF-8 locale then you will get UTF-8 support.

diff --git a/src/python/nxt_python.c b/src/python/nxt_python.c
index bdb04579..ce497d33 100644
--- a/src/python/nxt_python.c
+++ b/src/python/nxt_python.c
@@ -75,8 +75,17 @@ static nxt_python_proto_t    nxt_py_proto;
 static nxt_int_t
 nxt_python3_init_config(nxt_int_t pep405)
 {
-    PyStatus  status;
-    PyConfig  config;
+    PyConfig     config;
+    PyStatus     status;
+    PyPreConfig  preconfig;
+
+    PyPreConfig_InitIsolatedConfig(&preconfig);
+    preconfig.utf8_mode = -1;
+
+    status = Py_PreInitialize(&preconfig);
+    if (PyStatus_Exception(status)) {
+        return NXT_ERROR;
+    }
 
     PyConfig_InitIsolatedConfig(&config);

ac000 on Jan 6, 2023

So for Python 3.8+ when you have the home option set we now go through the PyConfig API.

When this happens config.filesystem_encoding defaults to NULL if not set (which we don’t) which seems to be the cause of the issue.

There’s a simple patch you could try to verify this…

diff --git a/src/python/nxt_python.c b/src/python/nxt_python.c
index bdb04579..0f9b373e 100644
--- a/src/python/nxt_python.c
+++ b/src/python/nxt_python.c
@@ -94,6 +94,8 @@ nxt_python3_init_config(nxt_int_t pep405)
         }
     }
 
+    PyConfig_SetString(&config, &config.filesystem_encoding, L"utf-8");
+
     status = Py_InitializeFromConfig(&config);
     if (PyStatus_Exception(status)) {
         goto pyinit_exception;

When we don’t go this route things seem to work out as before.

Sill some work to be done…

ac000 on Dec 30, 2022

So it seems virtualenv is what tickles this particular bug. Don’t know much about it but this looks like a crucial piece of information. Thanks!.

ac000 on Dec 28, 2022