pytest: Parametrized tests don't display unicode in ids correctly.

@pytest.mark.parametrize(
    "string1", [
        pytest.param("test1", id="unicode in id い")
])
def test_unicode_in_param_id(string1:str) -> None:
    assert string1 == "test1"

running it with pytest -v test.py gives me

platform darwin -- Python 3.9.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/solarmist/repos/pytest
collected 1 item

test.py::test_unicode_in_param_id[unicode in id \u3044] PASSED                                                                                                                                                                                         [100%]

I expected the [] to show unicode in id い

pip list

---------- -------
pip        21.2.4
setuptools 57.4.0
wheel      0.36.2```

This is on a macOS 11.4.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 4
Comments: 16 (8 by maintainers)

Most upvoted comments

Since we no longer support python 2 we ought to revisit that mr

RonnyPfannschmidt on Sep 13, 2023

I just saw this message in the docs, but in 2021 this does not feel reasonable to me anymore. Unicode should have 1st class support everywhere nowadays. Almost anyone using python for business applications needs some level of international text support.

I can understand plugins and OS display issues still being an issue and for the warning to exist for the forseeable future, but pytest itself should fully support unicode without “forfeit[ing] all rights to community support”.

Note pytest by default escapes any non-ascii characters used in unicode strings for the parametrization because it has several downsides. If however you would like to use unicode strings in parametrization and see them in the terminal as is (non-escaped), use this option in your pytest.ini:

[pytest] disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True Keep in mind however that this might cause unwanted side effects and even bugs depending on the OS used and plugins currently installed, so use it at your own risk.

solarmist on Aug 25, 2021

Also I put the option in my pyproject.toml file as
[tool.pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = true
[…]

but was unable to get any of those variations to display unescaped unicode.

So I found that if you’re explicitly doing pytest.param(..., id="..."), it’ll always escape, regardless of configuration: https://github.com/pytest-dev/pytest/blob/245a8c23dd0ffeb03710f0fbabd1f5e6a56611ab/src/_pytest/mark/structures.py#L95

As opposed to when the parameter itself is a string: https://github.com/pytest-dev/pytest/blob/245a8c23dd0ffeb03710f0fbabd1f5e6a56611ab/src/_pytest/python.py#L1050

So perhaps it’s a bug in that pytest should use the same logic with both explicit id="..." and implicit ids. But personally, I would suggest that explicitly set ids should not be escaped. I would also accept the solution of add a escape=False argument to pytest.param to turn off escaping, if pytest devs want to avoid breaking backwards compat. My reasoning is that I don’t want to enable this unsupported feature for the entire project, just for this specific test I need to test unicode for.

I also want to echo other sentiments that this shouldn’t be hidden in a flag, unicode should be the default everywhere, and OSes that have issues with pytest showing unicode should be updated (or perhaps a flag to enable unicode escaping in pytest).

brandonchinn178 on Sep 1, 2022

i created a project as a starting point, we can take it from there and move as there is time to chip away on it

RonnyPfannschmidt on Aug 26, 2021

Sure, I’m not in a hurry. I’m viewing this as a very long term project, so a week isn’t anything. The biggest thing it just to give it visibility so it doesn’t stay this way long term.

I’ll probably just chip away at it little by little as I have time myself.

solarmist on Aug 25, 2021

i’d like to note that i’m currently very short on time, so it may be a little while before i can get to bootstrap this effort, we can certainly create a milestone/project by the end of the week and declare some goals for it so the details can progress

RonnyPfannschmidt on Aug 25, 2021

Also I put the option in my pyproject.toml file as

[tool.pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = true

and

[tool.pytest.ini_options]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = true

And a pytest.ini file as

[pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True

but was unable to get any of those variations to display unescaped unicode.

solarmist on Aug 25, 2021