pytest: Pypy 3.9 segfaults while rewriting test assertions

a detailed description of the bug or problem you are having

When testing the project I work on in CI I frequently see a segfault in pytest’s test assert rewriting when running with pypy 3.9. (I haven’t yet seen it with 3.10, but I only just added that to CI.)

Run coverage run -m pytest --ignore=tests/simplepoll
  coverage run -m pytest --ignore=tests/simplepoll
  shell: /usr/bin/bash -e {0}
  env:
    CARGO_INCREMENTAL: 0
    CARGO_TERM_COLOR: always
    CACHE_ON_FAILURE: false
    pythonLocation: /opt/hostedtoolcache/PyPy/3.9.16/x64/bin
    PYTHONFAULTHANDLER: 1
    RUST_BACKTRACE: 1
Fatal Python error: Segmentation fault

Stack (most recent call first, approximate line numbers):
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py", line 346 in _rewrite_test
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py", line 138 in exec_module
  File "<frozen importlib._bootstrap>", line 659 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1002 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1018 in _gcd_import
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/importlib/__init__.py", line 109 in import_module
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/pathlib.py", line 486 in import_path
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 613 in _importtestmodule
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 527 in _getobj
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 305 in obj
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 536 in _inject_setup_module_fixture
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 530 in collect
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 372 in <lambda>
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 318 in from_call
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 371 in pytest_make_collect_report
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 544 in collect_one_node
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 827 in genitems
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 827 in genitems
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 610 in perform_collect
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 333 in pytest_collection
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 320 in _main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 257 in wrap_session
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 134 in main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 182 in console_main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 1 in <module>
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/execfile.py", line 169 in run
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 813 in do_run
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 611 in command_line
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 964 in main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/bin/coverage", line 3 in <module>
  File "<builtin>/app_main.py", line 1012 in execfile
  File "<builtin>/app_main.py", line 131 in run_toplevel
  File "<builtin>/app_main.py", line 733 in run_command_line
  File "<builtin>/app_main.py", line 1126 in entry_point
/home/runner/actions-runner/_work/_temp/4b79962b-6f8c-4a60-9e1a-65886acc7eb7.sh: line 1:  1738 Segmentation fault      (core dumped) coverage run -m pytest --ignore=tests/simplepoll
Error: Process completed with exit code 139.

output of pip list from the virtual environment you are using

Package                Version               Editable project location
---------------------- --------------------- --------------------------------------------------
amqp                   5.1.1
anyio                  3.7.0
asgiref                3.6.0
attrs                  23.1.0
billiard               4.1.0
black                  23.3.0
boto3                  1.27.0
botocore               1.30.0
build                  0.10.0
celery                 5.3.1
Cerberus               1.3.4
certifi                2023.5.7
cffi                   1.15.1
charset-normalizer     3.1.0
click                  8.1.3
click-didyoumean       0.3.0
click-plugins          1.1.1
click-repl             0.3.0
colored                1.4.4
coverage               7.2.7
dj-database-url        2.0.0
Django                 3.2.20
django-environ         0.10.0
django-hashid-field    3.3.7
django-pgcrypto-fields 2.6.0
djangorestframework    3.14.0
exceptiongroup         1.1.2
factory-boy            3.2.1
Faker                  18.11.2
freezegun              1.2.2
greenlet               0.4.13
h11                    0.14.0
hashids                1.3.1
hpy                    0.0.4.dev179+g9b5d200
httpcore               0.17.2
httpretty              1.1.4
httpx                  0.24.1
huey                   2.4.5
idna                   3.4
iniconfig              2.0.0
Jinja2                 3.1.2
jmespath               1.0.1
kolo                   2.11.0+local          /home/runner/actions-runner/_work/kolo/kolo/python
kombu                  5.3.1
MarkupSafe             2.1.3
maturin                1.1.0
more-itertools         9.1.0
mypy-extensions        1.0.0
packaging              23.1
pathspec               0.11.1
pip                    23.0.1
platformdirs           3.8.0
pluggy                 1.2.0
prompt-toolkit         3.0.38
pyproject_hooks        1.0.0
pytest                 7.4.0
pytest-asyncio         0.21.0
pytest-celery          0.0.0
pytest-django          4.5.2
pytest-httpx           0.22.0
python-dateutil        2.8.2
pytz                   2023.3
readline               6.2.4.1
requests               2.31.0
s3transfer             0.6.1
setuptools             58.1.0
six                    1.16.0
sniffio                1.3.0
sqlglot                17.0.0
sqlparse               0.4.4
syrupy                 4.0.4
temppathlib            1.2.0
tomli                  2.0.1
tomli_w                1.0.0
toolz                  0.12.0
types-freezegun        1.1.10
types-requests         2.31.0.1
types-urllib3          1.26.25.13
typing_extensions      4.7.1
tzdata                 2023.3
ulid-py                1.1.0
urllib3                1.26.15
vine                   5.0.0
wcwidth                0.2.6

pytest and operating system versions pytest 7.4.0 ubuntu 22.04 (buildjet-2vcpu-ubuntu-2204 https://buildjet.com/for-github-actions/docs/runners/hardware)
minimal example if possible I’m not sure how to narrow down to the problematic files.

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 25 (4 by maintainers)

Most upvoted comments

just to give a brief update, I am finally making progress on the bug. it’s very likely not related to pytest at all, but a bug in pypy’s GC(!) that just accidentally seems to manifest in pytest AST rewriting (in several projects). I hope to get a fix merged in the next week or two.

cfbolz on Mar 24, 2024

I’d love to see a write up of the problem once you’ve fixed it.

already working on a draft of that 😊

cfbolz on Mar 24, 2024

This is now fixed upstream: https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html

LilyFoote on Mar 27, 2024

Awesome!! I’d love to see a write up of the problem once you’ve fixed it.

LilyFoote on Mar 24, 2024

have reported this to the pypy project, with some further investigation of my own: https://foss.heptapod.net/pypy/pypy/-/issues/3960

fwiw, PyPy has moved to github, so the upstream issue is now at https://github.com/pypy/pypy/issues/3959

mattip on Feb 1, 2024

ah, I see. they still want a credit card, which I don’t have. if I open a PR on your repo with my changes, we could run it on your repo, if that’s ok for you @LilyFoote?

cfbolz on Jul 7, 2023

Yeah, that looks like it crashes somewhere deep inside PyPy internals. I’m not really familiar with them, unfortunately. Perhaps it would make more sense to report this there?

The-Compiler on Jul 5, 2023

Perhaps trying to get something like gdb -ex "set debuginfod enabled on" -ex r -ex bt -ex q --args $(which python) -m pytest to run on GitHub Actions could help, since it hopefully will end up displaying a C stacktrace.

The-Compiler on Jul 4, 2023

Thanks for the report! I think this is probably a PyPy bug, but let’s see if we can narrow it down a bit before we report it upstream.

Since it seems to happen while rewriting assertions, I expect it’s possible to trigger this with a single test file, and there’s good tooling to analyse what’s happening from there.

Do you have a reproducible example? The goal here is a local shell command which reliably triggers the segfault, even if we have to run pytest many times in a loop - debugging when you have to wait for CI each time is… not fun, and being unable to trust your observations is even worse.
Let’s try to narrow down which files are implicated. Can you run pytest on (roughly) half of your test suite, and bisect that way? Alternatively, you could patch pytest to print what module it’s about to rewrite (with flush=True) and then see what’s printed just before the segfault.
Try simplifying the environment and removing dependencies. This is much easier once you have a single file; at that point try deleting chunks of code, imports, etc. with the goal of reproducing in a virtual environment with no proprietary code - that confirms it’s an OSS bug somewhere, and lets us try to debug it too.

Zac-HD on Jul 4, 2023