MINGW-packages: [Investigation] The MSYS2 + meson + Python crash issue

Another issue (old one: https://github.com/msys2/MINGW-packages/issues/11864) to collect some information and what was tried so far.

I’ve created a small repo for reproducing the issue: https://github.com/lazka/python-crash-test

Any ideas regarding what we could try welcome.

The issue

meson fails with STATUS_ACCESS_VIOLATION sometimes like this when being called from ninja, and we haven’t found a way to reproduce it locally:

FAILED: libmy-shared-lib6.dll.p/libmy-shared-lib6.dll.symbols 
"D:/a/_temp/msys64/ucrt64/bin/meson" "--internal" "symbolextractor" "D:/a/python-crash-test/python-crash-test/project/_build" libmy-shared-lib6.dll "libmy-shared-lib6.dll.a" libmy-shared-lib6.dll.p/libmy-shared-lib6.dll.symbols

This below is now out of date, see the followup answers for the cause

When has it started

When we updated from Python 3.9 to 3.10

Where it failed so far

It happened in:
It happend with:
- GHA Windows Server 2022
- GHA Windows Server 2019
- Gitlab CI using Windows Server 2016

Where it hasn’t failed so far

In MINGW32 / CLANG32
Locally, outside of CI

What makes the error go away

Running everything via powershell (and just setting PATH), and not from bash
Setting MSYS=winjitdebug
Downgrading to our last Python 3.9 mingw version (I created a rebuild for the last version we had)
Using the official CPython 3.9 installed via setup-python
Using the official CPython 3.11 installed via setup-python

What doesn’t make the error go away

Disabling Python forward-slash-path mode via MSYSTEM= doesn’t help.
Using the official CPython 3.10 installed via setup-python can also fail: https://github.com/lazka/python-crash-test/actions/runs/5157613189/jobs/9290253297 (with both 2019 and 2022)

Bisecting:

Testing with official CPython builds:

NO-ERROR: 3.9.13, 3.10.0-alpha.1, 3.10.0-alpha.2, 3.10.0-alpha.3 <-> 3.11.0-alpha.4, 3.11.0-alpha.5, 3.11.0-alpha.6, 3.11.0-alpha.7, 3.11.3
ERROR: 3.10.0-alpha.4, 3.10.0-alpha.5, 3.10.0-alpha.7, 3.10.11, 3.11.0-alpha.1, 3.11.0-alpha.3

Error started: v3.10.0a3…v3.10.0a4 Fixed: v3.11.0a3…v3.11.0a4

Bisect, see next post.

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (10 by maintainers)

Commits related to this issue

ci/msys2: switch to Python 3.11 to fix crashes during build There is a longstanding bug with random crashes of Python 3.10 on CI. See: https://github.com/python/cpython/issues/105400 https://github.... — committed to kasper93/mpv by kasper93 a year ago
ci/msys2: switch to Python 3.11 to fix crashes during build There is a long-standing bug with random crashes of Python 3.10 on CI. See: https://github.com/python/cpython/issues/105400 https://github... — committed to kasper93/mpv by kasper93 a year ago
ci/msys2: switch to Python 3.11 to fix crashes during build There is a long-standing bug with random crashes of Python 3.10 on CI. See: https://github.com/python/cpython/issues/105400 https://github... — committed to kasper93/mpv by kasper93 a year ago
ci/msys2: switch to Python 3.11 to fix crashes during build There is a long-standing bug with random crashes of Python 3.10 on CI. See: https://github.com/python/cpython/issues/105400 https://github... — committed to mpv-player/mpv by kasper93 a year ago
ci/msys2: switch to Python 3.11 to fix crashes during build There is a long-standing bug with random crashes of Python 3.10 on CI. See: https://github.com/python/cpython/issues/105400 https://github... — committed to dyphire/mpv by kasper93 a year ago
ci/msys2: switch to Python 3.11 to fix crashes during build There is a long-standing bug with random crashes of Python 3.10 on CI. See: https://github.com/python/cpython/issues/105400 https://github... — committed to dyphire/mpv by kasper93 a year ago
Revert "CI: add potential workaround for python crashes in MSYS2" This reverts commit e945f35cd72402d0d204ff10870e2a95c59b6192. With MSYS2 udpating to Python 3.11, this should no longer be needed. S... — committed to lazka/meson by lazka a year ago
Revert "CI: add potential workaround for python crashes in MSYS2" This reverts commit e945f35cd72402d0d204ff10870e2a95c59b6192. With MSYS2 udpating to Python 3.11, this should no longer be needed. S... — committed to lazka/meson by lazka a year ago
Revert "CI: add potential workaround for python crashes in MSYS2" This reverts commit e945f35cd72402d0d204ff10870e2a95c59b6192. With MSYS2 udpating to Python 3.11, this should no longer be needed. S... — committed to mesonbuild/meson by lazka a year ago
Revert "CI: add potential workaround for python crashes in MSYS2" This reverts commit e945f35cd72402d0d204ff10870e2a95c59b6192. With MSYS2 udpating to Python 3.11, this should no longer be needed. S... — committed to mesonbuild/meson by lazka a year ago
Revert "CI: add potential workaround for python crashes in MSYS2" This reverts commit e945f35cd72402d0d204ff10870e2a95c59b6192. With MSYS2 udpating to Python 3.11, this should no longer be needed. S... — committed to mesonbuild/meson by lazka a year ago
ci: fix transient MSYS2 failures Make sure Python is updated to >=3.11 (fix https://github.com/msys2/MINGW-packages/issues/17415). — committed to benoit-pierre/wrapdb by benoit-pierre 10 months ago
ci: fix transient MSYS2 failures Make sure Python is updated to >=3.11 (fix https://github.com/msys2/MINGW-packages/issues/17415). — committed to mesonbuild/wrapdb by benoit-pierre 10 months ago
Revert "CI: add potential workaround for python crashes in MSYS2" This reverts commit e945f35cd72402d0d204ff10870e2a95c59b6192. With MSYS2 udpating to Python 3.11, this should no longer be needed. S... — committed to bruchar1/meson by lazka a year ago
CI: Remove workaround for Python in MSYS2 jobs It was added in 13fe2e0c, but it's now unnecessary as the issue has been fixed. See https://github.com/msys2/MINGW-packages/issues/17415 — committed to GNOME/glib by lb90 9 months ago
CI: Remove workaround for Python in MSYS2 jobs It was added in 13fe2e0c, but it's now unnecessary since the issue has been fixed. See https://github.com/msys2/MINGW-packages/issues/17415 — committed to GNOME/glib by lb90 9 months ago
ci: fix transient MSYS2 failures Make sure Python is updated to >=3.11 (fix https://github.com/msys2/MINGW-packages/issues/17415). — committed to willwray/wrapdb by benoit-pierre 10 months ago

Most upvoted comments

I’ve reduced the reproducer to no longer include MSYS2 now: https://github.com/lazka/python-crash-test/blob/70349faa496cdf19c4bd5152435542542eb7a14f/.github/workflows/main.yml

lazka on Jun 6, 2023

I can confirm that a mingw build of 3.11 also doesn’t crash: https://github.com/msys2-contrib/cpython-mingw/pull/139#issuecomment-1605917011

lazka on Jun 25, 2023

Small update. I’ve reviewed meson’s symbolextractor.py code. While there are few things that could be improved, it is not related to this issue.

One important thing I noticed that the issue happens only with llvm-nm which is preferred by meson, it does not happen with mingw’s nm. With this python-crash-test reproducer.

I think next step would be to remove meson and ninja (if possible) from the reproducer. I tested with script that only runs llvm-nm on the library. The reproducer is pretty minimal, good job on that, but removing components like meson would really help to focus on important parts.

But my fear is that since it doesn’t happen with Python 3.11 there will not be much interest (if at all) to find the root cause of this. I think during process exit some resources/fd are not in a valid state which causes the issue.

kasper93 on Jun 9, 2023

It’s a Github Actions runner defined at https://github.com/lazka/python-crash-test/blob/70349faa496cdf19c4bd5152435542542eb7a14f/.github/workflows/main.yml

https://github.com/lazka/python-crash-test/actions/runs/5190923933/jobs/9358053943

eli-schwartz on Jun 6, 2023