cython: [BUG] Py_TPFLAGS_HEAPTYPE is set on static PyTypeObject
I’m working on the pyston python implementation (pyston v2 is mostly a cpython 3.8 fork and pretends to be cpython to cython) and some users notified us that they run into issues using gevent with pyston. I tracked it down to a memory corruption issue/out of bounds write generated by the cythonized code in connection with ours.
It’s caused by cython setting Py_TPFLAGS_HEAPTYPE temporarily when calling __Pyx_PyType_Ready:

Pyston assumes (because Py_TPFLAGS_HEAPTYPE is set) that it got a larger PyHeapTypeObject and will clear some fields (which overwrite some random memory which causes the gevent bug)

Maybe we can workaround the issue in pyston but in general I think setting this flag could be dangerous for cpython too it seems to check for Py_TPFLAGS_HEAPTYPE several times and the cython comment about it maybe outdated.
What do you think?
To Reproduce Code to reproduce the behaviour:
taken from the pyston bug report:
david@davitude:~$ pyenv install pyston-2.2
david@davitude:~$ ~/.pyenv/versions/pyston-2.2/bin/python -m pip install gevent
david@davitude:~$ ~/.pyenv/versions/pyston-2.2/bin/python -c "from gevent.monkey import is_object_patched"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/__init__.py", line 85, in <module>
from gevent._config import config
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 699, in <module>
Loop().get()
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 146, in get
self.value = self.validate(self._default())
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 248, in validate
return self._import_one_of([self.shortname_map.get(x, x) for x in value])
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 219, in _import_one_of
return self._import_one(item)
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 237, in _import_one
module = importlib.import_module(module)
File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "src/gevent/libev/corecext.pyx", line 979, in init gevent.libev.corecext
SystemError: Bad call flags for CyFunction
Expected behavior
Py_TPFLAGS_HEAPTYPE is only set if the PyTypeObject is a subclass of PyHeapTypeObject.
Environment (please complete the following information):
- OS: Linux
- Python version Pyston 2.2
- Cython version 3.0a6
Additional context Pyston bug report: https://github.com/pyston/pyston/issues/42
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- remove pyston special casing Looking at the Cython code I spoted a few areas where it contains special paths for Pyston v1. Pyston v1 only supports python2 and is dead but we did release a new Pyston ... — committed to undingen/cython by undingen 3 years ago
- remove pyston special casing Looking at the Cython code I spotted a few areas where it contains special paths for Pyston v1. Pyston v1 only supports python2 and is dead but we did release a new Pyston... — committed to undingen/cython by undingen 3 years ago
- remove pyston special casing Looking at the Cython code I spotted a few areas where it contains special paths for Pyston v1. Pyston v1 only supports python2 and is dead but we did release a new Pyston... — committed to undingen/cython by undingen 3 years ago
- remove special handling of Pyston Looking at the Cython code I spotted a few areas where it contains special paths for Pyston v1. Pyston v1 only supports python2 and is dead but we did release a new P... — committed to undingen/cython by undingen 3 years ago
- Remove special handling of Pyston v1 (GH-4211) Looking at the Cython code I spotted a few areas where it contains special paths for Pyston v1. Pyston v1 only supports python2 and is dead but we did ... — committed to cython/cython by undingen 3 years ago
- Avoid setting Py_TPFLAGS_HEAPTYPE in Pyston See https://github.com/cython/cython/issues/4200 Signed-off-by: Stefan Behnel <stefan_ml@behnel.de> — committed to cython/cython by da-woods 3 years ago
- pyston support: backport memory corruption fix #4200 to v0.29 branch fixes an issue which pyston triggers https://github.com/cython/cython/issues/4200 — committed to undingen/cython by undingen 3 years ago
- pyston support: backport memory corruption fix GH-4200 to 0.29.x branch (GH-4488) Fixes an issue which Pyston triggers. See https://github.com/cython/cython/issues/4200 — committed to cython/cython by undingen 3 years ago
- pyston support: backport memory corruption fix GH-4200 to 0.29.x branch (GH-4488) Fixes an issue which Pyston triggers. See https://github.com/cython/cython/issues/4200 — committed to cython/cython by undingen 3 years ago
I think we can close this ticket, right? While it’s not resolved in general (the hack still applies to CPython), I don’t think there is currently anything left that hurts. At least not for Pyston2, for which this ticket was opened.
I committed the change in https://github.com/cython/cython/commit/b19b8d840378b556c86e40e769cd7b93d28b1f43
In which case, @undingen have a look at https://github.com/da-woods/cython/tree/pyston-tpflags and see if it fixes most of your issues.
I still thing longer-term a better solution might be useful though.
Pyston has version macros