cython: [BUG] Py_TPFLAGS_HEAPTYPE is set on static PyTypeObject

I’m working on the pyston python implementation (pyston v2 is mostly a cpython 3.8 fork and pretends to be cpython to cython) and some users notified us that they run into issues using gevent with pyston. I tracked it down to a memory corruption issue/out of bounds write generated by the cythonized code in connection with ours.

It’s caused by cython setting Py_TPFLAGS_HEAPTYPE temporarily when calling __Pyx_PyType_Ready: image

https://github.com/cython/cython/blob/6889482cebefe93d5599b56f3a0389224e92bff5/Cython/Utility/ExtensionTypes.c#L118

Pyston assumes (because Py_TPFLAGS_HEAPTYPE is set) that it got a larger PyHeapTypeObject and will clear some fields (which overwrite some random memory which causes the gevent bug)

image

https://github.com/pyston/pyston/blob/832ec0d76facb2e1e958324c7c1ce2c3879283ca/Objects/typeobject.c#L281

Maybe we can workaround the issue in pyston but in general I think setting this flag could be dangerous for cpython too it seems to check for Py_TPFLAGS_HEAPTYPE several times and the cython comment about it maybe outdated.

What do you think?

To Reproduce Code to reproduce the behaviour:

taken from the pyston bug report:
david@davitude:~$ pyenv install pyston-2.2
david@davitude:~$ ~/.pyenv/versions/pyston-2.2/bin/python -m pip install gevent
david@davitude:~$ ~/.pyenv/versions/pyston-2.2/bin/python -c "from gevent.monkey import is_object_patched"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/__init__.py", line 85, in <module>
    from gevent._config import config
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 699, in <module>
    Loop().get()
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 146, in get
    self.value = self.validate(self._default())
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 248, in validate
    return self._import_one_of([self.shortname_map.get(x, x) for x in value])
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 219, in _import_one_of
    return self._import_one(item)
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/site-packages/gevent/_config.py", line 237, in _import_one
    module = importlib.import_module(module)
  File "/home/david/.pyenv/versions/pyston-2.2/lib/pyston3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "src/gevent/libev/corecext.pyx", line 979, in init gevent.libev.corecext
SystemError: Bad call flags for CyFunction

Expected behavior Py_TPFLAGS_HEAPTYPE is only set if the PyTypeObject is a subclass of PyHeapTypeObject.

Environment (please complete the following information):

  • OS: Linux
  • Python version Pyston 2.2
  • Cython version 3.0a6

Additional context Pyston bug report: https://github.com/pyston/pyston/issues/42

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (15 by maintainers)

Commits related to this issue

Most upvoted comments

I think we can close this ticket, right? While it’s not resolved in general (the hack still applies to CPython), I don’t think there is currently anything left that hurts. At least not for Pyston2, for which this ticket was opened.

In which case, @undingen have a look at https://github.com/da-woods/cython/tree/pyston-tpflags and see if it fixes most of your issues.

I still thing longer-term a better solution might be useful though.

… a macro to test (e.g. #if defined(PYSTON_VERSION)) …

Pyston has version macros

#define PYSTON_MAJOR_VERSION        2
#define PYSTON_MINOR_VERSION        2
#define PYSTON_MICRO_VERSION        0