pyyaml: "AttributeError: cython_sources" with Cython 3.0.0a10

Any update on this, now that Cython 3.0 has been released?

Our workaround for the time being is, in our build process, to install a compatible cython first and then use --no-build-isolation. Eg

pip install "Cython<3.0" pyyaml --no-build-isolation # new line
pip install -r requirements.txt # original build step, which has pyyaml in it

if that helps anyone

Is it possible to release a 5.4.2 version with Cpython restricted to “❤️.0.0” ? This should solve some of the immediate problems related to 3rd party libraries pinning down PyYaml to versions lower than 6.0.

As with all the mentions, this is breaking our ability to build our production apps.

Is there any chance of getting a release that pins Cython? Failing that does anyone know how to work around this by pinning Cython when using poetry?

Edit: I’ve realized that this is happening for PyYaml 6.0 too

Two workarounds exist:

• Preinstall cython<3, then install pyyaml without build isolation, then install the rest of your dependencies https://github.com/yaml/pyyaml/issues/601#issuecomment-1660456820

$ pip install "cython<3.0.0" wheel
$ pip install "pyyaml==5.4.1" --no-build-isolation
$ pip install -r requirements.txt

• Use a constraints file to force pip to use cython<3 at build time https://github.com/yaml/pyyaml/issues/601#issuecomment-1661960662

$ echo "cython<3" > /tmp/constraint.txt
$ PIP_CONSTRAINT=/tmp/constraint.txt pip install -r requirements.txt

Solution

pip install "Cython<3.0" pyyaml --no-build-isolation
pip install -r requirements.txt

For Solving this on Docker: Include these 2 lines in your Dockerfile

RUN pip install "Cython<3.0" pyyaml --no-build-isolation
RUN pip install -r requirements.txt

I hope it solves your problem 😃

Try ths： pip install PyYaml==3.10

Ping. Cython 3.0.0 has already reached beta releases, so we need some solution sooner than later.

@r-asiebert The specific issue here is if you’re trying to install PyYAML on a system that doesn’t have a wheel available. Cython is only involved when building from source. Most of the report are on Alpine because there aren’t musl wheels available. If 6.x is working, my guess is you’re using Python 3.10 or 3.11 which didn’t have wheels available in the 5.4.1 release. This will continue to be an issue until the build dependencies are properly constrained.

I’m using python 11 on alpine on a Fresh environment using Poetry… But, as mentioned by kylebarron, this affects me anyways…

Totally solved on 6.0.1 for me

For my poetry based projects building in docker I was able to work around this by doing the following in my docker file:

RUN  poetry run pip install "Cython<3.0" pyyaml --no-build-isolation && \
     poetry install

Is it possible to release a 5.4.2 version with Cpython restricted to “❤️.0.0” ? This should solve some of the immediate problems related to 3rd party libraries pinning down PyYaml to versions lower than 6.0.

This would help out a ton! Half of my CI builds are still wrecked by third-party dependencies that keep trying to install 5.4.1

Sorry for commenting on someone else but DO NOT USE @yinxs2003 solution without considerations. It will install more than a 10-year-old version. In the meantime, PyYAML has at least one critical vulnerability fixed.

@yinxs2003 If You could at least inform people of the danger of using “older” code it would be great.

For folks relying on aws-cli, pinning pyyaml<=5.3.1 helped us.

NOTE - please read this note from aws-cli maintainers before you use pyyaml<=5.3.1:

We do not recommend installing an older version of PyYAML as PyYAML version 5.3.1 is associated with CVE-2020-14343 that was fixed in version 5.4.

#702 was merged, 6.0.1 released, and this can be closed https://github.com/yaml/pyyaml/pull/702#issuecomment-1639058497

@nitzmahone is there anyway we can get https://github.com/yaml/pyyaml/pull/702 merged and released in 6.x to help stem the current build issues? There unfortunately isn’t a robust way for downstream packages to mitigate this without defining a runtime dependency on Cython for constraint.

This issue affects tensorflow/models object detection api installation and gives the same set of errors.

Downgrading pyyaml to 5.3.1 ( from 6.0.1 ) manually resolves it.

Just building on @pinghedm’s solution. If you happen to have another package that depends on pyyaml 5.x (like awscli). You can install it like this:

pip install "Cython<3.0" "pyyaml<6" --no-build-isolation
pip install -r requirements.txt

Thanks @jwmatthews I will check this , For now I have

pip install “cython<3.0.0” wheel && pip install pyyaml==5.4.1 --no-build-isolation, It seems my build happens fine. I will check if my tests pass

I would dearly love to see https://github.com/yaml/pyyaml/pull/702 get merged and released as 6.0.1, as a workaround to this issue, at least for folks using version 6.0.0

^^above not working for me

Here’s my dockerFile

FROM python:3.10-alpine

RUN pip install "Cython<3.0" pyyaml --no-build-isolation
RUN pip install -r requirements.txt

• requirements.txt

PyGithub==1.57
PyYAML==6.0

It’s definitely a regression between Cython 3.0.0a9 and a10; every previous 3.0.0 alpha has been able to build the pyyaml extension fine. This PR (included in a10) swapped out the base class on Cython’s custom build_ext distutils command (which pyyaml further subclasses to customize some stuff). I haven’t dug down all the way yet to see if it’s something we should try to address in pyyaml’s customization or if it’s just a bug in the new base class.

Thanks for the report- regardless, we’ll want to get this sorted one way or another, so I’ll figure out where the problem is and work with the Cython/setuptools folks if necessary to get it working.

For anyone who’s struggling with the issue @nanonyme mentioned (installing current Cython first and using --no-build-isolation to install PyYAML), I’ve found that patching line 85 of the PyYAML setup.py script from

    from Cython.Distutils import build_ext as _build_ext

to

    from Cython.Distutils.old_build_ext import old_build_ext as _build_ext

before installing works for me with Cython 3.0.7 and PyYAML 6.0.1. It seems that the old build_ext was replaced in Cython but still lives in old_build_ext.py. That’s probably a temporary solution, though, until old_build_ext will be removed.

Edit: I see that option is discussed (and rejected for long-term use) in https://github.com/cython/cython/issues/4568.

I encountered this error with Pipenv… any help to workaround it is appreciated. Thanks I do not know how to convert the workaround pip install "cython<3.0.0" wheel && pip install pyyaml==5.4.1 --no-build-isolation in Pipefile.

Thanks @jwmatthews, this works for me

$ cat constraints.txt                                                       
cython < 3.0
PIP_CONSTRAINT=constraints.txt pip3 install -r requirements.txt --no-cache

@r-asiebert

The specific issue here is if you’re trying to install PyYAML on a system that doesn’t have a wheel available. Cython is only involved when building from source. Most of the report are on Alpine because there aren’t musl wheels available. If 6.x is working, my guess is you’re using Python 3.10 or 3.11 which didn’t have wheels available in the 5.4.1 release.

This will continue to be an issue until the build dependencies are properly constrained.

adding pip install wheel fixed it for us

FROM python:alpine

RUN apk add bash
RUN pip install wheel
RUN pip install "Cython<3.0" "pyyaml<6" --no-build-isolation
RUN pip install --no-cache-dir cfn-lint==0.65.1

@jimbojd72 I had the same problem and the resolution was to pip install wheel before the pip install "Cython<3.0" ... command

It’s not necessarily to document workarounds. You can take pip or pypa/build and keep build isolation on. (which is the default) This will install compatible dependencies into virtualenv/venv to produce a wheel. You can then install wheel outside virtualenv/venv. This issue is about fixing compatibility with new Cython.

https://stackoverflow.com/a/77491847/13086128

IMHO, cython should be listed as permanent dependency of pyyaml if installing it might fail when it is missing, otherwise is a source of pain for the users.

Also this translates to other problems down the line because if that dependency is not listed, people would have to create an additional step to pre-install ‘hidden’ dependencies before installing normal ones, as installation order might make it fail otherwise.

I observed this bug not only with latest pyyaml 3.x but also on macos and ubuntu.

At the risk of repeating what has been said a dozen times already, I found this while trying to install docker-compose with pipx. This was my workaround:

$ echo "cython<3" > /tmp/constraint.txt
$ PIP_CONSTRAINT=/tmp/constraint.txt pipx install docker-compose

Two workarounds exist:

* Preinstall `cython<3`, then install pyyaml without build isolation, then install the rest of your dependencies ["AttributeError: cython_sources" with Cython 3.0.0a10 #601 (comment)](https://github.com/yaml/pyyaml/issues/601#issuecomment-1660456820)

$ pip install "cython<3.0.0" wheel
$ pip install "pyyaml==5.4.1" --no-build-isolation
$ pip install -r requirements.txt

* Use a constraints file to force pip to use `cython<3` at build time ["AttributeError: cython_sources" with Cython 3.0.0a10 #601 (comment)](https://github.com/yaml/pyyaml/issues/601#issuecomment-1661960662)

$ echo "cython<3" > /tmp/constraint.txt
$ PIP_CONSTRAINT=/tmp/constraint.txt pip install -r requirements.txt

Thanks!

Two workarounds exist:

• Preinstall cython<3, then install pyyaml without build isolation, then install the rest of your dependencies “AttributeError: cython_sources” with Cython 3.0.0a10 #601 (comment)

$ pip install "cython<3.0.0" wheel
$ pip install "pyyaml==5.4.1" --no-build-isolation
$ pip install -r requirements.txt

• Use a constraints file to force pip to use cython<3 at build time “AttributeError: cython_sources” with Cython 3.0.0a10 #601 (comment)

$ echo "cython<3" > /tmp/constraint.txt
$ PIP_CONSTRAINT=/tmp/constraint.txt pip install -r requirements.txt

Thank you. The first approach worked for me

Totally solved on 6.0.1 for me

The same for me 🎉 Encountered this issue on 6.0, bumping version to 6.0.1 solved it.

Could we just get a .post1 out that addresses only this?

Our project’s requirements/CI/CD setup is quite complex and it’s grounded until this is addressed.

Just building on @pinghedm’s solution. If you happen to have another package that depends on pyyaml 5.x (like awscli). You can install it like this:

pip install "Cython<3.0" "pyyaml<6" --no-build-isolation
pip install -r requirements.txt

A version of this that worked with us using pipenv install in our CI pipeline:

pipenv run -- pip install "Cython<3.0" "pyyaml<6" --no-build-isolation # new line for workaround
pipenv install # existing line, installs successfully from Pipfile.lock

This is breaking all my testing for 3.12, which doesn’t have wheels (they can be uploaded pretty safely now with 3.12.0b4).

You can set PIP_CONSTRAINTS_FILE to a file with a constraint on pyyaml that will lock an older version if that helps. Or on Cython, which would help, I think. I’m not sure Poetry will respect it, but most tools use pip which will.

Reporting identical issues in an Ubuntu 20.04 container running Python 3.10.1 compiled from source + pip 21.3.

cython compilation error specifically with pyyaml==5.4.1, installs fine with latest (6.0) or 5.3.1.

Sad morning to discover PEP 518 (--no-build-isolation). So much for freezing my requirements.txt 😕

and stop using outdated distutils, as well. Use instead setuptools, or meson.

Well, yes, clearly there is no other long-term solution than making pyyaml support Cython 3.

@JannKleen yes, however, as it appears, some users are installing pyyaml through first installing cython and then using --no-build-isolation. This will bypass the build dependency requirement written into pyproject.toml and reproduce the original problem.

Version 6.0.1 now seems to require Cython < 3.0.0, so the workaround isn’t required anymore I believe.

Hello. pyyaml 3.13, 5.3.1, 5.4.1, 6.0.1 py 2.7, 3.5, 3.8 python setup build

result:

python38 setup.py build
/usr/lib64/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
running build
running build_py
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
Traceback (most recent call last):
  File "setup.py", line 291, in <module>
    setup(
  File "/usr/lib64/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib64/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 185, in run
    _build_ext.run(self)
  File "/usr/lib64/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "setup.py", line 227, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/usr/lib64/python3.8/distutils/cmd.py", line 103, in __getattr__
    raise AttributeError(attr)
AttributeError: cython_sources

all

This normal?

Same for me

$ pip install awscli
Collecting awscli
  Downloading awscli-1.29.3-py3-none-any.whl (4.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 39.5 MB/s eta 0:00:00
Collecting botocore==1.31.3 (from awscli)
  Downloading botocore-1.31.3-py3-none-any.whl (11.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.0/11.0 MB 116.6 MB/s eta 0:00:00
Collecting docutils<0.17,>=0.10 (from awscli)
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 548.2/548.2 kB 57.2 MB/s eta 0:00:00
Collecting s3transfer<0.7.0,>=0.6.0 (from awscli)
  Downloading s3transfer-0.6.1-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 kB 13.4 MB/s eta 0:00:00
Collecting PyYAML<5.5,>=3.10 (from awscli)
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 27.3 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      /tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-rlbi9ugx/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
``

Does anybody solved the issue for python:3.9-slim-bullseye docker image?

@r-asiebert Are you running into this issue with 6.0?

It does, at least for me (I cut in the stack trace for brevity):

$ docker run --rm python:alpine python -m pip install PyYAML 
Collecting PyYAML
  Downloading PyYAML-6.0.tar.gz (124 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.0/125.0 kB 4.4 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        (...)
        File "/tmp/pip-build-env-n1npha3u/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Is there any way to specify --no-build-isolation inside of a setup.cfg or requirements.txt file?

this fixed it for me.

RUN apk update && apk add build-base libffi-dev
RUN pip install "Cython<3.0" pyyaml --no-build-isolation
RUN pip install -r requirements.txt

@muleyashutosh

error: command ‘gcc’ failed: No such file or directory

Install gcc and other required build dependencies before doing the pip install (apk add gcc etc)

I encountered this error when attempting to lock my dependencies with pipenv; I was able to fix it in my case by setting

[pipenv]
allow_prereleases = false

just figured I’d share this in case it helps someone else.

Two workarounds exist:

• Preinstall cython<3, then install pyyaml without build isolation, then install the rest of your dependencies “AttributeError: cython_sources” with Cython 3.0.0a10 #601 (comment)

$ pip install "cython<3.0.0" wheel
$ pip install "pyyaml==5.4.1" --no-build-isolation
$ pip install -r requirements.txt

• Use a constraints file to force pip to use cython<3 at build time “AttributeError: cython_sources” with Cython 3.0.0a10 #601 (comment)

$ echo "cython<3" > /tmp/constraint.txt
$ PIP_CONSTRAINT=/tmp/constraint.txt pip install -r requirements.txt

same, this worked for me as well. thanks!

https://stackoverflow.com/questions/77490435/attributeerror-cython-sources/77491847#77491847

Solution

That is just a workaround and doesn’t solve the underlying problem which is especially important if you can’t downgrade cython because another project actually requires 3.0

This works if you compile from build file from GoHlke

I downgraded pyyaml version:

[tool.poetry.dependencies]
...
Cython = "<3.0.0"
pyyaml = "5.3.1"

It’s working! For the time being. I think.

For the poetry users, when you run a command like the following in your Dockerfile, just make sure the pyyaml version matches whatever is in your poetry.lock file, otherwise poetry will try to install a new version and you get the same error.

poetry run pip install "Cython<3.0" "pyyaml==5.4.1" --no-build-isolation

same for me on python3.9 (alpine3.18 )

#11 17.85 Collecting PyYAML>=5.1 (from drf-spectacular==0.17.3->-r requirements/production.txt (line 15))
#11 17.86   Downloading PyYAML-6.0.tar.gz (124 kB)
#11 17.86      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.0/125.0 kB 36.9 MB/s eta 0:00:00
#11 17.93   Installing build dependencies: started
#11 20.94   Installing build dependencies: finished with status 'done'
#11 20.94   Getting requirements to build wheel: started
#11 21.05   Getting requirements to build wheel: finished with status 'error'
#11 21.08   error: subprocess-exited-with-error
#11 21.08   
#11 21.08   × Getting requirements to build wheel did not run successfully.
#11 21.08   │ exit code: 1
#11 21.08   ╰─> [48 lines of output]
#11 21.08       running egg_info
#11 21.08       writing lib/PyYAML.egg-info/PKG-INFO
#11 21.08       writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
#11 21.08       writing top-level names to lib/PyYAML.egg-info/top_level.txt
#11 21.08       Traceback (most recent call last):
#11 21.08         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#11 21.08           main()
#11 21.08         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#11 21.08           json_out['return_val'] = hook(**hook_input['kwargs'])
#11 21.08         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
#11 21.08           return hook(config_settings)
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
#11 21.08           return self._get_build_requires(config_settings, requirements=['wheel'])
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
#11 21.08           self.run_setup()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
#11 21.08           exec(code, locals())
#11 21.08         File "<string>", line 288, in <module>
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
#11 21.08           return distutils.core.setup(**attrs)
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
#11 21.08           return run_commands(dist)
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#11 21.08           dist.run_commands()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#11 21.08           self.run_command(cmd)
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
#11 21.08           super().run_command(command)
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#11 21.08           cmd_obj.run()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 314, in run
#11 21.08           self.find_sources()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
#11 21.08           mm.run()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 551, in run
#11 21.08           self.add_defaults()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
#11 21.08           sdist.add_defaults(self)
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
#11 21.08           super().add_defaults()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#11 21.08           self._add_defaults_ext()
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#11 21.08           self.filelist.extend(build_ext.get_source_files())
#11 21.08         File "<string>", line 204, in get_source_files
#11 21.08         File "/tmp/pip-build-env-wdo84unn/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#11 21.08           raise AttributeError(attr)
#11 21.08       AttributeError: cython_sources
#11 21.08       [end of output]
#11 21.08   
#11 21.08   note: This error originates from a subprocess, and is likely not a problem with pip.
#11 21.08 error: subprocess-exited-with-error

I was able to work around this in Alpine. We’re using Alpine 3.15, but I think the same solution should work: https://github.com/Cray-HPE/cray-product-catalog/pull/263

@Kaelten thanks for the suggestion! That works for me on Poetry 1.5.1!

Anyone using poetry? It seems I only can use 6 version with it.

Does anyone had the same problem?

There unfortunately isn’t a robust way for downstream packages to mitigate this without defining a runtime dependency on Cython for constraint.

For some package managers, e.g. Poetry, even defining a Cython runtime dependency doesn’t help because Poetry builds wheels for sdist packages upon installation in a temporary directory that’s isolated from the runtime environment and only includes packages defined in PEP 517’s build-requires

Workaound doesn’t work for me as well (no gcc error though)

FROM python:3.11.4-alpine3.18

RUN apk add \
    bash \
    conntrack-tools \
    iptables \
 && rm -rf /var/cache/apk/*

WORKDIR /usr/src/kamailio-metrics/

COPY requirements.txt .
COPY pystn ./pystn

RUN python3 -m venv .venv && \
    source .venv/bin/activate && \
    pip install --upgrade pip && \
    pip install "Cython<3.0" "pyyaml<6" --no-build-isolation && \
    pip wheel install --no-cache-dir cfn-lint==0.78.1

❯ make build &&  make push
docker build -t jcr.io/platform/kamailio-metrics:jdesmarais .
[+] Building 10.0s (10/11)                                                                                                                                                                                             docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.06kB                                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3.11.4-alpine3.18                                                                                                                                                            0.8s
 => [1/7] FROM docker.io/library/python:3.11.4-alpine3.18@sha256:25df32b602118dab046b58f0fe920e3301da0727b5b07430c8bcd4b139627fdc                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                      0.0s
 => => transferring context: 3.78kB                                                                                                                                                                                                    0.0s
 => CACHED [2/7] RUN apk add     bash     conntrack-tools     iptables  && rm -rf /var/cache/apk/*                                                                                                                                     0.0s
 => CACHED [3/7] WORKDIR /usr/src/kamailio-metrics/                                                                                                                                                                                    0.0s
 => CACHED [4/7] COPY requirements.txt .                                                                                                                                                                                               0.0s
 => CACHED [5/7] COPY pystn ./pystn                                                                                                                                                                                                    0.0s
 => ERROR [6/7] RUN python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1   9.1s
------
 > [6/7] RUN python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1 &&     pip install -r requirements.txt:
4.962 Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.1.2)
5.115 Collecting pip
5.379   Downloading pip-23.2-py3-none-any.whl (2.1 MB)
5.611      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 9.1 MB/s eta 0:00:00
5.676 Installing collected packages: pip
5.676   Attempting uninstall: pip
5.679     Found existing installation: pip 23.1.2
5.727     Uninstalling pip-23.1.2:
5.734       Successfully uninstalled pip-23.1.2
7.056 Successfully installed pip-23.2
8.028 Collecting Cython<3.0
8.028   Obtaining dependency information for Cython<3.0 from https://files.pythonhosted.org/packages/cf/a3/43bf5b3ba528a8fa2aa25dd48317e493b46ce838098605277b9efc7a0c1a/Cython-0.29.36-cp311-cp311-musllinux_1_1_x86_64.whl.metadata
8.264   Downloading Cython-0.29.36-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (3.1 kB)
8.333 Collecting pyyaml<6
8.377   Downloading PyYAML-5.4.1.tar.gz (175 kB)
8.479      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 1.8 MB/s eta 0:00:00
8.651   Preparing metadata (pyproject.toml): started
8.929   Preparing metadata (pyproject.toml): finished with status 'error'
8.935   error: subprocess-exited-with-error
8.935
8.935   × Preparing metadata (pyproject.toml) did not run successfully.
8.935   │ exit code: 1
8.935   ╰─> [14 lines of output]
8.935       /usr/src/kamailio-metrics/.venv/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
8.935         warnings.warn(msg, warning_class)
8.935       running dist_info
8.935       creating /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info
8.935       writing /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/PKG-INFO
8.935       writing dependency_links to /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/dependency_links.txt
8.935       writing top-level names to /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/top_level.txt
8.935       writing manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       reading manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       reading manifest template 'MANIFEST.in'
8.935       adding license file 'LICENSE'
8.935       writing manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       creating '/tmp/pip-modern-metadata-i70halss/PyYAML-5.4.1.dist-info'
8.935       error: invalid command 'bdist_wheel'
8.935       [end of output]
8.935
8.935   note: This error originates from a subprocess, and is likely not a problem with pip.
8.938 error: metadata-generation-failed
8.938
8.938 × Encountered error while generating package metadata.
8.938 ╰─> See above for output.
8.938
8.938 note: This is an issue with the package mentioned above, not pip.
8.938 hint: See above for details.
------
Dockerfile:38
--------------------
  37 |
  38 | >>> RUN python3 -m venv .venv && \
  39 | >>>     source .venv/bin/activate && \
  40 | >>>     pip install --upgrade pip && \
  41 | >>>     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation && \
  42 | >>>     pip wheel install --no-cache-dir cfn-lint==0.78.1 && \
  43 | >>>     pip install -r requirements.txt
  44 |
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install \"Cython<3.0\" \"pyyaml<6\" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1 &&     pip install -r requirements.txt" did not complete successfully: exit code: 1

Building with pyyaml pinned to version 5.3.1 fixed this issue for me

RUN python3 -m venv .venv && \
    source .venv/bin/activate && \
    pip install --upgrade pip && \
    pip install 'pyyaml<=5.3.1' && \
    pip install wheel && \
    pip install -r requirements.txt

@nitzmahone That’s my bad, I think! The tests failed immediately (try to import imp, keeping an eye on https://github.com/cython/cython/issues/5285) but I thought it’d failed to compile too.

I’m going to test a bunch of reverse dependencies like pyyaml in Gentoo and unleash it if it works. Apologies for the error!

EDIT: Done!

@mgorny where are you hearing that? Empirically speaking, Cython 0.29.x has been working flawlessly on 3.12- PyYAML extension builds have been passing the full test suite all along through the current Python 3.12.0b1.

(not that I don’t want a better excuse to spend the time shredding PyYAML’s creaky old packaging code to work under Cython3 while still doing everything it needs to do 😆 )

@jmakov the likely short-term fix will be to pin the Cython PEP518 build deps to pre-3.0 Cython in a 6.x release (see #702), but we generally consider the 5.x release branch dead, so hadn’t planned on re-publishing the build change back there. You might want to try and get the Pycaret folks (or whoever’s ultimately pinning PyYAML to ~=5.1) to update their deps.

Cool- we’ve been poking at doing another release, and since we sat on it long enough, setuptools and pip now fully support PEP517/518 config passthru from the CLI, which should make the replacement of the dynamic bits a whole lot easier.

Yep, unfortunately --pre seems to affect all deps and build deps as well under pip.