python: Python images (At least bookwork-slim) relased today break installation of sdist packages

Following the discussion her https://github.com/pypa/wheel/issues/660#issuecomment-2788333871 - images relesed today started to break CI - Airflow, but also other people have similar issue.

I explained it in the issue but let me copy it here as well:

Example failure: https://github.com/apache/airflow/actions/runs/14347343802/job/40220981486#step:6:809

6 13.37 Building wheels for collected packages: pyspark
docker-library/official-images#6 13.37   Building wheel for pyspark (setup.py): started
docker-library/official-images#6 14.11   Building wheel for pyspark (setup.py): finished with status 'error'
docker-library/official-images#6 14.11   error: subprocess-exited-with-error
docker-library/official-images#6 14.11   
docker-library/official-images#6 14.11   × python setup.py bdist_wheel did not run successfully.
docker-library/official-images#6 14.11   │ exit code: 1
docker-library/official-images#6 14.11   ╰─> [8 lines of output]
docker-library/official-images#6 14.11       /home/airflow/.local/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
docker-library/official-images#6 14.11         warnings.warn(
docker-library/official-images#6 14.11       usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
docker-library/official-images#6 14.11          or: setup.py --help [cmd1 cmd2 ...]
docker-library/official-images#6 14.11          or: setup.py --help-commands
docker-library/official-images#6 14.11          or: setup.py cmd --help
docker-library/official-images#6 14.11       
docker-library/official-images#6 14.11       error: invalid command 'bdist_wheel'
docker-library/official-images#6 14.11       [end of output]
docker-library/official-images#6 14.11   
docker-library/official-images#6 14.11   note: This error originates from a subprocess, and is likely not a problem with pip.
docker-library/official-images#6 14.11   ERROR: Failed building wheel for pyspark
docker-library/official-images#6 14.12   Running setup.py clean for pyspark

And after some (non-obvious investigation I found out what the root cause is and that’s somethign you might want to talk to the maintainers of official Python images in dockerhub, because probably a lot of people using them will have similar issue after 0.46+ wheel that does not contain bdist_wheel any more and rely on setuptools > 70.1.

The problem is that some - latest - official python images available in dockerhub are causing this problem to happen.

If you look - for example - at latest Python debian bookworm slim images you will see this:

[jarek:~/code/airflow] main ± docker run -it python:3.9-slim-bookworm bash
root@fb321ef4723e:/# pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 58.1.0
wheel      0.45.1

[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
root@fb321ef4723e:/# exit
exit
[jarek:~/code/airflow] main 14s ± docker run -it python:3.10-slim-bookworm bash 
root@2234dee7aba0:/# pip list
Package    Version
---------- -------
packaging  24.2
pip        23.0.1
setuptools 65.5.1
wheel      0.46.1

[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
root@2234dee7aba0:/# exit
exit
[jarek:~/code/airflow] main 25s ± docker run -it python:3.11-slim-bookworm bash
root@b62343d129a6:/# pip list
Package    Version
---------- -------
packaging  24.2
pip        24.0
setuptools 65.5.1
wheel      0.46.1

[jarek:~/code/airflow] main 1m5s ± docker run -it python:3.12-slim-bookworm bash
root@411574e67a7e:/# pip list
Package Version
------- -------
pip     25.0.1
root@411574e67a7e:/# 

As you can see - default version of setuptools in all those images causes problem:

  • Python 3.9 -> both wheel and setuptools are old, but if you just upgrade wheel but do not upgrade setuptools, sdist installations will fail
  • Python 3.10, 3.11 -> wheel is already updated, but setuptools is < 70.1 - which mean that by default installing somethign from sdist wil fail there - even if you do not touch setuptools and wheel
  • Python 3.12 -> all good, no setuptools, no wheel, newest versions of those will be installed

I am going to upgrade setuptools and wheels to latest versions in our images, so we will solve the problem in Airflow, but I think currently released “official images” need to be rebuild and latest setuptools should be installed in all of them.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 18
  • Comments: 21 (8 by maintainers)

Commits related to this issue

Most upvoted comments

We’ll be making a PR to pin wheel shortly as that seems to be the least breaking change.

Hi Jarek!

The repo for the Docker Python images is: https://github.com/docker-library/python/issues

(Maybe someone with permissions can use the issue transfer feature to move this issue there? I’m not a maintainer)

Those images intentionally don’t use latest setuptools (since there are breaking changes in newer setuptools), but instead use the same version provided by the ensurepip command (with a slight fixup for 65.5.0 -> 65.5.1).

The issue is that ensurepip doesn’t include wheel (only pip and setuptools; at least until Python 3.12+ where it also drops setuptools), so the Python docker image scripts don’t have a “reference” version of wheel they can use, and instead pull in the latest wheel version here: https://github.com/docker-library/python/blob/f3c69c5acce909a76e287ef1adf8f10c476eb2f0/Dockerfile-linux.template#L299-L304

This hasn’t historically caused problems, since wheel hasn’t made many breaking changes. However, with wheel 0.46 dropping support for older setuptools, this is no longer the case.

The options presumably are:

  1. Add a cap to the wheel requirement (eg wheel<0.46) so the installed wheel version remains compatible with the older setuptools. This could even just be hardcoded, since for Python 3.12+ the images don’t install either setuptools or wheel, so there won’t be any image combination where wheel 0.46+ needs to be installed.
  2. Upgrade setuptools to a newer version too, so it’s compatible with the newer wheel version. (But this would be breaking both now and any time setuptools makes changes, and diverge from the current strategy of matching ensurepip behaviour.)
  3. Stop installing wheel in the Python 3.9-3.12 images entirely, to truly match the ensurepip behaviour. This will cause pip to use isolated build environments by default when building sdists, and inside those isolated environments newer setuptools will be used that doesn’t depend on wheel. This presumably would be almost as breaking as (2) from a setuptools perspective, but not quite (since it doesn’t affect the version of setuptools used at run-time, outside of those PEP 517 isolated build environments). However, it would be more breaking for people who use wheel directly.

Out of those, (1) seems the safest to me - and I suppose is in keeping with the ~“we don’t upgrade tooling to latest, but generally prefer stability” approach?

cc @tianon @yosifkit (who are the Python docker image maintainers)

For 3.11-slim

RUN pip install --no-cache-dir setuptools==78.1.0 wheel==0.46.1 COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt

Also, not really sure how to log the version that was working previously 😦

252.1 Downloading jsonpointer-3.0.0-py2.py3-none-any.whl (7.6 kB)
252.2 Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
252.3 Downloading pycparser-2.22-py3-none-any.whl (117 kB)
252.4    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 881.3 kB/s eta 0:00:00
252.5 Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
252.6 Building wheels for collected packages: starkbank-ecdsa, jstyleson
252.6   Building wheel for starkbank-ecdsa (setup.py): started
252.7   Building wheel for starkbank-ecdsa (setup.py): finished with status 'error'
252.7   error: subprocess-exited-with-error
252.7   
252.7   × python setup.py bdist_wheel did not run successfully.
252.7   │ exit code: 1
252.7   ╰─> [6 lines of output]
252.7       usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
252.7          or: setup.py --help [cmd1 cmd2 ...]
252.7          or: setup.py --help-commands
252.7          or: setup.py cmd --help
252.7       
252.7       error: invalid command 'bdist_wheel'
252.7       [end of output]
252.7   
252.7   note: This error originates from a subprocess, and is likely not a problem with pip.
252.7   ERROR: Failed building wheel for starkbank-ecdsa
252.7   Running setup.py clean for starkbank-ecdsa
252.7   Building wheel for jstyleson (setup.py): started
252.8   Building wheel for jstyleson (setup.py): finished with status 'error'
252.8   error: subprocess-exited-with-error
252.8   
252.8   × python setup.py bdist_wheel did not run successfully.
252.8   │ exit code: 1
252.8   ╰─> [8 lines of output]
252.8       /usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
252.8         warnings.warn(
252.8       usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
252.8          or: setup.py --help [cmd1 cmd2 ...]
252.8          or: setup.py --help-commands
252.8          or: setup.py cmd --help
252.8       
252.8       error: invalid command 'bdist_wheel'
252.8       [end of output]
252.8   
252.8   note: This error originates from a subprocess, and is likely not a problem with pip.
252.8   ERROR: Failed building wheel for jstyleson
252.8   Running setup.py clean for jstyleson
252.9 Failed to build starkbank-ecdsa jstyleson
252.9 ERROR: Could not build wheels for starkbank-ecdsa, jstyleson, which is required to install pyproject.toml-based projects
253.2 
253.2 [notice] A new release of pip is available: 24.0 -> 25.0.1
253.2 [notice] To update, run: pip install --upgrade pip
------
failed to solve: process "/bin/sh -c pip install --no-cache-dir -r requirements.txt" did not complete successfully: exit code: 1

Python 3.9 -> both wheel and setuptools are old, but if you just upgrade wheel but do not upgrade setuptools, sdist installations will fail

Btw the different behaviour for Python 3.9 is temporary - and due to the 3.9.22 release being delayed by https://github.com/python/cpython/issues/132287 - ie: the Python 3.9 images haven’t been rebuilt yet since newer wheel came out. Once https://github.com/docker-library/official-images/pull/18807 is merged the Python 3.9.22 images will have newer wheel too (and so also be broken like the 3.10-3.11 images).

Not really - wheel version does not matter or it even can be not installed at all if setuptools > 70.1 . Setuptools 70.1 vendored-in the bdist_wheel command so essentially they don’t care what version of wheel is installed.

I wish this was the case, but since June 2024, setuptools does import and use wheel if it’s installed! A bad decision IMHO.

Once this is fixed, it might be worth adding an sdist pip install test - either in this existing test, or in a new one: https://github.com/docker-library/official-images/blob/master/test/tests/python-pip-requests-ssl/container.py

It’s also worth noting, that even if the wheel maintainer were to yank the 0.46.1 wheel release it wouldn’t automatically fix the Python docker images, since they would still need a rebuild (and a manifest bump like https://github.com/docker-library/official-images/pull/18803) to pick up the yank. So the fix needs to occur on the docker images side regardless.

Yep. I updated the description to clarify it.

If you meant fixing the wheel version too, to match, then yeah I agree - and that’s “option 1” above 😃

Not really - wheel version does not matter or it even can be not installed at all if setuptools > 70.1 . Setuptools 70.1 vendored-in the bdist_wheel command so essentially they don’t care what version of wheel is installed. So my proposal is quite different from 1 😃 … Just bump to a fixed setuptools version that is > 70.1

If you meant fixing the wheel version too, to match, then yeah I agree - and that’s “option 1” above 😃

I think possibly fixing (i.e. hard-coding) setuptools version is likely the solution I would recommend

The versions of setuptools in the images is already fixed, eg: https://github.com/docker-library/python/blob/f3c69c5acce909a76e287ef1adf8f10c476eb2f0/versions.sh#L158-L188 https://github.com/docker-library/python/blob/f3c69c5acce909a76e287ef1adf8f10c476eb2f0/versions.json#L8-L9