pyzmq: BUG: pyzmq v26.0.0 installation fails on Bullseye (build from sources)

What pyzmq version?**

26.0.0

What libzmq version?**

4.3.5 (seperatly build and installed)

Python version (and how it was installed)

python 3.9 via apt-get

OS

Raspberry Pi OS Bullseye 32-bit / Github CI Docker Debian Bullseye

What happened?

In our projekt we build pyzmq from sources with a seperatly prebuild libzmq installation. Prior to the v26.0.0 release this succeeded on Bullseye and Bookworm. Since the new version release this fails on Bullseye.

Traceback, if applicable

Collecting pyzmq
  Downloading pyzmq-26.0.0.tar.gz (266 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 266.6/266.6 kB 4.9 MB/s eta 0:00:00
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Ignoring cffi: markers 'implementation_name == "pypy"' don't match your environment
  Collecting cython>=3.0.0
    Downloading https://www.piwheels.org/simple/cython/Cython-3.0.10-cp39-cp39-linux_armv7l.whl (10.7 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 3.7 MB/s eta 0:00:00
  Collecting packaging
    Downloading https://www.piwheels.org/simple/packaging/packaging-24.0-py3-none-any.whl (53 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 5.0 MB/s eta 0:00:00
  Collecting scikit-build-core
    Downloading https://www.piwheels.org/simple/scikit-build-core/scikit_build_core-0.8.2-py3-none-any.whl (140 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.5/140.5 kB 8.8 MB/s eta 0:00:00
  Collecting exceptiongroup (from scikit-build-core)
    Downloading https://www.piwheels.org/simple/exceptiongroup/exceptiongroup-1.2.0-py3-none-any.whl (16 kB)
  Collecting tomli>=1.1 (from scikit-build-core)
    Downloading https://www.piwheels.org/simple/tomli/tomli-2.0.1-py3-none-any.whl (12 kB)
  Installing collected packages: tomli, packaging, exceptiongroup, cython, scikit-build-core
  Successfully installed cython-3.0.10 exceptiongroup-1.2.0 packaging-24.0 scikit-build-core-0.8.2 tomli-2.0.1
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Collecting pyproject_metadata
    Downloading https://www.piwheels.org/simple/pyproject-metadata/pyproject_metadata-0.7.1-py3-none-any.whl (7.4 kB)
  Collecting pathspec
    Downloading https://www.piwheels.org/simple/pathspec/pathspec-0.12.1-py3-none-any.whl (31 kB)
  Collecting cmake>=3.14
    Downloading cmake-3.29.2.tar.gz (30 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'error'
    error: subprocess-exited-with-error

    × pip subprocess to install backend dependencies did not run successfully.
    │ exit code: 2
    ╰─> [55 lines of output]
        Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
        Collecting ninja>=1.5
          Downloading https://www.piwheels.org/simple/ninja/ninja-1.11.1.1-cp39-cp39-linux_armv7l.whl (132 kB)
             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.5/132.5 kB 406.0 kB/s eta 0:00:00
        Collecting cmake
          Using cached cmake-3.29.2.tar.gz (30 kB)
        ERROR: Exception:
        Traceback (most recent call last):
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
            status = run_func(*args)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
            return func(self, options, args)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
            requirement_set = resolver.resolve(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
            result = self._result = resolver.resolve(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
            state = resolution.resolve(requirements, max_rounds=max_rounds)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
            self._add_to_criteria(self.state.criteria, r, parent=None)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
            if not criterion.candidates:
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
            return bool(self._sequence)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
            return any(self)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
            return (c for c in iterator if id(c) not in self._incompatible_ids)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
            candidate = func()
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
            base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
            self._link_candidate_cache[link] = LinkCandidate(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
            super().__init__(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
            self.dist = self._prepare()
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
            dist = self._prepare_distribution()
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
            return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
            return self._prepare_linked_requirement(req, parallel_builds)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
            dist = _get_prepared_distribution(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 70, in _get_prepared_distribution
            with build_tracker.track(req, tracker_id):
          File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
            return next(self.gen)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/build/build_tracker.py", line 137, in track
            self.add(req, tracker_id)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/build/build_tracker.py", line 103, in add
            raise LookupError(message)
        LookupError: https://files.pythonhosted.org/packages/80/bf/4f9a9f754507992be28b985d1e9b17f93a2271106b5916a212efe1d65205/cmake-3.29.2.tar.gz (from https://pypi.org/simple/cmake/) (requires-python:>=3.7) is already being built: cmake>=3.14 from https://files.pythonhosted.org/packages/80/bf/4f9a9f754507992be28b985d1e9b17f93a2271106b5916a212efe1d65205/cmake-3.29.2.tar.gz
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 2
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3 /home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-vnwo_md6/normal --no-warn-script-location --no-binary pyzmq --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- pyproject_metadata pathspec 'cmake>=3.14'
  cwd: [inherit]
  Installing backend dependencies: finished with status 'error'
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

More info

  • libzmq prebuild and copied to “/usr/local”.

  • Build pyzmq ZMQ_PREFIX=“/usr/local” ZMQ_DRAFT_API=1 pip install -v pyzmq --no-binary pyzmq

We had also problems with the prerelease some time ago: https://github.com/zeromq/pyzmq/issues/1937

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (10 by maintainers)

Most upvoted comments

Hello, there seem to be an issue, many builds are missing from https://pypi.org/project/pyzmq/26.0.1/#files , especially amd64 build, which were present in https://pypi.org/project/pyzmq/26.0.0/#files

I believe #1977 fixed the issue preventing the piwheels builds of pyzmq 26, I’ll do 26.0.1 with that tonight or tomorrow.

Why is ldconfig needed after all?

See this answer.

This is unrelated to pyzmq, but ld itself. ld has a cache, so you may need to run ldconfig to refresh its cache after installing a new library, especially when you already have another version of the same library at a lower priority on ld’s path.

Using a “custom” prefix does not work, see the branch test/prefix in the testrepo.

“Custom” prefix usually needs to be combined with setting the runtime path (rpath), e.g -Wl,-rpath,$ZMQ_PREFIX/lib, as demonstrated here, and shown to be working here.

I made a mistake in my suggestion where I specified the prefix (-Wl,-rpath,$ZMQ_PREFIX), which should have been the library dir (-Wl,-rpath,$ZMQ_PREFIX/lib).

It’s arguable whether this should be done by pyzmq by default or not, because it doesn’t seem to be standard practice for other packages to do this, but I did used to do it in setup.py, so I’ve put it back in #1983.

Can you add

export LDFLAGS="-Wl,-rpath,/usr/local/lib"
export SKBUILD_CMAKE_VERBOSE=true

and share the output again?

EDIT: added missing /lib on rpath

Can you add sudo ldconfig between installing libzmq and building pyzmq? Not sure if it would help, but worth a try.

it would also help to have the full build output of pyzmq, just to see what libzmq it is finding, as well as ldd output for the .so file in pyzmq.

based on the output, it really seems like pyzmq is built with a libzmq with drafts, but a different libzmq is being found at runtime. Check if there is more than one on your system. It may be that you need to add rpath arguments if there’s more than one (ldconfig might also help with this) to ensure that the right one is loaded.

The 26.0.2 builds appeared to work fine on piwheels: https://www.piwheels.org/project/pyzmq/

The CI upload took a while today, should be all there now.