pixi: Tracking issue: PyPI dependencies pixi can't manage yet.

Mediapipe

1. pixi add --pypi mediapipe
2. × RECORD file doesn't match wheel contents: missing hash for mediapipe/version.txt (expected sha256=-fE2KU)
3. python -m pip install mediapipe does work
4. platform osx-arm64
5. work around was just to use pip to install using pixi tasks

[project]
name = "ipdscan"
version = "0.1.0"
description = "Add a short description here"
authors = ["pablovela5620 <pablovela5620@gmail.com>"]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tasks]
mp-install = "python -m pip install mediapipe"

[dependencies]
python = "3.11.*"
pip = ">=23.3.2,<23.4"
rerun-sdk = ">=0.12.0,<0.13"
requests = ">=2.31.0,<2.32"
tqdm = ">=4.66.1,<4.67"

[pypi-dependencies]
imutils = "*"

Sorry, my bad. meshkernel actually does work fine when you add the following:

[system-requirements]
macos = "14.0"

We should make the error more actionable and figure out a better default behavior.

Thanks for the quick fix! This does indeed work. I’m looking forward to further improvements to pixi.

@liblaf I wonder if for you it’s a similar issue and adding glibc 2.28 or higher would fix it.

e.g.

[system-requirements]
glibc = "2.28"

@wolfv thx! pixi add --pypi "trimesh[all]" now works for me with the following config:

[system-requirements]
libc = "2.39"

@liblaf I wonder if for you it’s a similar issue and adding glibc 2.28 or higher would fix it.

e.g.

[system-requirements]
glibc = "2.28"

What did you run and what was the outcome?

pixi add --platform osx-arm64 --pypi "meshkernel==4.1"

What error did pixi return?

  × failed to resolve `pypi-dependencies`, due to underlying error
  ╰─▶ The following packages are incompatible
      └─ meshkernel ==4.1 cannot be installed because there are no viable options:
         └─ meshkernel 4.1.0 is excluded because none of the artifacts are compatible with the Python interpreter or glibc version and there are no supported sdists

Can pip install the package?

pixi run pip install "meshkernel==4.1" works on the osx-arm64 platform.

What platform are you on?

This happens on the osx-64 and osx-arm64 platforms (it works on win-64 and linux-64). I expect this to be some mismatching in the name/metadata of/in the .whl files for the macos specific ones.

Did you find a workaround, if so please explain.

Not yet, it would help if the resolver can be more verbose (-vvv has no effect on it) how it matches the platform/interpreter/glibc version and what’s incompatible. Possibly renaming the wheels on the meshkernel side would be enough to fix it.

Encountered in https://github.com/Deltares/Ribasim/pull/1137

Trying to install torch-geometric-temporal with pixi add. It fails as it cannot find torch, which had already been install previously

It does indeed not work. Testing the repro gives me this error:

  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ├─▶ Failed to download and build: torch-scatter==2.1.2
  ├─▶ Failed to build: torch-scatter==2.1.2
  ╰─▶ Build backend failed to determine extra requires with `build_wheel()`:
      --- stdout:
      
      --- stderr:
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in
      get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in
      _get_build_requires
          self.run_setup()
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in
      run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in
      run_setup
          exec(code, locals())
        File "<string>", line 8, in <module>
      ModuleNotFoundError: No module named 'torch'
      ---

In both the uv and rip workflow although import torch in the pixi run python does work for this pixi.toml.

Thanks for the info @jacobbieker

Would be kind-of useful if we can keep the build environments for uv as well, so it’s easier to debug these things.

@roaldarbol The error message is absolutely terrible but if you add:

[system-requirements]
macos = "11.0"

It should work.

Most likely you are missing a system requirement: https://pixi.sh/latest/configuration/#the-system-requirements-table

Most likely macos=12.0

Tensorflow metal on Apple silicon MacOS 14.2.1:

pixi init tf-metal && cd tf-metal
pixi add "python>=3.11" "tensorflow>=2.13" pip
pixi add --pypi tensorflow-metal
> × failed to resolve `pypi-dependencies`, due to underlying error
  ╰─▶ The following packages are incompatible
      └─ tensorflow-metal * cannot be installed because there are no viable options:
         └─ tensorflow-metal 0.1.0 | 0.1.1 | 0.1.2 | 0.2.0 | 0.3.0 | 0.4.0 | 0.5.0 | 0.5.1 | 0.6.0 | 0.7.0 | 0.7.1
      | 0.8.0 | 1.0.0 | 1.0.1 | 1.1.0 is excluded because none of the artifacts are compatible with the Python
      interpreter or glibc version and there are no supported sdists

pixi run pip install tensorflow-metal
> Successfully installed tensorflow-metal-1.1.0

@pablovela5620 so it seems mediapipe 10.9 is a package with an invalid RECORD file, I manually checked it and it’s incorrect.

This is mentioned in the PyPa

During extraction, wheel installers verify all the hashes in RECORD against the file contents. Apart from RECORD and its signatures, installation will fail if any file in the archive is not both mentioned and correctly hashed in RECORD.

Mediapipe has a version.txt that is not mentioned in the RECORD for the 10.9 release.

Which in this case triggers the error, I’m unsure why pip does no do this. But I feel its good to adhere to the standard here.

In any case, mediapipe 10.8 does seem to work, you could use that instead.

Also see: https://github.com/google/mediapipe/issues/5025

@liquidcarbon I cannot reproduce this, neither on OSX or on windows. It’s strange it does not select the .whl for some reason.

@liquidcarbon the duckdb on conda-forge has been continued on python-duckdb which should be available on windows. We’ll keep the example for pypi to test more!

In this pattern the environment definition is fragmented between pixi.toml tasks and pixi.lock. It could work, esp with pinned pip installs in tasks, but is it the intention?

@liquidcarbon This is indeed not what we want as UX but we simply need to develop more to support all PyPI packages. Its a weird bunch of requirements we have to support to be equivalent to pip. So please keep posting non working packages!

1. work around was just to use pip to install using pixi tasks

@pablovela5620 I am also finding this necessary quite often, probably more often than @ruben-arts would like 😃

In this pattern the environment definition is fragmented between pixi.toml tasks and pixi.lock. It could work, esp with pinned pip installs in tasks, but is it the intention?