uv: uv fails to use extra index url

Please consider dependency confusion attacks: https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

Use of --extra-index-url as they are presently used are a security vulnerability.

PEP 708 is a yet-to-be-implemented approach to improving the security posture.

In short:

• I want to specify multiple indices
• I want uv to use them in that order
• If a package isn’t found in the first index, I want uv to fall back onto the next index
• Other users might want to prevent falling back to the next index for specific packages

pypiserver by default falls back to PyPI.org when it can’t find the specified project within its own distributions. If it finds the package within its own dists, it does not look into PyPI.org. So if I allow it to fall back, and I point uv at it, I get what I wanted: my private, local packages take precedence over packages on PyPI.org, even if more recent versions are on PyPI.org 🎉 Perfs are good 🙂

Tagging latest maintainer @dee-me-tree-or-love in case that’s of interest to them 😄

I’ve quickly read PEP 708, and I definitely support it instead of ordered indexes. So you can discard my previous comments stating “what I want”. With PEP 708 I’ll be able to set private projects to “track” the same ones on PyPI.org. This should give me what I want. pypiserver actually has a “fallback” feature which is possibly already solving my use-case needs (though depending on its implementation could cancel the perfs offered by uv, I’ll see and report back).

I’ll explain my use-case since uv might want to deviate from what pip does (for good reasons) 🙂

My projects follow a sponsorware strategy, where there’s a public version, and a private version with more features. Sponsorships above a certain amount per month grant access to these private repositories on GitHub. To simplify local development, as well as allowing contributors without access to these private versions, I use a local index to store built distributions of these private projects. Details here: https://pawamoy.github.io/pypi-insiders/. It means I can specify some-project in my dependencies, instead of hardcoding git+https://token@github.com/org/private-repo. pip is then able to fetch packages from both PyPI.org and my local index, if configured as such (pip’s config file, env var, cli flag, etc.).

Current situation with pip:

• pip (or other package managers) will search for the highest compatible version of the dependency (some-project above) in both indices (PyPI.org and http://localhost:XXXX).
• Since the private projects I’m developing (or using: I’m not the only one following this sponsorware strategy) use a versioning scheme like major.minor.patch for public versions, and major.minor.patch.pmajor.pminor.ppatch where pmajor.pminor.ppatch is the private version (based on the public one, as a fork), private versions are always “higher” in terms of versioning than the public ones. So pip fetches private versions from the local index, unless there’s an even higher public version on PyPI.org without its private equivalent (for example 2.0.0 on PyPI.org, but only 1.2.3.1.0.0 and no 2.0.0.1.0.0 in the local index).
• This is documented as a limitation: you cannot enforce usage of a private version if there’s a higher public version.

This leads me to this desired situation with uv:

• If uv is capable of enforcing the fetching of distributions from a specific index (when they exist in this index), configurable per user, this will lift the limitation above: I will be able to enforce the use of private versions even if there are higher public versions 👍 Emphasis on per user configuration: I do not want to specify relevant packages, and I do not want to add any configuration in pyproject.toml or other per-project tracked file. A simple prefer-index = "http://localhost:8000/simple/" in ~/.config/uv/conf.toml (totally invented) would be perfect. This prefer-index setting would be no-op if it is not also passed as an extra-index-url. This way it lets me choose whether to fetch packages from my preferred index, or not when I need to test public versions:
  1. no configured extra index: install from main index
  2. configured extra index, no configured preferred index: install most compatible version (depending on solving strategy), so install from one or the other indices (look into both)
  3. configured extra index, configured preferred index: if package exists on preferred index, install most compatible from this index only, otherwise install from main index

Note that in my use-case, there are no security concerns, as there is no concept of internal versus public projects, with the latter being able to shadow the former. Projects are the same in both indices, just with different versions. For use-cases actually involving internal packages, I do understand the need to enforce fetching specific packages from a specific index, and I believe the mentioned prefer-index setting above would solve that too?

Maybe case 2 above shouldn’t be supported at all, and then we would just need to reverse the semantic of extra-index-url, without needing any new config option: give extra-index-url precedence over the main index. If you can keep the order of multiple extra-index-urls, then give them precedence according to this order. But if an extra index is not reachable, or does not contain a package, do not fail and fall back to the next one (maybe not secure enough though, as you never want to fall back for internal packages).

Let me know if anything was unclear!

Thanks for the clear write-up!

Yes! Currently, I expect pip to look into every specified index to satisfy the dependency specification (without order/precedence).

I actually don’t know what the “right” behavior is here. My guess is that we’re “supposed” to look at both indexes (though the order in which indexes are searched is not guaranteed in pip).