conda-forge.github.io: Many noarch packages that are installable but broken on early Pythons

Your question:

As Python is release more often and the use of requires-python/python-requires metadata is getting adopted, we are creating modern packages that should not be installable on earlier version of Python due to the use of >= in both host and run.

Many issues like https://github.com/conda-forge/urllib3-feedstock/issues/84 are cropping up and the bot does not use grayskull to update Python, only the other dependencies. Here is what we discussed in today’s meeting.

Pin the minimum Python in host and test to ensure a recipe will break when building if the min Python version was updated.

requirements:
  host:
    - python 3.8.*
  run:
    # option 1
    - python >=3.8
    # option 2
    - {{ pin_compatible("python", max_pin=None) }}
test:
  requires:
    - python 3.8.*

or we could improve the bot to use grayskull to fully regenerate the recipe, or both togehter.

The former seems to have a clear path for implementation: 1 linter update, migration to modify the recipes.

Note that this will cause a lot of churn! There are many noarch recipes.

cc: @isuruf, @jaimergp, @jakirkham, and @beckermr who are probably the key people who may be interested in this.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 32 (32 by maintainers)

Most upvoted comments

OK so I think we’re at consensus. We’ll do the usual, announcement + migrator + lint + hint thing here.

@h-vetinari I’m at peace with that. I will note that my original comment was in the context of a linter message blindly telling people to specify “> 3.10” rather than a more formal change to conda-forge’s infrastructure and pinning around noarch.

So I think our consensus is to have

host:
  - python {{ minpy }}
run:
  - python >={{ minpy }}

test:
 requires:
    - python ={{ minpy }}

We’ll need a better name for minpy as suggested above.

(edit: changed to pin_compatible since that will respect the patch version too.) (edit: changed back to >= since we don’t want to pin against patch version.)

To me it seems the maintainer(s) should be mostly maintaining only the meta.yaml file, which means they’d increase the min python version when it changes upstream

There’s intentionally a global default python_min for all of conda-forge (set in the pinning repo). This is what feedstocks will get by default if they don’t manually override python_min in recipe/conda_build_config.yaml or as a Jinja variable in meta.yaml.

The vast majority of packages will just get the last non-EOL python (now 3.9, because although the 3.8 artefacts are still available, we’re not maintaining 3.8 anymore and bitrot sets in quickly). For the small minority of package that have a higher lower bound, (or more rarely, if you explicitly want to stay compatible with EOL python versions), they can override this per feedstock.

In short: feedstock maintainers don’t have to care about the value of python_min, except when you explicitly want to diverge from the global default.

I would be extremely confused if I went to install an upstream package that says it supports 3.6 into this environment, and the conda solver gave me either an older version or failed completely to resolve the environment.

This is IMO a much lesser evil than shipping broken packages because we get the bounds wrong the other way without noticing.

To me, conda-forge definitely has the right to make choices here, like “we support CPython versions for 5 years, after that it’s over for all new builds”, even if that ends up tightening the upstream package requirements.

You’d still get the old builds of course if you want to use a py36 environment today. And there’d be an escape hatch too, in that the feedstock can choose to override min_py locally to a lower version than the pinning. That to me seems plenty accomodative for way-EOL python versions.

@ocefpaf If we parameterize the bound in the pinnings, it should be easy enough to roll back globally via a custom pinnings file for your delay, right?

I was guessing on what Ryan meant by projecting an old requirement that I no longer have, but yes. I could use a custom pinning in that case. As it turns out, it was easier for everyone at the day-job to just keep updated with Python instead for many other reasons beyond conda-forge.

BTW, I am +1 to https://github.com/conda-forge/conda-forge.github.io/issues/2210#issuecomment-2315570288

Yeah people can add a bound instead of overriding (say minpy=3.9):

host:
 - python >={{ minpy }}
 - python >=3.11
run:
 - python >={{ minpy }}
 - python >=3.11

However, more to the point, I don’t think people should have to manually change their recipes when we drop a python version. That leaves the bot or explicit bounds in pinnings.

The bot can only take us so far in updates due to inherent limits in the ability to parse and understand a recipe (i.e., When we find python >=XYZ in host for noarch, it is probably the right thing to change, but I am sure we will hit edge cases.).

Having an explicit bound in the pinnings ensures no manual updates, specifies the intent precisely, and ensures that we can change it globally. Folks who opt-out of a global pinning have always been on their own in conda-forge, and this case is no different.

Isn’t it more of a linter change? We nag users saying “add lower constraints in your noarch python!” but we could change the message and say instead “Use the =lower.bound in host and test, and >=lower.bound in run, thankssss”?