importlib_metadata: Breaking change in 7e5bae4 (importlib_metadata 5)

I appreciate everyone’s help on getting this through. I know it was painful for some of you, and I thank you for your patience and civil tone engaging with this somewhat tricky transition. I believe the biggest fires are out and the proposed solutions are having the intended effect of enabling projects to move forward.

I’m going to close this issue, but pin it for visibility, and I’ll gladly re-open if there’s more that this project can do to help.

I really appreciate the feedback.

On that note, it might be helpful to have messaging regarding a date or version when breaking changes are coming. That way folks can plan for it, especially for a commonly used package.

That’s useful feedback. I’ve been operating under a less rigorous versioning strategy and I’ve only seen a few examples of projects that have adopted a rigorous strategy for breaking changes (Python recently with the two-version deprecation period, and pip with the breaking changes annually). I’ve found that different breaking changes have different blast radii, so it’s difficult to devise a specific strategy that accommodates the variety of impacts. I try to ascertain the level of impact and use that to devise a timeline for deprecation/removal. Often, I will put a specific date in the code “not to be removed before YYYY-MM-DD” to provide downstream consumers a minimum time. I have found that attempting to say specifically when or in what version a breaking change will occur is folly and often gets missed.

I didn’t put a particular date on this change in part because the deprecation itself was so impactful, I’d expected the major consumers to have already addressed the issue early and the 16 month delay gave sufficient time for systems to upgrade.

I do think it would have helped to put a minimum date on the deprecation warning. I’ll do that for the future and possibly remaining deprecation warnings.

It’s super easy to see “this functionality is deprecated” and treat it as “I won’t use it more in the future” instead of “I will stop using it now”.

That’s an interesting perspective. I’ve never taken that perspective. Whenever I’ve encountered a DeprecationWarning, that says to me that the functionality is slated for removal and could be removed at any time in any future version (unless stated otherwise). Interestingly, the Python documentation doesn’t clarify what the meaning of a DeprecationWarning is.

If some maintainers hold the opinion that DeprecationWarning just means limit additional usage, they will necessarily encounter the breakage every time it happens (even if it’s years in the future). That is, unless no project ever changes any expectation ever.

I could imagine some projects employing a PendingDeprecationWarning to signal what you’re suggesting, that further use be curtailed, but that existing uses are still okay for now… but it still implies that a deprecation is imminent and when that happens, one should take action and adapt.

Per the Python deprecation policy, this behavior is expected to change in Python 3.12 as well, which is why I wanted to get the backport change out sooner, so users would have a chance to adapt where they have more control over the behavior (pinning dependencies, adding backports, etc).

Unfortunately importlib_metadata is used deep within many automated systems (build systems, containers, etc…), so the deprecation warnings are likely unseen by most and ignored by many others.

Good point. If there’s a specific example where the directly-affected project (the project using the interface) did not have visibility to the deprecation warning, I’d like to investigate that, because that’s an assumption I rely on to enact these changes. I know other projects downstream of the direct-consuming projects will be affected, and I rely on the direct consumers to take action when deprecations affect their projects and their users.

My current understanding is that the big projects affected are openstack (stevedore) and flake8, both projects with which I engaged directly at the time of the deprecation (or prior to it) and proposed multiple fixes to address the breakage early. If any project had reached out and requested that the removal be delayed, I certainly would have considered that and likely built it into the approach. Stability and predictability are important, but if a project chooses to ignore the warnings, they do so at their own risk (true of any project).

Breaks flake8 too, which uses .get().

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.14/x64/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 363, in run
    self._run(argv)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 350, in _run
    self.initialize(argv)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 330, in initialize
    self.find_plugins(config_finder)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 153, in find_plugins
    self.check_plugins = plugin_manager.Checkers(local_plugins.extension)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 357, in __init__
    self.namespace, local_plugins=local_plugins
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 238, in __init__
    self._load_entrypoint_plugins()
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 254, in _load_entrypoint_plugins
    eps = importlib_metadata.entry_points().get(self.namespace, ())
AttributeError: 'EntryPoints' object has no attribute 'get'

Ah, I think I figured it out! https://importlib-metadata.readthedocs.io/en/latest/using.html#entry-points The documentation was clear. =)

eps = importlib_metadata.entry_points()
namespace_eps = eps.select(group=self.namespace) if self.namespace in eps.groups else []

I’m pretty sure you can simplify this to:

nampspace_eps = importlib_metadata.entry_points(group=self.namespace)

Because .select() (and also the same parameters to entry_points()) will return an empty EntryPoints if no entry points match for the parameters passed.

I really appreciate that .select() has been around since 3.6. It makes adapting to this change pretty elegant for me.

fwiw, Celery is now tracking this - https://github.com/celery/celery/issues/7783 - which was why i came to the issue in this repo in the first place.

i understand that the change to 5.x was semantically a breaking change. on the other hand, because my project isn’t using poetry (😭) i didn’t even know i was using importlib-metadata so the error took several hours to track down to root cause. just sharing, not suggesting that you need to change anything about your process!

Any idea why an older version of flake8 is ending up with a brand new version of importlib_metadata?

Other things in the dep chain need a newer importlib_metadata, so pip walks back the flake8 versions until it finds flake8 3.9.2 which is the last release that includes importlib_metadata without a version pin. It doesn’t work, of course, but pip can’t know that.

If I manually pin to flake8 >=4.0.0, the first release which includes the importlib_metadata pin, then pip instead walks back the versions of other things to a point where the version of importlib_metadata they need is compatible with the older version flake8 needs. (I’m not sure yet if this is acceptable or not for our usage, but at least it’s something.)

seconded. i don’t have specific details like @inno but i am able to trace this back to upgrading to 5.0.0. the package that triggered the error was celery

for the time being i have downgraded to <5.0.0