setuptools: [BUG] Parsing of __init__.py fails to reach _read_utf8_with_fallback at the end of file

When running Deluge 2.1.1.dev99 (https://github.com/deluge-torrent/deluge/commit/7f3f7f69ee78610e95bea07d99f699e9310c4e08) with Setuptools 70.0.0, an error occurs claiming that _read_utf8_with_fallback is not defined in setuptools’ init.py. By moving the function position to the head of the referred file, after the imports, the program runs normally. This error does not occur on Setuptools 69.x and earlier.

System: linux x64, kernel 6.9.0 Python: 3.11.9

Discussed in https://github.com/pypa/setuptools/discussions/4398

<div type='discussions-op-text'>

Originally posted by autogris May 24, 2024 Hi. I’m trying to debug an error that emerged since upgrading setuptools to 70.0.0. When running Deluge, it gives me the following error:

Traceback (most recent call last):
  File "/usr/bin/deluge-gtk", line 33, in <module>
    sys.exit(load_entry_point('deluge==2.1.1.dev99', 'gui_scripts', 'deluge-gtk')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/__init__.py", line 63, in start
    Gtk().start()
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/__init__.py", line 44, in start
    from .gtkui import GtkUI
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/gtkui.py", line 61, in <module>
    from .pluginmanager import PluginManager
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/pluginmanager.py", line 12, in <module>
    import deluge.pluginmanagerbase
  File "/usr/lib/python3.11/site-packages/deluge/pluginmanagerbase.py", line 15, in <module>
    import pkg_resources
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3282, in <module>
    @_call_aside
     ^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3295, in _initialize_master_working_set
    working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 589, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 926, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 787, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 816, in _resolve_dist
    env = Environment(self.entries)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1014, in __init__
    self.scan(search_path)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1046, in scan
    for dist in find_distributions(item):
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2091, in find_on_path
    yield from factory(fullpath)
               ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2183, in resolve_egg_link
    return next(dist_groups, ())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2179, in <genexpr>
    resolved_paths = (
                     ^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2167, in non_empty_lines
    for line in _read_utf8_with_fallback(path).splitlines():
                ^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_read_utf8_with_fallback' is not defined

I’ve tried to recompile Deluge, reinstall all python modules, to no success. The supposedly undefined function is in fact present at python3.11/site-packages/setuptools/unicode_utils.py . Is this a bug on setuptools side or Deluge? I’m on Python 3.11.9</div>

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 21 (11 by maintainers)

Most upvoted comments

If we try to make pip “more deterministic”, we also don’t see the error:

> docker run --rm -it python:3.11-bookworm /bin/bash
git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
pip install -U setuptools wheel
pip install -e . --use-pep517 --no-build-isolation
mypackage
# => Print me if you can

OR

> docker run --rm -it python:3.11-bookworm /bin/bash
git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
pip install -e . --use-pep517
mypackage
# => Print me if you can

OR

git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools==70.0.0"]
build-backend = "setuptools.build_meta"
EOF
pip install -e .
mypackage
# => Print me if you can

So I suppose this only happens when pip calls python setup.py develop (“legacy”, pre PEP 517), which is deprecated and should not be used 😅.

Thank you, I can also see that this seems to be an “editable” install problem only:

git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
pip install -U setuptools
pip install .
mypackage
# => Print me if you can

Great idea to reproduce it in docker!

I can confirm the following works as a reproduction path in 3.10, 3.11 and 3.12

> docker run --rm -it python:3.11-bookworm /bin/bash
git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
pip install -U setuptools
pip install -e .
mypackage

Could you try again with python:3.10-bookworm? It might be limited to 3.10

Also make sure to explicitly update setuptools to 70.0

Ok, I’ve done some digging and came up with a minimal reproducer: https://github.com/NiklasMM/setuptools-minimal-reproducer

It seems to have something to do with the dependency zope-interface, which we have through twisted and Deluge, which was in the original report, also seems to have.

I’m not quite sure why, but it can be defined as zope-interface or zope.interface. Having it with the dash and then importing iter_entry_points in code that is itself run through a dynamic entry point import such as a console_script seems to trigger the error.

I could only observe this behavior in Python 3.10, not it 3.12. (Others I haven’t tested)

That’s how war I could narrow it down so far. Hope it helps.

Hi @abravalheri . I myself am still trying to understand root cause. I will be out of the office starting on 6/6 and will return on 6/17, so I can revisit this at that time to investigate more and provide feedback if I’m able to easily reproduce it.