pip: AIX: `pip download package` does not work since 21.2

Description

Recently got started with needing to update a number of packages - and ran into a problem that I could not download ansible-base (so I suspect it is an issue with any packages that are not pure Python).

So, rolled back pip to a much older version (20.2.4) and all was okay.

In increments I updated pip and 21.1.3 was the last version that worked as expected (as far as download is concerned, have not tried anything else).

Where I think the regression occurred

aixtools@x064:[/data/prj/python/git/pip]git diff 21.1.3 21.2.1 -- ./src/pip/_internal/utils/unpacking.py
diff --git a/src/pip/_internal/utils/unpacking.py b/src/pip/_internal/utils/unpacking.py
index 44ac47535..bffb3cd65 100644
--- a/src/pip/_internal/utils/unpacking.py
+++ b/src/pip/_internal/utils/unpacking.py
@@ -178,7 +178,7 @@ def untar_file(filename, location):
             filename,
         )
         mode = "r:*"
-    tar = tarfile.open(filename, mode)
+    tar = tarfile.open(filename, mode, encoding="utf-8")
     try:
         leading = has_leading_dir([member.name for member in tar.getmembers()])
         for member in tar.getmembers():

Expected behavior

• Note: I get the same problem as above when trying to download ansible-base==2.10.16, but the ansible I have been using for two years is based on 2.10.1 - so I tried that version again.
• I saw you are withdrawing support for py36 - so, please note, this is not a request for py36 support. I first saw this on Python3-9 which is what I wanted to update. py36 is only being used because that is known to be working - and I was looking for when the regression appeared.

(py360) aixtools@x064:[/home/aixtools/download/py360]pip download ansible-base==2.10.1
Collecting ansible-base==2.10.1
  Using cached ansible-base-2.10.1.tar.gz (6.0 MB)
Collecting jinja2
  Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting PyYAML
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting cryptography
  Using cached cryptography-36.0.1.tar.gz (572 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting packaging
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     |################################| 40 kB 44 kB/s
Collecting cffi>=1.12
  Using cached cffi-1.15.0.tar.gz (484 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1.tar.gz (18 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)
     |################################| 98 kB 286 kB/s
Saved ./ansible-base-2.10.1.tar.gz
Saved ./cryptography-36.0.1.tar.gz
Saved ./cffi-1.15.0.tar.gz
Saved ./Jinja2-3.0.3-py3-none-any.whl
Saved ./MarkupSafe-2.0.1.tar.gz
Saved ./packaging-21.3-py3-none-any.whl
Saved ./pyparsing-3.0.7-py3-none-any.whl
Saved ./pycparser-2.21-py2.py3-none-any.whl
Saved ./PyYAML-6.0.tar.gz

pip version

21.3.1, 21.2.4, 21.2

Python version

3.6, 3.9

OS

AIX

How to Reproduce

1. System Admin Installs Python3-9 (or 3-6) including virtualenv
2. Following steps as regular (no root powers)
3. virtualenv py360
4. mkdir -p downloads/py360
5. . py360/bin/activate
6. cd downloads/py360
7. pip3 download ansible-base==2.10.1

Output

- Example:

(py360) aixtools@x064:[/home/aixtools/download/py360]pip install pip --upgrade
Requirement already satisfied: pip in /home/aixtools/py360/lib/python3.6/site-packages (21.1.3)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed pip-21.3.1
(py360) aixtools@x064:[/home/aixtools/download/py360]pip download ansible-base==2.10.1
Collecting ansible-base==2.10.1
  File was already downloaded /home/aixtools/download/py360/ansible-base-2.10.1.tar.gz
ERROR: Exception:
Traceback (most recent call last):
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 128, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 93, in resolve
    collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 349, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    version=version,
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 287, in __init__
    version=version,
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 528, in _prepare_linked_requirement
    link, req.source_dir, self._download, self.download_dir, hashes
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 223, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 247, in unpack_file
    untar_file(filename, location)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 218, in untar_file
    with open(path, "wb") as destfp:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 138-141: ordinal not in range(256)

Code of Conduct

• I agree to follow the PSF Code of Conduct.