cocotb: RHEL8/Python 3.8: cocotb 1.7.1 wheel install fails with "ModuleNotFoundError: No module named 'pygpi'"

    I've uninstalled cocotb, changed Python to 3.9 and installed cocotb again

pip3 uninstall cocotb
dnf module install python39
alternatives --config python3
pip3 install cocotb --user --verbose

Wheel install:

Collecting cocotb
  Created temporary directory: /tmp/pip-unpack-otqwhxbk
  Looking up "https://files.pythonhosted.org/packages/77/90/5a819a2eaa92fbdd53e5031b86ef49d610cb8283cea2efa5e22864ce7b14/cocotb-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/77/90/5a819a2eaa92fbdd53e5031b86ef49d610cb8283cea2efa5e22864ce7b14/cocotb-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl HTTP/1.1" 200 3695683
  Downloading cocotb-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.7 MB)
     |████████████████████████████████| 3.7 MB 6.4 MB/s eta 0:00:01  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/77/90/5a819a2eaa92fbdd53e5031b86ef49d610cb8283cea2efa5e22864ce7b14/cocotb-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
  Caching due to etag
     |████████████████████████████████| 3.7 MB 6.4 MB/s
  Added cocotb from https://files.pythonhosted.org/packages/77/90/5a819a2eaa92fbdd53e5031b86ef49d610cb8283cea2efa5e22864ce7b14/cocotb-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=6eded67546534bb51dd7b2f95ea0e2f7ac90579a7fad72a170fc796d05ac775a to build tracker '/tmp/pip-req-tracker-1rsv85er'
  Removed cocotb from https://files.pythonhosted.org/packages/77/90/5a819a2eaa92fbdd53e5031b86ef49d610cb8283cea2efa5e22864ce7b14/cocotb-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=6eded67546534bb51dd7b2f95ea0e2f7ac90579a7fad72a170fc796d05ac775a from build tracker '/tmp/pip-req-tracker-1rsv8
5er'
Installing collected packages: cocotb

However now the simulation fails with:

asim   +access +w -interceptcoutput -O2 -dbg -pli /home/adrian/.local/lib/python3.9/site-packages/cocotb/libs/libcocotbvpi_aldec.so endian_swapper_mixed
# ELBREAD: Elaboration process.
# ELBREAD: Elaboration time 0.0 [s].
# KERNEL: Main thread initiated.
# KERNEL: Kernel process initialization phase.
# ELAB2: Elaboration final pass...
# KERNEL: PLI/VHPI kernel's engine initialization done.
# PLI: Loading library '/home/adrian/.local/lib/python3.9/site-packages/cocotb/libs/libcocotbvpi_aldec.so'
# COUT:      -.--ns INFO     gpi                                ..mbed/gpi_embed.cpp:76   in set_program_name_in_venv        Did not detect Python virtual environment. Using system-wide Python interpreter
# COUT:      -.--ns INFO     gpi                                ../gpi/GpiCommon.cpp:101  in gpi_print_registered_impl       VPI registered
# COUT:      -.--ns INFO     gpi                                ../gpi/GpiCommon.cpp:101  in gpi_print_registered_impl       VHPI registered
# PLI: Loading library '/opt/aldec/rivierapro/bin/libsystf.so'
# ELAB2: Create instances ...
# KERNEL: Time resolution set to 1ps.
# ELAB2: Create instances complete.
# ELAB2: Elaboration final pass complete - time: 0.0 [s].
# KERNEL: Kernel process initialization done.
# Allocation: Simulator allocated 6642 kB (elbread=427 elab2=6064 kernel=150 sdf=0)
# KERNEL: ASDB file was created in location /afs/cern.ch/work/a/adbyszuk/praca/temp/cocotb/examples/mixed_language/tests/dataset.asdb
run -all
# COUT: ModuleNotFoundError: No module named 'pygpi'

_Originally posted by @abyszuk in https://github.com/cocotb/cocotb/issues/3078#issuecomment-1249218313_

@imphil I’ve tried debugging this ModuleNotFoundError and it could be a problem in cocotb-config, but that’s just my guess.

adrian@pcte247806:tests$ cocotb-config --libpython
/usr/lib64/libpython3.so

However, having multiple python versions installed, there are also multiple libpython binaries:

adrian@pcte247806:/$ ls -l /usr/lib64/libpython3.so 
-rwxr-xr-x 1 root root 6888 Sep 10  2021 /usr/lib64/libpython3.so

adrian@pcte247806:/$ ls -l /lib64/libpython3.9.so.1.0 
-rwxr-xr-x 1 root root 3918968 Aug 25  2021 /lib64/libpython3.9.so.1.0

adrian@pcte247806:/$ ls -l /lib64/libpython3.6m.so.1.0 
-rwxr-xr-x 1 root root 3289680 Sep 10  2021 /lib64/libpython3.6m.so.1.0

I’ve managed to make some progress by manually modifying Makefile.riviera and adding LIBPYTHON_LOC=/lib64/libpython3.9.so.1.0