scipy: "ModuleNotFoundError: No module named 'scipy'" CI Test Failures (in master)?

Reproducing code example:

See CI of gh-13441, gh-13463

Error message:

Four (separate) failures occurring in both of these PRs. I’m not sure if the failures are related, but I noticed them at the same time, so I’m putting them all here for now.

In Windows builds on Azure:

==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : 2.180.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\ca35efdd-7955-4298-a881-94018e1affc9.ps1'"
Traceback (most recent call last):
  File "runtests.py", line 565, in <module>
    main(argv=sys.argv[1:])
  File "runtests.py", line 291, in main
    __import__(PROJECT_MODULE)
ModuleNotFoundError: No module named 'scipy'
##[error]PowerShell exited with code '1'.
Finishing: Run SciPy Test Suite

In refguide_asv_check and Linux Tests we’re seeing directed_hausdorff failures:

     _________________________ TestHausdorff.test_symmetry __________________________
[gw0] linux -- Python 3.8.7 /opt/hostedtoolcache/Python/3.8.7/x64/bin/python
/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/tests/test_hausdorff.py:37: in test_symmetry
    forward = directed_hausdorff(self.path_1, self.path_2)[0]
        self       = <scipy.spatial.tests.test_hausdorff.TestHausdorff object at 0x7f28fbea4970>
/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/distance.py:461: in directed_hausdorff
    result = _hausdorff.directed_hausdorff(u, v, seed)
        seed       = 0
        u          = array([[ 0.35923135,  0.93324854,  0.        ],
       [-0.71983479, -0.69414542,  0.        ],
       [-0.92442618,  ...0695034,  0.        ],
       [ 0.93629495,  0.3512147 ,  0.        ],
       [-0.48493437, -0.87455055,  0.        ]])
        v          = array([[ 1.18546344,  3.07972019,  0.        ],
       [-1.43966959, -1.38829085,  0.        ],
       [-1.84885236,  ...1390068,  0.        ],
       [ 1.8725899 ,  0.7024294 ,  0.        ],
       [-0.96986874, -1.74910109,  0.        ]])
_hausdorff.pyx:37: in scipy.spatial._hausdorff.directed_hausdorff
    ???
        __all__    = ['directed_hausdorff']
        __builtins__ = <builtins>
        __doc__    = '\nDirected Hausdorff Code\n\n.. versionadded:: 0.19.0\n\n'
        __file__   = '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/_hausdorff.cpython-38-x86_64-linux-gnu.so'
        __loader__ = <_frozen_importlib_external.ExtensionFileLoader object at 0x7f2939a17490>
        __name__   = 'scipy.spatial._hausdorff'
        __package__ = 'scipy.spatial'
        __pyx_unpickle_Enum = <built-in function __pyx_unpickle_Enum>
        __spec__   = ModuleSpec(name='scipy.spatial._hausdorff', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f2939a.../hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/_hausdorff.cpython-38-x86_64-linux-gnu.so')
        __test__   = {}
        directed_hausdorff = <built-in function directed_hausdorff>
        np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/__init__.py'>
mtrand.pyx:4478: in numpy.random.mtrand.RandomState.shuffle
 E   UserWarning: `x` isn't a recognized object; `shuffle` is not guaranteed to behave correctly. E.g., non-numpy array/tensor objects with view semantics may contain duplicates after shuffling.
        MutableSequence = <class 'collections.abc.MutableSequence'>
        RandomState = <class 'numpy.random.mtrand.RandomState'>
        _MT19937   = <class 'numpy.random._mt19937.MT19937'>
        __all__    = ['beta', 'binomial', 'bytes', 'chisquare', 'choice', 'dirichlet', ...]
        __builtins__ = <builtins>
        __doc__    = None
        __file__   = '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so'
        __loader__ = <_frozen_importlib_external.ExtensionFileLoader object at 0x7f2942eb7d60>
        __name__   = 'numpy.random.mtrand'
        __package__ = 'numpy.random'
        __spec__   = ModuleSpec(name='numpy.random.mtrand', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f2942eb7d60...'/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so')
        __test__   = {'RandomState.binomial (line 3262)': '\n        binomial(n, p, size=None)\n\n        Draw samples from a binomial dist...      array(['pooh', 'pooh', 'pooh', 'Christopher', 'piglet'], # random\n              dtype='<U11')\n\n        ", ...}
        _rand      = RandomState(MT19937) at 0x7F2943136E40
        beta       = <built-in method beta of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        binomial   = <built-in method binomial of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        bytes      = <built-in method bytes of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        chisquare  = <built-in method chisquare of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        choice     = <built-in method choice of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        dirichlet  = <built-in method dirichlet of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        exponential = <built-in method exponential of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        f          = <built-in method f of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        gamma      = <built-in method gamma of numpy.random.mtrand.RandomState object at 0x7f2943136e40>

        geometric  = <built-in method geometric of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        get_state  = <built-in method get_state of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        gumbel     = <built-in method gumbel of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        hypergeometric = <built-in method hypergeometric of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        laplace    = <built-in method laplace of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        logistic   = <built-in method logistic of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        lognormal  = <built-in method lognormal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        logseries  = <built-in method logseries of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        multinomial = <built-in method multinomial of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        multivariate_normal = <built-in method multivariate_normal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        negative_binomial = <built-in method negative_binomial of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        noncentral_chisquare = <built-in method noncentral_chisquare of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        noncentral_f = <built-in method noncentral_f of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        normal     = <built-in method normal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/__init__.py'>
        operator   = <module 'operator' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/operator.py'>
        pareto     = <built-in method pareto of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        permutation = <built-in method permutation of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        poisson    = <built-in method poisson of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        power      = <built-in method power of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        rand       = <built-in method rand of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        randint    = <built-in method randint of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        randn      = <built-in method randn of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        random     = <built-in method random of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        random_integers = <built-in method random_integers of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        random_sample = <built-in method random_sample of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        ranf       = <built-in function ranf>
        rayleigh   = <built-in method rayleigh of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        seed       = <built-in method seed of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        set_state  = <built-in method set_state of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        shuffle    = <built-in method shuffle of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_cauchy = <built-in method standard_cauchy of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_exponential = <built-in method standard_exponential of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_gamma = <built-in method standard_gamma of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_normal = <built-in method standard_normal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_t = <built-in method standard_t of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        triangular = <built-in method triangular of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        uniform    = <built-in method uniform of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        vonmises   = <built-in method vonmises of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        wald       = <built-in method wald of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        warnings   = <module 'warnings' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/warnings.py'>
        weibull    = <built-in method weibull of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        zipf       = <built-in method zipf of numpy.random.mtrand.RandomState object at 0x7f2943136e40>

On AppVeyor:

C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgtc.o C:\Use
rs\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgtf.o 
C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgts.o -LC:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\lib\gcc\x86_64-w64-mingw32\6.3.0 
-Lc:\python37-x64\libs -Lc:\python37-x64\PCbuild\amd64 -Lbuild\temp.win-amd64-3.7 -o build\temp.win-amd64-3.7\Release\.libs\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.dll 
build\temp.win-amd64-3.7\Release\.libs\libopenblas.3HBPCJB5BPQGKWVZAVEBXNNJ2Q2G3TUP.gfortran-win_amd64.dll -Wl,--allow-multiple-definition 
-Wl,--output-def,build\temp.win-amd64-3.7\Release\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy
Failed to build scipy
ERROR: Failed to build one or more wheels
Exception information:
Traceback (most recent call last):
  File "c:\python37-x64\lib\site-packages\pip\_internal\cli\base_command.py", line 189, in _main
    status = self.run(options, args)
  File "c:\python37-x64\lib\site-packages\pip\_internal\cli\req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "c:\python37-x64\lib\site-packages\pip\_internal\commands\wheel.py", line 191, in run
    "Failed to build one or more wheels"
pip._internal.exceptions.CommandError: Failed to build one or more wheels

I am also seeing this in macOS tests, but that might be unrelated, as I’ve seen it intermittently before today:

gcc: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/source.c
gcc /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/source.o -L/usr/local/lib -lopenblas -o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/a.out
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /usr/local/include/python3.9 is invalid.
  return self.get_paths(self.section, key)
blas_opt_info:
blas_mkl_info:
  libraries mkl_rt not found in ['/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_info:
C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/opt/sqlite/include -I/usr/local/opt/sqlite/include

creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b
compile options: '-c'
gcc: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/source.c
gcc /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/source.o -L/usr/local/lib -lopenblas -o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/a.out
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

blas_info:
Traceback (most recent call last):
  File "/Users/runner/work/scipy/scipy/setup.py", line 605, in <module>
    setup_package()
  File "/Users/runner/work/scipy/scipy/setup.py", line 601, in setup_package
    setup(**metadata)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/core.py", line 135, in setup
    config = configuration()
  File "/Users/runner/work/scipy/scipy/setup.py", line 524, in configuration
    config.add_subpackage('scipy')
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/setup.py", line 18, in configuration
    config.add_subpackage('optimize')
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/optimize/setup.py", line 102, in configuration
    ext = pythran.dist.PythranExtension(
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/pythran/dist.py", line 131, in __init__
    cfg_ext = cfg.make_extension(python=True, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/pythran/config.py", line 220, in make_extension
    numpy_blas = numpy_sys.get_info(user_blas)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 584, in get_info
    return cl().get_info(notfound_action)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 844, in get_info
    self.calc_info()
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 2040, in calc_info
    info = self.check_libs(lib_dirs, blas_libs, [])
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 986, in check_libs
    info = self._check_libs(lib_dirs, libs, opt_libs, [ext])
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 1057, in _check_libs
    found_dirs, found_libs = self._find_libs(lib_dirs, libs, exts)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 1041, in _find_libs
    found_lib = self._find_lib(lib_dir, lib, exts)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 1022, in _find_lib
    raise RuntimeError(_accel_msg.format(filename=p[0]))
RuntimeError: Found /usr/lib/libblas.dylib, but that file is a symbolic link to the MacOS Accelerate framework, which is not supported by NumPy. You must configure the build to use a different optimized library, or disable the use of optimized BLAS and LAPACK by setting the environment variables NPY_BLAS_ORDER="" and NPY_LAPACK_ORDER="" before building NumPy.

Build failed! (0:01:47.617718 elapsed)
Error: Process completed with exit code 1.