scipy: BUG: MacOS universal2 wheels have two gfortran shared libraries, which trips up PyInstaller

Describe your issue.

Starting with release 1.8.0 there seems to be a mismatch of shared libraries. Please see the failure message on this github actions job: https://github.com/danielhrisca/asammdf/runs/5109724435?check_suite_focus=true

Reproducing Code Example

Try to build a stand-alone application using PyInstaller

Error message

Traceback (most recent call last):
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/bin/pyinstaller", line 8, in <module>
      sys.exit(run())
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/__main__.py", line 124, in run
      run_build(pyi_config, spec_file, **vars(args))
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/__main__.py", line 58, in run_build
      PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 793, in main
      build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 725, in build
      exec(code, spec_namespace)
    File "asammdf.spec", line 83, in <module>
      exe = EXE(
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/api.py", line 490, in __init__
      self.pkg = PKG(
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/api.py", line 205, in __init__
      self.__postinit__()
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 155, in __postinit__
      self.assemble()
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/api.py", line 268, in assemble
      fnm = checkCache(
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/building/utils.py", line 361, in checkCache
      osxutils.binary_to_target_arch(cachedfile, target_arch, display_name=fnm)
    File "/Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/PyInstaller/utils/osx.py", line 315, in binary_to_target_arch
      assert target_arch in archs, \
  AssertionError: /Users/runner/work/asammdf/asammdf/.tox/exe/lib/python3.9/site-packages/scipy/linalg/../.dylibs/libgfortran.5.dylib is incompatible with target arch x86_64 (has arch: arm64)!

SciPy/NumPy/Python version information

1.8.0 1.22.2 CPython 3.9.10 on macOS-10.16-x86_64-i386-64bit