scipy: Test failures for `gges` and `qz` for float32 input in macOS CI

These started showing up consistently over the last day (example log). Only in the macOS tests (meson) job. That job uses conda-forge, the root cause is very likely the new build (0.23.1 build _3) of OpenBLAS pushed to https://anaconda.org/conda-forge/openblas about 16 hours ago.

___________________________ test_gges_tgexc[float32] ___________________________
[gw0] darwin -- Python 3.10.6 /Users/runner/miniconda3/envs/scipy-dev/bin/python
scipy/linalg/tests/test_lapack.py:3087: in test_gges_tgexc
    assert_allclose(q @ s @ z.conj().T, a, rtol=0, atol=atol)
E   AssertionError: 
E   Not equal to tolerance rtol=0, atol=1.19209e-05
E   
E   Mismatched elements: 100 / 100 (100%)
E   Max absolute difference: 0.5596407
E   Max relative difference: 45.19812
E    x: array([[ 0.269565,  0.510163,  0.307816,  0.720287,  0.713599,  0.25576 ,
E            0.131274,  0.843207,  0.835774,  1.005878],
E          [ 0.364995, -0.002645,  0.148027,  0.409446,  0.200136,  0.596843,...
E    y: array([[0.191519, 0.622109, 0.437728, 0.785359, 0.779976, 0.272593,
E           0.276464, 0.801872, 0.958139, 0.875933],
E          [0.357817, 0.500995, 0.683463, 0.712702, 0.370251, 0.561196,...
        a          = array([[0.19151945, 0.62210876, 0.43772775, 0.7853586 , 0.77997583,
        0.2725926 , 0.27646425, 0.8018722 , 0.9581...43 , 0.9514288 , 0.48035917,
        0.50255954, 0.53687817, 0.81920207, 0.05711564, 0.66942173]],
      dtype=float32)
        atol       = 1.1920928955078125e-05
        b          = array([[0.7671166 , 0.70811534, 0.7968672 , 0.55776083, 0.9658365 ,
        0.1471569 , 0.029647  , 0.59389347, 0.1140...526, 0.7370865 , 0.12702939,
        0.3696499 , 0.604334  , 0.10310444, 0.8023742 , 0.94555324]],
      dtype=float32)
        d1         = -10.973611
        d2         = 0.65421426
        dtype      = <class 'numpy.float32'>
        gges       = <fortran object>
        n          = 10
        q          = array([[ 0.36440596,  0.14539199, -0.48573047,  0.18267933, -0.10118745,
         0.15227516, -0.48237514,  0.06109872....19003534, -0.15249437,
        -0.1685543 , -0.12377075, -0.35968414, -0.3202702 , -0.10042791]],
      dtype=float32)
        result     = (array([[-1.00654995e+00, -1.57352880e-01, -7.44751692e-02,
        -1.94980763e-03,  1.06006734e-01,  7.20489264e-01,... 0.36549678, 0.9152956 ,
       1.0120404 , 0.9464567 , 0.94311225, 0.        , 0.9372638 ],
      dtype=float32), ...)
        s          = array([[-1.00654995e+00, -1.57352880e-01, -7.44751692e-02,
        -1.94980763e-03,  1.06006734e-01,  7.20489264e-01,
...,  0.00000000e+00,
         0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
         3.35054606e-01]], dtype=float32)
        t          = array([[ 0.09172459,  0.18612614, -0.10655626, -0.10534273, -0.5556871 ,
         0.8352932 , -0.11271314,  0.13819972....        ,  0.        ,
         0.        ,  0.        ,  0.        ,  0.        ,  0.9372638 ]],
      dtype=float32)
        tgexc      = <fortran object>
        z          = array([[ 0.45143077,  0.07711446,  0.04053342, -0.06187871, -0.2751032 ,
        -0.56598544, -0.22434531, -0.289968  ....21228388, -0.15313199,
         0.13994187, -0.6340214 ,  0.16412702,  0.56076413,  0.37706485]],
      dtype=float32)
___________________________ test_gges_tgsen[float32] ___________________________
[gw0] darwin -- Python 3.10.6 /Users/runner/miniconda3/envs/scipy-dev/bin/python
scipy/linalg/tests/test_lapack.py:3257: in test_gges_tgsen
    assert_allclose(q @ s @ z.conj().T, a, rtol=0, atol=atol)
E   AssertionError: 
E   Not equal to tolerance rtol=0, atol=1.19209e-05
E   
E   Mismatched elements: 100 / 100 (100%)
E   Max absolute difference: 0.5596407
E   Max relative difference: 45.19812
E    x: array([[ 0.269565,  0.510163,  0.307816,  0.720287,  0.713599,  0.25576 ,
E            0.131274,  0.843207,  0.835774,  1.005878],
E          [ 0.364995, -0.002645,  0.148027,  0.409446,  0.200136,  0.596843,...
E    y: array([[0.191519, 0.622109, 0.437728, 0.785359, 0.779976, 0.272593,
E           0.276464, 0.801872, 0.958139, 0.875933],
E          [0.357817, 0.500995, 0.683463, 0.712702, 0.370251, 0.561196,...
        a          = array([[0.19151945, 0.62210876, 0.43772775, 0.7853586 , 0.77997583,
        0.2725926 , 0.27646425, 0.8018722 , 0.9581...43 , 0.9514288 , 0.48035917,
        0.50255954, 0.53687817, 0.81920207, 0.05711564, 0.66942173]],
      dtype=float32)
        atol       = 1.1920928955078125e-05
        b          = array([[0.7671166 , 0.70811534, 0.7968672 , 0.55776083, 0.9658365 ,
        0.1471569 , 0.029647  , 0.59389347, 0.1140...526, 0.7370865 , 0.12702939,
        0.3696499 , 0.604334  , 0.10310444, 0.8023742 , 0.94555324]],
      dtype=float32)
        d1         = -10.973611
        d2         = 0.65421426
        dtype      = <class 'numpy.float32'>
        gges       = <fortran object>
        n          = 10
        q          = array([[ 0.36440596,  0.14539199, -0.48573047,  0.18267933, -0.10118745,
         0.15227516, -0.48237514,  0.06109872....19003534, -0.15249437,
        -0.1685543 , -0.12377075, -0.35968414, -0.3202702 , -0.10042791]],
      dtype=float32)
        result     = (array([[-1.00654995e+00, -1.57352880e-01, -7.44751692e-02,
        -1.94980763e-03,  1.06006734e-01,  7.20489264e-01,... 0.36549678, 0.9152956 ,
       1.0120404 , 0.9464567 , 0.94311225, 0.        , 0.9372638 ],
      dtype=float32), ...)
        s          = array([[-1.00654995e+00, -1.57352880e-01, -7.44751692e-02,
        -1.94980763e-03,  1.06006734e-01,  7.20489264e-01,
...,  0.00000000e+00,
         0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
         3.35054606e-01]], dtype=float32)
        t          = array([[ 0.09172459,  0.18612614, -0.10655626, -0.10534273, -0.5556871 ,
         0.8352932 , -0.11271314,  0.13819972....        ,  0.        ,
         0.        ,  0.        ,  0.        ,  0.        ,  0.9372638 ]],
      dtype=float32)
        tgsen      = <fortran object>
        tgsen_lwork = <fortran object>
        z          = array([[ 0.45143077,  0.07711446,  0.04053342, -0.06187871, -0.2751032 ,
        -0.56598544, -0.22434531, -0.289968  ....21228388, -0.15313199,
         0.13994187, -0.6340214 ,  0.16412702,  0.56076413,  0.37706485]],
      dtype=float32)
____________________________ TestQZ.test_qz_single _____________________________
[gw1] darwin -- Python 3.10.6 /Users/runner/miniconda3/envs/scipy-dev/bin/python
scipy/linalg/tests/test_decomp.py:2039: in test_qz_single
    assert_array_almost_equal(Q @ AA @ Z.T, A, decimal=5)
E   AssertionError: 
E   Arrays are not almost equal to 5 decimals
E   
E   Mismatched elements: 25 / 25 (100%)
E   Max absolute difference: 1.9282217
E   Max relative difference: 12.652091
E    x: array([[ 0.25942,  0.42247,  0.29118,  1.3622 ,  0.07058],
E          [ 1.54704,  1.04254,  0.65235, -1.17932,  1.4969 ],
E          [ 0.37006,  0.76962, -0.09774,  1.07189, -0.13269],...
E    y: array([[0.92962, 0.31638, 0.18392, 0.20456, 0.56773],
E          [0.59554, 0.96451, 0.65318, 0.74891, 0.65357],
E          [0.74771, 0.96131, 0.00839, 0.10644, 0.2987 ],...
        A          = array([[0.9296161 , 0.31637555, 0.18391882, 0.20456028, 0.567725  ],
       [0.5955447 , 0.9645145 , 0.6531771 , 0.748...93, 0.9646476 , 0.7236853 ],
       [0.6424753 , 0.7174536 , 0.467599  , 0.32558468, 0.4396446 ]],
      dtype=float32)
        AA         = array([[ 0.39677864, -0.87905234,  0.31681818,  0.25420108, -0.31528485],
       [ 0.        ,  2.6049669 ,  0.5027338...10585941, -0.06299087],
       [ 0.        ,  0.        ,  0.        ,  0.        , -0.35764655]],
      dtype=float32)
        B          = array([[0.72968906, 0.99401456, 0.6768737 , 0.7908225 , 0.17091426],
       [0.02684928, 0.8003702 , 0.9037225 , 0.024...4 , 0.09596852, 0.21895005],
       [0.25871906, 0.46810576, 0.4593732 , 0.7095098 , 0.178053  ]],
      dtype=float32)
        BB         = array([[ 0.0964531 , -0.7481051 ,  0.0091472 , -0.6136254 ,  0.07135387],
       [ 0.        ,  2.5299823 ,  0.       ...        ,  0.01339715],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.80180556]],
      dtype=float32)
        Q          = array([[-0.5236176 ,  0.43169242,  0.54998064, -0.28841138, -0.39218202],
       [ 0.10638957,  0.3615428 , -0.7044734...05049007,  0.65567476],
       [-0.04425064,  0.36560142, -0.14396492,  0.8444302 , -0.36137164]],
      dtype=float32)
        Z          = array([[-4.1415977e-01,  4.8910376e-01,  6.0941523e-01,  2.2133094e-01,
         4.1094345e-01],
       [ 8.1124866e-0...e-01],
       [-2.1641581e-01,  2.0534211e-01, -2.5105488e-01, -8.4537733e-01,
         3.6511230e-01]], dtype=float32)
        n          = 5
        self       = <scipy.linalg.tests.test_decomp.TestQZ object at 0x12de02ad0>
=========================== short test summary info ============================
FAILED scipy/linalg/tests/test_lapack.py::test_gges_tgexc[float32] - Assertio...
FAILED scipy/linalg/tests/test_lapack.py::test_gges_tgsen[float32] - Assertio...
FAILED scipy/linalg/tests/test_decomp.py::TestQZ::test_qz_single - AssertionE...

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 49 (29 by maintainers)

Commits related to this issue

CI: pin OpenBLAS to specific build in macOS job to avoid gges issue xref gh-16949. This doesn't fix the root cause of the problem, but stops the CI job from failing. [skip azp] [skip circle] — committed to rgommers/scipy by rgommers 2 years ago
CI: pin OpenBLAS to specific build in macOS job to avoid gges issue xref gh-16949. This doesn't fix the root cause of the problem, but stops the CI job from failing. [skip azp] [skip circle] — committed to rgommers/scipy by rgommers 2 years ago
TST: mark float32 `gges` and `qz` tests as knownfail See gh-16949 — committed to rgommers/scipy by rgommers 2 years ago
TST: mark float32 `gges` and `qz` tests as knownfail See gh-16949 — committed to rgommers/scipy by rgommers 2 years ago
TST: mark float32 gges and qz tests as knownfail See gh-16949 — committed to tylerjereddy/scipy by rgommers 2 years ago
TST: mark float32 gges and qz tests as knownfail See gh-16949 — committed to tylerjereddy/scipy by rgommers 2 years ago
TST: linalg: undo xfails of QZ and CARE cf gh-16926, gh-16949 — committed to ev-br/scipy by ev-br 3 months ago
TST: linalg: undo xfails of QZ and CARE cf gh-16926, gh-16949 — committed to ev-br/scipy by ev-br 3 months ago

Most upvoted comments

Note that qz is implemented in Python and uses gges internally, so there is only one problem here it looks like gges with float32 input.

rgommers on Sep 2, 2022

Yes, and/or get conda-forge to rebuild their openblas package with -fno-tree-vectorize set for gfortran.

martin-frbg on Feb 8, 2023

The issue in CI was fixed by xfail-ing the relevant tests in gh-17057. The sgges issue needs reassessing before the next release. Things seem to work with OpenBLAS 0.3.18, but are broken with 0.3.20/21. As long as we continue to ship 0.3.18 in our own wheels we should be good there - but that doesn’t help conda-forge, Debian et al.

rgommers on Sep 20, 2022

I am going to upload all the artifacts to my personal channel. I can then test them in a CI (I no longer have access to osx-64 machines easily at home). Done: https://anaconda.org/ngam/openblas/files (-c ngam)

ngam on Sep 3, 2022

Depends on how you’ve installed or built SciPy. If a dev build according to the docs with dev.py, then:

$ python dev.py test -s linalg  # for the whole linalg module
$ python dev.py test -t scipy.linalg.tests.test_lapack  # for a specific file
$ python dev.py test -t scipy.linalg.tests.test_lapack::test_gges_tgexc  # for a specific test failing here

If you have SciPy installed, e.g. from conda-forge, then you can use pytest in a similar way:

$ pytest --pyargs scipy.linalg.tests.test_lapack
$ pytest --pyargs scipy.linalg.tests.test_lapack::test_gges_tgexc

rgommers on Sep 3, 2022