spopt: `TestAZP.test_azp_basic_from_w` CI failure

TestAZP.test_azp_basic_from_w CI failure.

diagnosis & potential causes

Seems to fail intermittently across versions and OSs (fails locally on MacOS) with the same resultant array
Could be an issue with the random state?
Could be a change in numpy or scipy?

major versions

geopandas==0.13.2
networkx== 3.1
numpy==1.25.0
pandas==2.0.2
scipy==1.10.1
shapely==2.0.1

=========================================================================== FAILURES ============================================================================
_________________________________________________________________ TestAZP.test_azp_basic_from_w _________________________________________________________________
[gw0] darwin -- Python 3.11.4 /Users/user/mambaforge/envs/py311_spopt/bin/python3.11

self = <spopt.tests.test_azp.TestAZP object at 0x162099350>

    def test_azp_basic_from_w(self):
        w = libpysal.weights.Queen.from_dataframe(self.mexico)
        attrs_name = [f"PCGDP{year}" for year in range(1950, 2010, 10)]
        args = (self.mexico, w, attrs_name)
        kwargs = {"n_clusters": 3, "random_state": RANDOM_STATE}
        model = AZP(*args, **kwargs)
        model.solve()
    
>       numpy.testing.assert_array_equal(model.labels_, self.basic_from_w_labels)

spopt/tests/test_azp.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<built-in function eq>, array([0., 0., 0., 0., 1., 1., 1., 2., 0., 1., 1., 0., 0., 2., 2., 2., 2.,
       2., 0., 2., 2., 2., 0., 1., 1., 0., 0., 1., 2., 1., 1., 2.]), [0, 0, 2, 0, 0, 2, ...])
kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Arrays are not equal
E           
E           Mismatched elements: 27 / 32 (84.4%)
E           Max absolute difference: 2.
E           Max relative difference: 1.
E            x: array([0., 0., 0., 0., 1., 1., 1., 2., 0., 1., 1., 0., 0., 2., 2., 2., 2.,
E                  2., 0., 2., 2., 2., 0., 1., 1., 0., 0., 1., 2., 1., 1., 2.])
E            y: array([0, 0, 2, 0, 0, 2, 2, 1, 1, 2, 2, 1, 2, 1, 1, 2, 1, 1, 1, 1, 1, 1,
E                  0, 0, 0, 2, 2, 2, 0, 0, 0, 1])

../mambaforge/envs/py311_spopt/lib/python3.11/contextlib.py:81: AssertionError
--------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------
n_regions_per_comp {0: 3}
comp_label 0
n_regions_in_comp 3
Regions in comp: {0, 1, 2}

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (16 by maintainers)

Most upvoted comments

@jGaboardi Gotcha! I caught the error. The mismatch over the python versions is displayed in my gist. It already has the proposed fix in _randomly_divide_connected_graph. The problem was the MST representation as I mentioned yesterday. The MST as sparse matrix is not correctly represented as it gives the directed adjacency matrix instead of giving an undirected adjacency. Then sometimes, over the python versions, it was throwing correct matrices but with some fluctuations in adj representation.

Suppose my MST is something like:

\begin{bmatrix} 
0 && 1 && 0\\ 
0 && 0 && 1\\ 
1 && 0 && 0 
\end{bmatrix}

The example of fluctuation:

\begin{bmatrix} 
0 && 1 && 1\\ 
0 && 0 && 1\\
0 && 0 && 0 
\end{bmatrix}

This example is also a MST but is another version of it that scipy.sparse also generates.

Then these minor changes affected the random choice of clusters in the util function.

The proposed fix, so, is creating the undirected graph sparse matrix based on a dok_matrix. Therefore, we must have to transform the csr_matrix into a dok_matrix, assign the new edges and then transform back the dok_matrix to the csr_matrix to continue doing the operations efficiently on it.

Maybe, there is a better fixing to it.

After fixing it, I already generated the labels for AZP and all are the same now in my environment.

gegen07 on Oct 18, 2023

sorry, i missed this for awhile.

No, the AZP stuff is old; everything here is ported from region. Serge would know better than I, but my guess would be that’s Aleks’s code. Cant say that ive seen azp used in the wild in decades, but theres a reference implementation in geoda we could test against. Looks like there an R one too

update: oh… if i’d actually read the whole thread, james already pointed to those

knaaptime on Oct 17, 2023

@gegen07 great job! That sounds like a sensible solution. Can you open a PR so we can follow the implementation discussion on top of the actual code?

martinfleis on Oct 17, 2023

Maybe we should start testing all methods under region util stuff to debug these errors more easily.

@gegen07 I was working over the weekend on this and I agree 100% with you. That is something that should have been happening from the beginning, but hindsight is 20/20…

This is great debugging; with my own stuff I was narrowing down on that, too. I have been working on some bash scripts that run though env creation --> solve --> plotting to highlight the differences when solving while incrementing clusters. I have found that there may even be differences outside of the (3.8 <–> 3.{9,10,11}) divide… Check the difference between azp_mexico_39_clusters4.png and azp_mexico_310_clusters4.png in the attached zip.

azp_plots_results.zip

Seems to be a result of untracked randomness and change in an upstream package.

jGaboardi on Oct 16, 2023

@jGaboardi I dug into that, but I couldn’t get the error. I found that there’s a difference between scipy versions 1.10.1 and 1.11.1, when using the 1.10.1 version the result is [0, 0, 0, 0, 1, 1, 1, 2, 0, 1, 1, 0, 0, 2, 2, 2, 2, 2, 0, 2, 2, 2, 0, 1, 1, 0, 0, 1, 2, 1, 1, 2] while the last version returns [0, 0, 1, 1, 0, 1, 1, 2, 2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1]. I debugged the AZP function pipeline and noticed another difference in line 518, where select randomly and remove any region from neighbors. The scipy 1.10.1 selects this array [0, 1, 2, 2, 1, 0, 0, 1] (the order is based on indexes), and the scipy 1.11.1 selects this array [0, 2, 1, 0, 2, 1, 2, 0, 1]

Furthermore, changing numpy versions and keeping the scipy the same doesn’t change anything on AZP results, neither on random choice.

Maybe the error is related to the random number generation, but this is so weird…

gegen07 on Jul 23, 2023