pytest: Missing item in dict from parametrize

A week or so ago, a very strange error appeared in our CI build here which looks like a Pytest bug to me: https://dev.azure.com/gammapy/gammapy/_build/results?buildId=25&view=logs

platform linux2 -- Python 2.7.15, pytest-3.10.1, py-1.7.0, pluggy-0.8.0

Our test code is

@pytest.mark.parametrize("case", get_test_cases())
def test_counts_predictor(case):
    desired = case.pop("npred")

and as you can see in https://github.com/gammapy/gammapy/blob/fb7b218f1e6a774271b80668a4e8dd31753327dd/gammapy/spectrum/tests/test_utils.py#L139-L141 the get_test_cases function returns a list of dicts, and those dicts do have an “npred” key.

Yet in this Python 2 build, pytest passes a dict where the “npred” key is missing:

2018-11-13T11:12:21.4346365Z _________________________ test_counts_predictor[case0] _________________________
2018-11-13T11:12:21.4346436Z 
2018-11-13T11:12:21.4347229Z case = {'e_true': <Quantity [  0.1       ,   0.10597662,   0.11231045,   0.11902282,
2018-11-13T11:12:21.4347393Z            ...927389,
2018-11-13T11:12:21.4347808Z             84.017...,  89.03890941,  94.36043101, 100.        ] TeV>, 'model': <gammapy.spectrum.models.PowerLaw object at 0x7f44e5e31990>}
2018-11-13T11:12:21.4347950Z 
2018-11-13T11:12:21.4348083Z     @pytest.mark.parametrize("case", get_test_cases())
2018-11-13T11:12:21.4348380Z     def test_counts_predictor(case):
2018-11-13T11:12:21.4348505Z >       desired = case.pop("npred")
2018-11-13T11:12:21.4348837Z E       KeyError: u'npred'
2018-11-13T11:12:21.4348915Z 
2018-11-13T11:12:21.4349045Z gammapy/spectrum/tests/test_utils.py:141: KeyError
2018-11-13T11:12:21.4366992Z _________________________ test_counts_predictor[case1] _________________________
2018-11-13T11:12:21.4367112Z 
2018-11-13T11:12:21.4367811Z case = {'aeff': <gammapy.irf.effective_area.EffectiveAreaTable object at 0x7f44e5e316d0>, 'livetime': <Quantity 10. h>, 'model': <gammapy.spectrum.models.PowerLaw object at 0x7f44e5e318d0>}
2018-11-13T11:12:21.4367983Z 
2018-11-13T11:12:21.4368082Z     @pytest.mark.parametrize("case", get_test_cases())
2018-11-13T11:12:21.4368201Z     def test_counts_predictor(case):
2018-11-13T11:12:21.4368324Z >       desired = case.pop("npred")
2018-11-13T11:12:21.4368609Z E       KeyError: u'npred'
2018-11-13T11:12:21.4368710Z 
2018-11-13T11:12:21.4368802Z gammapy/spectrum/tests/test_utils.py:141: KeyError

This code hasn’t changed on our side, and I cannot reproduce the error locally, on MacOS with Python 2.7 and pytest 3.10.1 as used in CI, and the issue doesn’t appear in our many other CI builds, only in this one build on Azure. So it looks like an edge case that is hard to reproduce.

Could you please have a look and see if this is a pytest bug? I think this way of using parametrize should be OK, to have a function that makes a list of dicts, and then to pop an item from that dict in the test, no?

cc @adonath from Gammapy.