pytest: "Last Failed" mode no longer reruns all tests on successful run

For about a month now (maybe less) our team is experiencing issues with the --last-failed mode. Since we regularly update our development dependencies there is a chance that it might be due to an issue in pytest. Unfortunately I don’t have an example to reproduce the issue.

I will try downgrading to pytest 4.4 and see if this fixes the issue. If I don’t forget it, I will update this issue if that’s the case.

When using “last-failed” mode we expect the following behaviour:

• As long as there are failures, only run the failed tests on each run
• The run following the first run without any failures should re-run all tests

What we observe instead:

• As long as there are failures, only run the failed tests on each run
• The run following the first run without any failures only some tests are re-run. Only a fraction of tests are executed.

Workaround:

• Running pytest once without --last-failed solves the issue (but only once).

pip list output

Package                       Version  Location
----------------------------- -------- ------------------------------
alabaster                     0.7.12
apipkg                        1.5
asn1crypto                    0.24.0
astroid                       2.2.5
atomicwrites                  1.3.0
attr                          0.3.1
attrs                         19.1.0
autopep8                      1.4.4
Babel                         2.6.0
bboss                         1.10.1
bcrypt                        3.1.6
certifi                       2019.3.9
cffi                          1.12.3
chardet                       3.0.4
colorama                      0.4.1
commonmark                    0.9.0
config-resolver               4.3.4
coverage                      4.5.3
cryptography                  2.6.1
distribute                    0.7.3
docutils                      0.14
entrypoints                   0.3
execnet                       1.6.0
flake8                        3.7.7
flake8-polyfill               1.0.2
future                        0.17.1
gouge                         1.3.0
idna                          2.8
imagesize                     1.1.0
isort                         4.3.20
jargon                        4.14.0
Jinja2                        2.10.1
lazy-object-proxy             1.4.1
mando                         0.6.4
MarkupSafe                    1.1.1
mccabe                        0.6.1
more-itertools                7.0.0
msgpack-python                0.5.6
mypy                          0.701
mypy-extensions               0.4.1
packaging                     19.0
paramiko                      2.4.2
pathlib2                      2.3.3
pexpect                       3.3
pip                           19.1.1
pkg-resources                 0.0.0
pluggy                        0.11.0
puresnmp                      1.5.1
py                            1.8.0
pyasn1                        0.4.5
pycodestyle                   2.5.0
pycparser                     2.19
pyflakes                      2.1.1
Pygments                      2.4.0
pylint                        2.3.1
PyNaCl                        1.3.0
pynet                         3.11.0   /home/users/malbert/work/pynet
pyparsing                     2.4.0
pytest                        4.5.0
pytest-cov                    2.7.1
pytest-cover                  3.0.0
pytest-coverage               0.0
pytest-forked                 1.0.2
pytest-xdist                  1.28.0
python-dateutil               2.8.0
pytz                          2019.1
PyYAML                        3.13
radon                         3.0.3
recommonmark                  0.5.0
requests                      2.22.0
schema                        5.2.3
setuptools                    41.0.1
six                           1.12.0
snowballstemmer               1.2.1
Sphinx                        2.0.1
sphinx-rtd-theme              0.4.3
sphinxcontrib-applehelp       1.0.1
sphinxcontrib-devhelp         1.0.1
sphinxcontrib-htmlhelp        1.0.2
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.2
sphinxcontrib-serializinghtml 1.1.3
typed-ast                     1.3.5
typing                        3.6.6
urllib3                       1.25.2
verlib                        0.1
wcwidth                       0.1.7
wheel                         0.33.4
wrapt                         1.11.1

OS Information

Linux BBS-nexus.ipsw.dt.ept.lu 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

Python 3.5.2