pytest: Deleting temporary directories early causes non-unique paths and other issues

With #10442, tmp_path paths are not necessarily unique anymore - in other words, this passed previously, but fails now:

some_global_state = set()


def test_some_very_long_test_name_first(tmp_path):
    print(tmp_path)
    some_global_state.add(tmp_path)


def test_some_very_long_test_name_second(tmp_path):
    print(tmp_path)
    assert tmp_path not in some_global_state

This is because those tests used to use unique temp directory names:

x.py::test_some_very_long_test_name_first /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_0
PASSED
x.py::test_some_very_long_test_name_second /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_1
FAILED

but now don’t do so anymore, instead, the second test reuses the path the first test did:

x.py::test_some_very_long_test_name_first /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_0
PASSED
x.py::test_some_very_long_test_name_second /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_0
FAILED

In my case, this caused issues with using Qt’s QSqlDatabase, which has a list of connections as global state. Since with sqlite, a connection is based on its path, two tests used the same connection name.

I fixed it by closing the database connection properly after the test. This is definitely something I should have done before already.

However, this still violates a basic expectation on how tmp_path should behave, which is pointed out in the first sentence of the docs (emphasis mine):

You can use the tmp_path fixture which will provide a temporary directory unique to the test invocation

It’s of course questionable whether the “unique” refers to the directory contents, or the path to it - but as shown above, assuming the latter isn’t really something out of the ordinary, especially when passing that path to a library somewhere.

---

I’ve also run into another issue with the same commit, which I don’t even know how to describe. The gist of it that I had a Qt QFileSystemModel watching the temporary directory for changes - and when it was deleted after the test, Qt/PyQt somehow tried to call a callback lambda on a Python object which was already deleted, thus segfaulting…

This one smells like a PyQt bug, but again demonstrates how files and file paths are inherently attached to some side effects, and deleting them after the test has finished might be too early, depending on what code in the background is doing with it. This also makes me a bit worried about #10546, which I haven’t tried yet.

---

On a more personal note, I also wonder if it’s really the right decision to change the default behavior (policy failed), even more so in a non-major release. As demonstrated above, it can have very subtle changes on a testsuite.

But, perhaps even more important than that, this makes something I consider a great feature of pytest less discoverable. In various places (books, but also my trainings for example) the current behavior of pytest is taught to people, and by the time they discover those files aren’t actually kept around anymore, it might already be too late (if e.g. debugging a flaky test). It seems to me that this could lead to much more confusion and frustration compared to it being problematic when storing lots of big files there. The latter seems like a non-issue to me for most cases where I see tmp_path being used.

@yusuke-kadowaki @nicoddemus @RonnyPfannschmidt @hmaarrfk what do you think?