filelock: filelock 3.11 breaks our multi-threaded parallel processing use case

Hi,

In #219 @csm10495 wondered

I’m curious though: Is there ever a case where this would break someone? I guess if someone expected the object to be the same across threads… but then why would they be using the lock to begin with?

I guess we have a use case that breaks:

We have a multi-threaded processing pipeline that is set up roughly like this:

1. Initialize and create a SoftFileLock for the output file.
2. Prepare some data for step 3.
3. Generate the output file with the data from step 2.
4. Clean up and release the lock acquired in 1.

These steps are executed for multiple inputs, each generating a different output file. Step 2 for one input can run in parallel to step 3 for another input. However, step 3 for one input must not run in parallel to step 3 for another input, as it uses resources that shall not be shared.

The SoftFileLock is required because there may be another process that is executed on an overlapping set of input files (thus generating an overlapping set of output files). With filelock 3.10, we could pass the lock from the thread executing step 1 to the thread executing step 4, but this broke with #219.

For this use case, I don’t see anything wrong with sharing the lock between threads, because the lock is intended to prevent concurrent access between different processes. What do you think?