tt-metal: [$500 BOUNTY] Cannot open device asynchronously from forked process.
The following test will hang:
def execute_process():
ttnn.open_device(device_id=0) # 3. WE OPEN THE DEVICE IN THE FORKED PROCESS
def test_fork_open_device(device): # 1. THIS OPENS A DEVICE VIA FIXTURE
import multiprocessing as mp
method = "fork" # change to "spawn" and test will pass
mp.set_start_method(method)
ttnn.close_device(device) # 2. WE CLOSE THE DEVICE IN THE PARENT PROCESS
process = mp.Process(
target=execute_process
)
process.start()
process.join() # Hangs here (because ttnn.open_device will hang in the child thread)
ttnn.open_device(device_id=0) # 4. WE RE-OPEN THE DEVICE SO FIXTURE CAN CLOSE IT AGAIN
Changing the process start method to “spawn” instead will let the test pass.
For linux POSIX users (except MacOS), the python multiprocessing module will use fork by default. So users will have to explicitly set mp.set_start_method("spawn") if they need to open devices asynchronously unless this is resolved.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
Commits related to this issue
- Fix device hang when opening from forked process - Add pthread_atfork handlers to properly reset DevicePool state in child - Clear singleton instance in child process to force reinitialization - Regi... — committed to jakobrichert/tt-metal by deleted user a year ago
- Fix device hang when opening from forked process - Add pthread_atfork handlers to properly reset DevicePool state in child - Clear singleton instance in child process to force reinitialization - Regi... — committed to jakobrichert/tt-metal by deleted user a year ago
- Fix device hang when opening from forked process - Add pthread_atfork handlers to properly reset DevicePool state in child - Clear singleton instance in child process to force reinitialization - Regi... — committed to jakobrichert/tt-metal by jakobrichert a year ago
Hey there @Shubhamsaboo
Yes I noticed there is a lot of missing information, which is why I couldn’t fix it for you immediately.
Your BOUNTY TERMS are Error 404 and Missing / Offline.
Please ask your Engineer to provide the missing details, and I will submit a proper PR !