tt-metal: [$500 BOUNTY] Cannot open device asynchronously from forked process.

The following test will hang:

def execute_process():
    ttnn.open_device(device_id=0) # 3. WE OPEN THE DEVICE IN THE FORKED PROCESS

def test_fork_open_device(device): # 1. THIS OPENS A DEVICE VIA FIXTURE
    import multiprocessing as mp
    method = "fork" # change to "spawn" and test will pass
    mp.set_start_method(method)

    ttnn.close_device(device) # 2. WE CLOSE THE DEVICE IN THE PARENT PROCESS

    process = mp.Process(
        target=execute_process
    )

    process.start()
    process.join() # Hangs here (because ttnn.open_device will hang in the child thread)
    ttnn.open_device(device_id=0) # 4. WE RE-OPEN THE DEVICE SO FIXTURE CAN CLOSE IT AGAIN

Changing the process start method to “spawn” instead will let the test pass.

For linux POSIX users (except MacOS), the python multiprocessing module will use fork by default. So users will have to explicitly set mp.set_start_method("spawn") if they need to open devices asynchronously unless this is resolved.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Hey there @Shubhamsaboo

Yes I noticed there is a lot of missing information, which is why I couldn’t fix it for you immediately.

Your BOUNTY TERMS are Error 404 and Missing / Offline.

Please ask your Engineer to provide the missing details, and I will submit a proper PR !