prefect: Submitting many concurrent tasks causes the flow to be stuck

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn’t find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

If is submit many tasks at the same time the tasks and thus the flow never finishes. The limit when this starts to happen seems to depend on the machine.

• on windows with WSL 10 submits is enough to lock up the flow
• on most other machines more tasks are needed (the 1000 in the example does the trick)
• on our production machine the flows gets stuck every so often (~20 concurrent tasks)

Reproduction

import time
from prefect import task, flow, get_run_logger

@task
def my_task(i):
    logger = get_run_logger()
    logger.info(f"Task start {i}")
    time.sleep(0.5)

@flow
def my_flow():
    for i in range(1000):
        my_task.submit(i)

my_flow()

Error

The execution get stuck, the status of the flow is running but it persist in that state without proceeding.

Versions

prefect 2.8.7

Additional context

The issue on the production machine only showed up after upgrading from Prefect 2.7.9 (might be a coincidence tho)