prefect: Flow run with Ray and agent throws: `ImportError: cannot import name 'SubprocessFlowRunner' from partially initialized module 'prefect.flow_runners'`
Opened from the Prefect Public Slack Community
christian.vogel: Hi Prefect Community, I am currently receiving an error when using a local (on my pc) prefect agent which pulls from a work queue in the the prefect cloud. The error only occurs when I try to use the RayTaskRunner:
(begin_task_run pid=141324) ImportError: cannot import name 'SubprocessFlowRunner' from partially initialized module 'prefect.flow_runners' (most likely due to a circular import) (/home/christian/Documents/ray_and_prefect/env/lib/python3.9/site-packages/prefect/flow_runners/__init__.py)
I am using the following dependencies: prefect==2.0b7 prefect-ray==0.1.0 ray==1.13.0 Apparently I am doing something wrong with my dependencies or when I am importing them. Do you have any idea?
christian.vogel: This is my flow and deployment:
from prefect import task, flow
from prefect_ray import RayTaskRunner
from prefect.deployments import DeploymentSpec
@task
def say_hello(name):
print(f"hello {name}")
@flow(name="temp-flow-example", task_runner=RayTaskRunner(address="auto"))
def greetings():
say_hello("Ford")
DeploymentSpec(
name="temp-flow-example",
flow=greetings
)
christian.vogel: What confuses me a bit, is the fact, that the flow only fails when pulled from the cloud and seems to work when i run it directly locally via python and a main method.
christian.vogel: Could it be related to the fact that I am running my agent in a virtual environment? Probably in the background the following is executed:
from prefect.flow_runners import SubprocessFlowRunner
which them leads to some import issues because of the already imported:
from prefect_ray import RayTaskRunner
Is that possible?
anna: do you run your agent in the same virtual environment as you run your local process? perhaps you can explicitly define your flow runner with the virtual environment you use for your agent?
flow_runner=SubprocessFlowRunner(condaenv="yourcondavenv"),
anna: I was trying to replicate and sth also didn’t work for me, even though I got a different error message using the same versions as you did
anna: my error was:
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-07_14-08-35_008202_6668/logs/dashboard.log'
anna: wow it took me forever to set this up but this helped https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support
looks like you need to be super careful about your environment with Ray and grpcio package - do you use Conda? on which machine do you run it?
christian.vogel: Tried specifying the SubprocessFlowRunner with the virtual env path in my case. but did not help
christian.vogel: I am running a venv on a linux ubuntu machine
anna: yup you’re right, I was able to reproduce your error, thanks for reporting that
<@ULVA73B9P> open “Flow run with Ray and agent throws: ImportError: cannot import name 'SubprocessFlowRunner' from partially initialized module 'prefect.flow_runners'”
Original thread can be found here.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (12 by maintainers)
We can close this issue since I think there already exists an issue here: https://github.com/PrefectHQ/prefect-ray/issues/33
I will close the prefect-ray issue once ray v2.0.1 is released on pypi
The ImportError was fixed by moving to the ray nightlies (ray-3.0.0.dev0 right now). This issue could probably be closed if it can be fixed from the Ray side, right?
i think the integration should be solved using Ray 2.0.1 or the current nightly.
It seems to be fixed with the latest Prefect 2 + prefect-ray.
Steps I took to reproduce:
Write to test.py
CLI
Results in:
Please let me know if this works for you, or if I’m missing something!
@anna-geller this is Bill Wang from Anyscale, the maintainer of Ray. Please let us know what we can do to help speed up.