llama_index: [Bug]: running with asyncio loop in uvicorn cause `WorkflowHandler.pending` to not be handled correctly in phoenix (arize)
Bug Description
I am running my python environment for running llama index inside a docker container. To prevent errors on starting up after introducing ragas in the whole project I added --loop asyncio to the starting command. Everything seems to be working fine accept for one thing: the traces of a workflow don’t combine into a single grouped, basically having the same problem as https://github.com/run-llama/llama_index/issues/16283 - showing "<WorkflowHandler pending>" and splitting up the steps
The docker image start command before (and screenshot of output in phoenix):
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "80", "--reload"]
The docker image start command after
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "80", "--reload", "--loop", "asyncio"]
Note the added last two arguments
I want to note I don’t get any weird logs, probably there’s a mismatch in context due to the async behaviour that is now wrapped around the application. Further information about the application is that I use fastapi as the main entrypoint to access my code and execute functions.
Version
0.12.10
Steps to Reproduce
1. Setup simpel workflow
from llama_index.core.workflow import StartEvent, StopEvent,Workflow, step
import time
class MyWorkflow(Workflow):
@step
async def my_step(self, ev: StartEvent) -> StopEvent:
time.sleep(3)
return StopEvent(result="Hi, world!")
w = MyWorkflow(timeout=10, verbose=False)
2. Run the workflow with the span handler
import llama_index.core.instrumentation as instrument
from fastapi import FastAPI
from phoenix.otel import register
tracer_provider = register(
# project_name="default", # Default is 'default'
endpoint="http://localhost:6006/v1/traces",
)
# Instrument the application
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
app = FastAPI()
@app.get("/")
async def read_root():
result = await w.run()
3. Setup docker with phoenix
Pull & run latest Phoenix image from Docker Hub:
docker pull arizephoenix/phoenix:latest
docker run -p 6006:6006 arizephoenix/phoenix:latest
Or use the setup from https://github.com/run-llama/llama_index/issues/16283 as I think it will yield the same output, but haven’t confirmed this.
This will expose the Phoenix on localhost:6006
4. Run the application with uvicorn
Make sure it’s installed
run uvicorn main:app --loop asyncio
5. Execute get request to fastapi
call in web-browser or wget localhost:8000/ and it should execute the workflow
Relevant Logs/Tracbacks
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16
Are you on our discord? Feel free to ping me there (you’ll see my name in every channel lol
Logan M) – if not, we can just continue hereAfter consulting the arize folks, I think this issue will be very hard to fix 😓 In the meantime, I’m happy to help you redesign your code so that you don’t need
nest_asyncio– is there a specific block of code that is requiring this?@SvenWesterlaken there is likely a way to structure your code without needing
nest_asyncio– but yea ok, let me see if I can do something for thisPackages
As for the packages @logan-markewich (deleted quite a lot of rows, as it was thrice as long, in case you miss a package you need to verify the version of, let me know):
Reproduction
1. Setup the workflow from the rag with reranking example
But without the ingestion to simplify it
2. Run the workflow from a basic endpoint using FastAPI & setup tracing
make sure to add an
__init__.pyso the relative import works3. Create file called
test.txtJust create a txt file with simple sentence to be indexed, in my case the content didn’t matter. As long as we have something to index.
4. Run phoenix in local docker container
This will expose the Phoenix on localhost:6006, it should pull 7.5.1 version of phoenix
5. Run app using uvicorn
Make sure it’s installed globally
run uvicorn main:app --loop asyncio --reload6. Access API endpoint to start workflow
call in web-browser, postman or wget localhost:8000/ and it should execute the workflow
Should return string of
"completed"on completion7. Go to phoenix to see the result
In case it still can be reproduced, there might be something with the docker container? What you can try is to replace step 5 with running the python file within a docker container:
Code lives in subfolder of ‘app’ next to the Dockerfile.
Let me try to create a local reproduction later today and send you the package versions.
It’s because I was a bit cooked yesterday after spending so much time coding under pressure. So I couldn’t get myself to put in the effort to create a full reproduction and test it. Also couldn’t just copy-paste my repo code as it has some sensitive info etc. so quickly grabbed the reproduction from the related issue in hopes that it would give enough for people to figure out the issue as I normally expect people to react after a week haha. So once again thanks for the quick responses and help @logan-markewich !
Don’t know if this is correct @logan-markewich, but it seems that the problem mainly arrises from the double context and thus resetting the context around the span id. Which causes the span to exit prematurely and forego the parent_id that is saved which causes the detachments of the steps. https://github.com/run-llama/llama_index/blob/2ea7c51585ae5a060f2bc22edde0e1cc4fafddd2/llama-index-core/llama_index/core/instrumentation/dispatcher.py#L311-L311
It is worth noting though that the steps do combine correctly, only the first step of the running of the workflow itself looks like it has some problems.
Probably now that the global loop is async, the context created doesn’t need to be reset?
Might be related: https://github.com/django/asgiref/issues/267 and a related PR to fix it: https://github.com/instana/python-sensor/pull/666
Thanks for the fast response @logan-markewich! only can add from my part that the asyncio global loop is the cause in my case. Removing it solves it, so would say it’s something with the global async, but I think you already go that from my initial message haha. Might be able to look into it later this week, but so far don’t think I have enough experience to be of much help.
@SvenWesterlaken Greetings! I’m here to assist you with any bugs, questions, or contributions you might have. Let me know how I can help!
The issue you’re experiencing with the
WorkflowHandler.pendingstatus not being handled correctly when using--loop asyncioin Uvicorn might be related to how asynchronous contexts are managed in LlamaIndex version 0.12.10. This version may have introduced changes in async handling that affect workflow tracing, particularly when combined with the--loop asynciooption.Here are a few steps you can take to troubleshoot and potentially resolve the issue:
Check Async Context Management: Ensure that the async context is correctly managed throughout your application. The use of
--loop asynciocan sometimes lead to context mismatches if the application or its dependencies are not fully compatible with this event loop.Review Tracing Configuration: Double-check your tracing setup with Phoenix and ensure that the
LlamaIndexInstrumentoris correctly configured. Make sure that the tracer provider is properly set up to handle async workflows.Test Without
--loop asyncio: Temporarily remove the--loop asynciooption to see if the issue persists. This can help determine if the problem is specifically related to the asyncio loop.Update Dependencies: Ensure all related packages, including LlamaIndex, FastAPI, and Uvicorn, are up to date. Sometimes, compatibility issues arise from outdated dependencies.
Consult Documentation: Review any available documentation or release notes for LlamaIndex version 0.12.10 to identify any known issues or changes related to async handling and tracing.
If these steps do not resolve the issue, it may be necessary to further investigate the specific interactions between Uvicorn’s asyncio loop and LlamaIndex’s tracing mechanisms.
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other