promptflow: [BUG] pf.run(..., stream=True) doesn't stream output

Describe the problem When using the following code to run my promptflow application in batch mode, I always get the log in one shot rather than see the streamed output.

from promptflow import PFClient

pf = PFClient()
column_mapping = {
        "query_id": "${data.query_id}",
        "query": "${data.query}",
        "text_passages": "${data.context}"
    }
run_batch = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping, stream=True)

When looking at the source code, I see that the problem lies in this line of code . The RubSubmitter doesn’t return the created_run immediately. Maybe the stream is not intended to return the logs in streaming mode, rather the output of the chatbot?

If this is the case, is it possible to see the intermediate results when running promptflow in batch mode?

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 15 (8 by maintainers)

Most upvoted comments

Hi, @faaany The latest version. And the parallelism has not changed for several months.

I updated my version to the latest 1.1.1. But the performance still couldn’t meet my expectations. Can we set up a quick call for discussion? It is a bit hard to reproduce my use case since I have all my services locally. You can find my E-mail address on my GitHub profile. Many Thanks!

Hi @faaany , the above result are from a simple test flow(see attachment). The perf improvement after increasing worker count maybe related to actual code logic. For example, if the code is compute intensive or I/O intensive. The speed will stop increase after reaching the upper limit of CPU or IO. So in summary, it’s not guaranteed to increase speed by 2x by setting worker count to 2x.

We have 2 ways to keep investigate the issue:

Use pf run visualize and compare Api calls for different worker count. Reference here for details.

Use cprofile to profile the program. Reference here for details. If the result shows promptflow APIs took long time to run. We can help check if there’s improvement we can do.

For follow up perf related questions, could you create a separate issue to track? We’ll use this issue to track the stream thing.

sample-flow.zip

Sure, we can close this issue and let me first try the methods you suggested to investigate the performance issue I have. And if there is indeed a performance issue, I will open another thread for that. Thanks a lot!

faaany on Dec 6, 2023