gorilla: [BFCL] The multi-turn generation hangs and doesn’t progress.

Hi,

I have been trying to run generation for multi-turn use cases, but after many attempts, it still doesn’t work ( I waited like 24 hours). Other tests run fine — the only issue is that multi-turn generation doesn’t work.

It usually gets stuck around here which it seems around 1% progress:

ID: long_context_86, Turn: 0, Step: 9
----------------------------------------------------------------------------------------------------
ID: long_context_52, Turn: 0, Step: 12
Failed to decode the model response. Proceed to next turn.
----------------------------------------------------------------------------------------------------
ID: long_context_139, Turn: 1, Step: 0
----------------------------------------------------------------------------------------------------
ID: long_context_129, Turn: 0, Step: 2
Failed to decode the model response. Proceed to next turn.
----------------------------------------------------------------------------------------------------
ID: long_context_142, Turn: 0, Step: 0
----------------------------------------------------------------------------------------------------
ID: long_context_47, Turn: 1, Step: 5
----------------------------------------------------------------------------------------------------
ID: base_199, Turn: 0, Step: 10
----------------------------------------------------------------------------------------------------
ID: long_context_97, Turn: 0, Step: 6
----------------------------------------------------------------------------------------------------
ID: long_context_115, Turn: 0, Step: 6
----------------------------------------------------------------------------------------------------
ID: long_context_73, Turn: 0, Step: 11
----------------------------------------------------------------------------------------------------
ID: long_context_80, Turn: 0, Step: 9
Failed to decode the model response. Proceed to next turn.
----------------------------------------------------------------------------------------------------
ID: long_context_109, Turn: 6, Step: 0
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
ID: long_context_81, Turn: 0, Step: 10
ID: base_155, Turn: 0, Step: 17
----------------------------------------------------------------------------------------------------
ID: base_163, Turn: 0, Step: 13

I use the following commands: bfcl generate --model Mymodel --test-category multi_turn --backend vllm --num-gpus 1 --gpu-memory-utilization 0.9 --include-input-log

and python -m vllm.entrypoints.openai.api_server --model Mymodel --dtype bfloat16 --port 1053

and the commit id is: 9108a651ec3

Any help would be highly appreciated. Thanks.

About this issue

Most upvoted comments

thanks, this is very helpful. I will follow above suggestions and will update you here.

Seems like long_context_105 is causing the issue. Still with 1 worker, run only that entry (you can do so with --run-ids), add a few print statements throughout this function to figure out which part is hanging. Is it stuck waiting for the server response, or is it stuck executing the model response tool call, etc. It might also be helpful if you print out the model response at each turn.