gorilla: [BFCL] The multi-turn generation hangs and doesn’t progress.
Hi,
I have been trying to run generation for multi-turn use cases, but after many attempts, it still doesn’t work ( I waited like 24 hours). Other tests run fine — the only issue is that multi-turn generation doesn’t work.
It usually gets stuck around here which it seems around 1% progress:
ID: long_context_86, Turn: 0, Step: 9
----------------------------------------------------------------------------------------------------
ID: long_context_52, Turn: 0, Step: 12
Failed to decode the model response. Proceed to next turn.
----------------------------------------------------------------------------------------------------
ID: long_context_139, Turn: 1, Step: 0
----------------------------------------------------------------------------------------------------
ID: long_context_129, Turn: 0, Step: 2
Failed to decode the model response. Proceed to next turn.
----------------------------------------------------------------------------------------------------
ID: long_context_142, Turn: 0, Step: 0
----------------------------------------------------------------------------------------------------
ID: long_context_47, Turn: 1, Step: 5
----------------------------------------------------------------------------------------------------
ID: base_199, Turn: 0, Step: 10
----------------------------------------------------------------------------------------------------
ID: long_context_97, Turn: 0, Step: 6
----------------------------------------------------------------------------------------------------
ID: long_context_115, Turn: 0, Step: 6
----------------------------------------------------------------------------------------------------
ID: long_context_73, Turn: 0, Step: 11
----------------------------------------------------------------------------------------------------
ID: long_context_80, Turn: 0, Step: 9
Failed to decode the model response. Proceed to next turn.
----------------------------------------------------------------------------------------------------
ID: long_context_109, Turn: 6, Step: 0
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
ID: long_context_81, Turn: 0, Step: 10
ID: base_155, Turn: 0, Step: 17
----------------------------------------------------------------------------------------------------
ID: base_163, Turn: 0, Step: 13
I use the following commands:
bfcl generate --model Mymodel --test-category multi_turn --backend vllm --num-gpus 1 --gpu-memory-utilization 0.9 --include-input-log
and
python -m vllm.entrypoints.openai.api_server --model Mymodel --dtype bfloat16 --port 1053
and the commit id is: 9108a651ec3
Any help would be highly appreciated. Thanks.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 21
thanks, this is very helpful. I will follow above suggestions and will update you here.
Seems like long_context_105 is causing the issue. Still with 1 worker, run only that entry (you can do so with --run-ids), add a few print statements throughout this function to figure out which part is hanging. Is it stuck waiting for the server response, or is it stuck executing the model response tool call, etc. It might also be helpful if you print out the model response at each turn.