sglang: [Bug] sglang run for few hours, it will stop returning valid response
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
sglang run for few hours, it will stop returning valid response, based on the pm2 logs, it does not triggering any error, or message
Expected its always return token output as below
I had to restart every 5-8 hours, when it stop working…
Reproduction
just keep sending request with 1-8 concurrent requests with large input max_new_token around 1000- 7000 for around 7-8 hours or lesser you will see it stop generating token outputs.
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 5000 --api-key xxxx --tp 8 --mem-fraction-static 0.8
# tried with 0.7 mem-fraction-static same things happend
above is my running command
Environment
main branch
Name: sglang
Version: 0.2.14.post2
Summary: SGLang is yet another fast serving framework for large language models and vision language models.
Home-page: https://github.com/sgl-project/sglang
Author:
Author-email:
License: Apache License
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 21 (9 by maintainers)
We have also encountered this problem before, and accepting the following PR can solve it https://github.com/vllm-project/vllm/issues/5866 @zhyncs
GPU: A100 with nvlink sglang version:0.3.5 vllm version: 0.6.3.post1 pytorch 2.4.0 flashinfer 0.1.6+cu121torch2.4
I also encountered this situation. When I used sglang to deploy the qwen2.5-3B lora fine-tuned model and adopted the openai restful api style, after a period of high concurrent requests, the model response would have repetitive problems, and this is the second time I have encountered it.
Thanks for reporting this. It has been fixed by #1779, #1782, and #1789
Try 0.3.4.post2
this worked
--disable-custom-all-reduceWe used sglang in production and did not meet these problems. A few tips for increasing the stability
--disable-custom-all-reduce. For @liho00, this custom allreduce is mostly for small batch sizes so it may not affect the throughput a lot. You can try it on your workloads.--disable-cuda-graph-padding. Again, this won’t hurt peak throughput.Okay, this is an issue worth paying attention to. We place great importance on stability. Can you provide more detailed steps to reproduce the issue? Because we haven’t encountered a similar problem online before.
Just a bit curious, why is there a vllm in the prefix of each line before the log?