sglang: [Bug] sglang run for few hours, it will stop returning valid response

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

sglang run for few hours, it will stop returning valid response, based on the pm2 logs, it does not triggering any error, or message image

Expected its always return token output as below image

I had to restart every 5-8 hours, when it stop working…

Reproduction

just keep sending request with 1-8 concurrent requests with large input max_new_token around 1000- 7000 for around 7-8 hours or lesser you will see it stop generating token outputs.

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 5000 --api-key xxxx --tp 8 --mem-fraction-static 0.8

# tried with 0.7 mem-fraction-static same things happend

above is my running command

Environment

main branch

Name: sglang
Version: 0.2.14.post2
Summary: SGLang is yet another fast serving framework for large language models and vision language models.
Home-page: https://github.com/sgl-project/sglang
Author: 
Author-email: 
License: Apache License

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (9 by maintainers)

Commits related to this issue

Most upvoted comments

We used sglang in production and did not meet these problems. A few tips for increasing the stability

  1. Try to disable custom all reduce by --disable-custom-all-reduce. For @liho00, this custom allreduce is mostly for small batch sizes so it may not affect the throughput a lot. You can try it on your workloads.
  2. Try --disable-cuda-graph-padding. Again, this won’t hurt peak throughput.
  3. Try other NCCL configs as suggested by @m0g1cian
  4. Try to match cuda driver versions.

We have also encountered this problem before, and accepting the following PR can solve it https://github.com/vllm-project/vllm/issues/5866 @zhyncs

GPU: A100 with nvlink sglang version:0.3.5 vllm version: 0.6.3.post1 pytorch 2.4.0 flashinfer 0.1.6+cu121torch2.4

image image

I also encountered this situation. When I used sglang to deploy the qwen2.5-3B lora fine-tuned model and adopted the openai restful api style, after a period of high concurrent requests, the model response would have repetitive problems, and this is the second time I have encountered it.

Thanks for reporting this. It has been fixed by #1779, #1782, and #1789

Try 0.3.4.post2

We used sglang in production and did not meet these problems. A few tips for increasing the stability

  1. Try to disable custom all reduce by --disable-custom-all-reduce. For @liho00, this custom allreduce is mostly for small batch sizes so it may not affect the throughput a lot. You can try it on your workloads.
  2. Try --disable-cuda-graph-padding. Again, this won’t hurt peak throughput.
  3. Try other NCCL configs as suggested by @m0g1cian
  4. Try to match cuda driver versions.

this worked --disable-custom-all-reduce

We used sglang in production and did not meet these problems. A few tips for increasing the stability

  1. Try to disable custom all reduce by --disable-custom-all-reduce. For @liho00, this custom allreduce is mostly for small batch sizes so it may not affect the throughput a lot. You can try it on your workloads.
  2. Try --disable-cuda-graph-padding. Again, this won’t hurt peak throughput.
  3. Try other NCCL configs as suggested by @m0g1cian
  4. Try to match cuda driver versions.

Okay, this is an issue worth paying attention to. We place great importance on stability. Can you provide more detailed steps to reproduce the issue? Because we haven’t encountered a similar problem online before.

Just a bit curious, why is there a vllm in the prefix of each line before the log?