vllm: [Bug]: no output of profile when VLLM_TORCH_PROFILER_DIR is enabled for vllm serve

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

Model Input Dumps

No response

πŸ› Describe the bug

I start a vlllm server with

VLLM_RPC_TIMEOUT=1800000 VLLM_TORCH_PROFILER_DIR=***/MLinference/language/llama2-70b/vllm_profile_output  vllm serve ${CHECKPOINT_PATH}   --tensor-parallel-size 4 2>&1 | tee vllm_serve_profile_opt-13b_log.log & 

then use scripts from MLinference/language/llama2-70b/main.py to feed request to the server. After the scripts is finished I cannot see any output of profile in the given folder. But I can see from the vllm_serve_profile_opt-13b_log.log that the torch profiler is enabled.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (1 by maintainers)

Most upvoted comments

https://docs.vllm.ai/en/latest/contributing/profiling.html?h=profile#openai-server has worked well for me. I run VLLM_TORCH_PROFILER_DIR=path/to/dir vllm serve ... and export VLLM_RPC_TIMEOUT=1800000 and then python benchmarks/benchmark_serving.py ... --profile... and I can see the trace files after it