vllm: [Bug]: no output of profile when VLLM_TORCH_PROFILER_DIR is enabled for vllm serve
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
Model Input Dumps
No response
π Describe the bug
I start a vlllm server with
VLLM_RPC_TIMEOUT=1800000 VLLM_TORCH_PROFILER_DIR=***/MLinference/language/llama2-70b/vllm_profile_output vllm serve ${CHECKPOINT_PATH} --tensor-parallel-size 4 2>&1 | tee vllm_serve_profile_opt-13b_log.log &
then use scripts from MLinference/language/llama2-70b/main.py to feed request to the server. After the scripts is finished I cannot see any output of profile in the given folder. But I can see from the vllm_serve_profile_opt-13b_log.log that the torch profiler is enabled.
Before submitting a new issueβ¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (1 by maintainers)
https://docs.vllm.ai/en/latest/contributing/profiling.html?h=profile#openai-server has worked well for me. I run
VLLM_TORCH_PROFILER_DIR=path/to/dir vllm serve ...andexport VLLM_RPC_TIMEOUT=1800000and thenpython benchmarks/benchmark_serving.py ... --profile...and I can see the trace files after it