vllm: Missing prometheus metrics in `0.3.0`

First of all, thanks for the great open source library!

The docs promise a few more additional metrics that I’m not seeing in vLLM 0.3.0, have these been removed? I.e. if I hit /metrics of the OpenAI API server for a deployed model… you’ll see no vllm:time_to_first_token_seconds or vllm:time_per_output_token_seconds or vllm:e2e_request_latency_seconds

image

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 17 (8 by maintainers)

Most upvoted comments

I think a new release will be pushed soon -> https://github.com/vllm-project/vllm/issues/2859

  • We implemented Prometheus metrics in v0.3.1
  • There is a bug in v0.5.4 re: Prometheus due to the multiprocessing in the openai server. Use —disable-frontend-multiprocessing to get around it
  • This is fixed on main