vllm: [RFC]: Returning Last Token, Last Layer Hidden States Through The OpenAI API

Motivation.

As a distributed inference provider, we use vLLM because it is fast, has cutting edge capabilities, and is well-maintained.

We serve millions of inference requests per day, all using the OpenAI API format.

Because we run a distributed network, we need to verify that our operators ran the model that was requested (and did not substitute some smaller model or use a canned response).

TopLOC is a very robust inference verification technique. However, TopLOC requires the last layer’s hidden states for a single token (typically the last token) of the response, something that isn’t exposed through the OpenAI API.

Rather than rebuilding an entire API stack, we’d love to leverage vLLM’s OpenAI API by incorporating the ability to return the last layer’s hidden states, but specifically for the last token only. Then, we will be able to easily implement TopLOC verification as part of our network.

Please note that this is a different use case from mechaninterp.

Proposed Change.

We are proposing the following schema changes:

  • ServerArgs: Add return_hidden_states to turn the feature on at the engine level.

ChatCompletionResponseChoice and CompletionResponseChoice: Change: Add hidden_states: Optional[List[List[float]]] = None, and omit the field entirely when not configured to return it.

ChatCompletionResponseStreamChoice and CompletionResponseStreamChoice: Change: Add hidden_states: Optional[List[List[float]]] = None to DeltaMessage on the final chunk (or possibly the usage chunk), and omit the field entirely when not configured to return it.

3.1-8b has a hidden dimension of 4096, so a hidden state would be of shape [1, 4096].

This is not small, but the payload size is smaller than requesting top 5 logprobs from a response of 1,000 tokens, a feature which the OpenAI API supports (1,000 tokens is not an unusual response size).

Feedback Period.

No response

CC List.

@simon-mo

Any Other Things.

We would like to scope the implementation of this RFC to v0, at least initially.

v1 has architectural constraints that make returning last token, last layer hidden states from the core engine somewhat more challenging. This requires some careful thought from our team.

No response

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 19 (2 by maintainers)

Most upvoted comments

cc @maxdebayser since we also have #12249

Thank you for your feedback @WoosukKwon

I’ll see if i can put together an implementation which hashes inside vLLM, and transmits only the hash in the EngineCoreOutput.

I’ll implement as much of the hashing operation on the GPU as possible in order to avoid GPU->CPU costs. Because only the hash is included on the EngineCoreOutput, the ZMQ serialization costs should be relatively low.

And no hashing will be performed if the feature is disabled at the server level.

Does this sound like a more acceptable design?

@kyle-pena-kuzco What do you exactly mean by the “last token”? Do you mean the last token of prefill? Also, can we implement hashing inside vllm and return hash (instead of the tensor)?

I asked for this RFC given it’s a very specific on last hidden states of last token for a production use case.