kubectl-ai: stream model will report simulated streaming failed during non-streaming call

as below `[root@k8s-test ~]# kubectl-ai --llm-provider=openai --model=qwen3-235b-a22b

Hey there, what can I help you with today?

show me all deployment in the deafult namespace E0508 11:30:21.562983 32248 openai.go:249] OpenAI ChatCompletion API error: POST “https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions”: 400 Bad Request {“code”:“invalid_parameter_error”,“param”:null,“message”:"This model only support stream mode, please enable the stream parameter to access the model. ",“type”:“invalid_request_error”} Error: simulated streaming failed during non-streaming call: OpenAI chat completion failed: POST “https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions”: 400 Bad Request {“code”:“invalid_parameter_error”,“param”:null,“message”:"This model only support stream mode, please enable the stream parameter to access the model. ",“type”:“invalid_request_error”}

exit [root@k8s-test ~]# kubectl-ai --llm-provider=openai --model=qwen-plus

Hey there, what can I help you with today?

show me all deployment in the deafult namespace Running: kubectl get deployments -n deafult`

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (12 by maintainers)

Most upvoted comments

Hmm, I wonder if it would be possible to provide a flag to control the addition of extra parameters or a separate stream flag to enable or disable streaming requests. such as kubectl-ai --llm-provider=openai --model=qwen3-235b-a22b --stream=true

Agree with @zvdy , streaming as a feature will be needed anyways.

Remember our goal here is to unblock users to be able to use whatever model meets their need best (many factors goes into why a model is best for them…).

Acceptable UX:

Add streaming as a feature in the gollm layer and add a command line flag (--use-streaming) to enable/disable feature (kubectl-ai and k8s-bench CLIs both). [Note we had to do this for tool-use, that’s why --enable-tool-use flag exists today]. kubectl-ai` can detect invalid stream mode use (can do on the basis of error signature) and can suggest the user a workaround to invoke the tool with a command line flag. (We can also document)

Note if supporting this flag for all the llm-providers turns out to be tricky, we can introduce llm-specific flag as well if that keeps the implementation simple. It will depend on how the implementation look like.

Auto-detecting Tracking model capabilities ourselves is going to be maintenance nightmare and will increase toil on us maintainers, so not desirable. I have doubts that llm-providers expose APIs to discover this capability easily but I could be wrong.

@droot I can take this