kubectl-ai: stream model will report simulated streaming failed during non-streaming call
as below `[root@k8s-test ~]# kubectl-ai --llm-provider=openai --model=qwen3-235b-a22b
Hey there, what can I help you with today?
show me all deployment in the deafult namespace E0508 11:30:21.562983 32248 openai.go:249] OpenAI ChatCompletion API error: POST “https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions”: 400 Bad Request {“code”:“invalid_parameter_error”,“param”:null,“message”:"This model only support stream mode, please enable the stream parameter to access the model. ",“type”:“invalid_request_error”} Error: simulated streaming failed during non-streaming call: OpenAI chat completion failed: POST “https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions”: 400 Bad Request {“code”:“invalid_parameter_error”,“param”:null,“message”:"This model only support stream mode, please enable the stream parameter to access the model. ",“type”:“invalid_request_error”}
exit [root@k8s-test ~]# kubectl-ai --llm-provider=openai --model=qwen-plus
Hey there, what can I help you with today?
show me all deployment in the deafult namespace Running: kubectl get deployments -n deafult`
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (12 by maintainers)
Hmm, I wonder if it would be possible to provide a flag to control the addition of extra parameters or a separate stream flag to enable or disable streaming requests. such as kubectl-ai --llm-provider=openai --model=qwen3-235b-a22b --stream=true
Agree with @zvdy , streaming as a feature will be needed anyways.
Remember our goal here is to unblock users to be able to use whatever model meets their need best (many factors goes into why a model is best for them…).
Acceptable UX:
Add streaming as a feature in the gollm layer and add a command line flag (
--use-streaming) to enable/disable feature (kubectl-ai and k8s-bench CLIs both). [Note we had to do this for tool-use, that’s why --enable-tool-use flag exists today]. kubectl-ai` can detect invalid stream mode use (can do on the basis of error signature) and can suggest the user a workaround to invoke the tool with a command line flag. (We can also document)Note if supporting this flag for all the llm-providers turns out to be tricky, we can introduce llm-specific flag as well if that keeps the implementation simple. It will depend on how the implementation look like.
Auto-detecting Tracking model capabilities ourselves is going to be maintenance nightmare and will increase toil on us maintainers, so not desirable. I have doubts that llm-providers expose APIs to discover this capability easily but I could be wrong.
@droot I can take this