Roo-Code: OpenAI Compatible does not submit max tokens to the API
App Version
3.18.4
API Provider
OpenAI Compatible
Model Used
N/A
π Steps to Reproduce
I use KoboldCpp which has a default max output of 512. Roo has a Max Output Tokens field and a Context Window Size field, neither is passed to the API from what I can see. What then happens is that the model keeps generating at the default 512 output and the files fail to generate.
To fix this you need to send a max_completion_tokens with the count and/or max_tokens in the json API call so that the API can automatically adjust the limit for that turn.
As a workaround KoboldCpp users can set a different default server side but this is undesirable in shared use as then many requests may end up wasteful.
π₯ Outcome Summary
All file writes fail as 512 is not enough, with the API not being adjusted higher.
π Relevant Logs or Errors (Optional)
Roo tried to use write_to_file for 'file' but the required parameter 'line_count' was missing or truncated after XXX lines of content were written. Retrying...
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17
Iβm surprised it got reopened despite you basically saying what I said but lets see what happens.
Its part of the official OpenAI standard. If a provider or backend doesnt include it technically they did it wrong and are not actually OpenAI compatible. For that reason if its a toggle I recommend it to be default enabled since there will be more backends that break by not having it than backends that cant have it.
A better strategy you can do is no toggle, but not sending it if not set above -1 or 0. That way those with broken providers have a way of turning it off without an extra toggle that influences an input field. If you do it that way starting with the disabled value makes sense because then the moment its set to a custom value it will immediately begin to apply. This would be better than high defaults as a default higher than the context size on the backend would break. I can also imagine 0 breaking some if sent.
I also noticed that Huggingface has unblocked providing a Chat Completions API inside Huggingface Spaces, that means https://koboldai-koboldcpp-tiefighter.hf.space/v1 can be used as a test API (key is ignored and can be anything, model is not good at coding but that should not matter for this test). That instance has a max context of only 4096 tokens, so the easiest way to test is setting your max output setting to a very low number such as 10 tokens and verifying if the output indeed stops after 10 tokens.
I like the idea of fixing this - my only question is whether always including this will break any providers. I wonder if it should be a checkbox that we expose.