CopilotKit: 🐛 Bug: Tool schema missing (incorrect) in the LLM prompt for all tools
♻️ Reproduction Steps
- Setup CopilotKit chat with local runtime
- Register some MCP server with tools with parameters (and ideally some complex schema)
- Use middleware to log LLM request content
- Ask the chat to perform some action…
As a result
- The action is often retried or fails
- When you examine at the LLM request log, it doesn’t contain the tools schema
This is because all tools in LLM prompt are listed with No parameters required instead of the actual schema
I tracked it down to here where it tries to access the parameters via tool.schema.properties
https://github.com/CopilotKit/CopilotKit/blob/c3d73e1f6e51943a7f5dae41a83d1ffc67c48ebb/CopilotKit/packages/runtime/src/lib/runtime/mcp-tools-utils.ts#L152
BUT… the code that prepares data for this function maps them to tool.schema.parameters.properties ! so they are never found
https://github.com/CopilotKit/CopilotKit/blob/c3d73e1f6e51943a7f5dae41a83d1ffc67c48ebb/CopilotKit/packages/runtime/src/lib/runtime/copilot-runtime.ts#L419
This code was introduced in https://github.com/CopilotKit/CopilotKit/commit/f81a5267e4803d002147e3127df6c5ed3dfa2199 and seems to be released like that since v1.8.9
I tried to downgrade the package to 1.8.8 (prior to this change), but the parameters are still not mapping. Assuming there were mapping problems even before that the commit tried to fix, but failed to do so…
✅ Expected Behavior
- MCP tools schema should be properly populated for every tool in LLM request
- Tool requests succeed on a first try
❌ Actual Behavior
- LLM gets
No parameters requiredfor all tools in the prompt
[1] '- whois_domain: Looksup whois information about the domain\n' +
[1] ' No parameters required\n' +
[1] '\n' +
[1] '- whois_tld: Looksup whois information about the Top Level Domain (TLD)\n' +
[1] ' No parameters required\n' +
[1] '\n' +
[1] '- whois_ip: Looksup whois information about the IP\n' +
[1] ' No parameters required\n' +
[1] '\n' +
- The request often fails, if you are lucky, it retries and succeeds after several retries
𝌚 CopilotKit Version
1.9.1
📄 Logs (Optional)
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17
Commits related to this issue
- fix: MCP tools schema extraction bug - extract parameters from schema.parameters.properties - Fixed generateMcpToolInstructions to properly extract parameters from schema.parameters.properties - Prev... — committed to CopilotKit/CopilotKit by suhasdeshpande a year ago
- fix: MCP tools schema extraction bug - extract parameters from schema.parameters.properties - Fixed generateMcpToolInstructions to properly extract parameters from schema.parameters.properties - Prev... — committed to CopilotKit/CopilotKit by suhasdeshpande a year ago
- fix: MCP tools schema extraction bug - extract parameters from schema.parameters.properties - Fixed generateMcpToolInstructions to properly extract parameters from schema.parameters.properties - Prev... — committed to CopilotKit/CopilotKit by suhasdeshpande a year ago
Hi @suhasdeshpande I tried 1.9.2-next.22, and it still doesn’t work properly even with the new supported schema options
The biggest problem is that the schema sent to the
toolsparameter of the LLM (already at the level ofactionInputs) still fallbacks tostringtype instead ofarrayin\"terms\":{\"type\":\"string\"so the LLM malforms the type of the tool input…The instructions sent to the LLM in prompt seem fine, though (better than before):
Although I don’t understand why the same adjustment with
Array of objects with properties: ...is done ingenerateMcpToolInstructionswhen its input already did this transformation earlier viaextractParametersFromSchema. This location doesn’t seem to do anything becausepropDetails.itemsare never present at this level of execution…I also briefly checked some of the advanced scenarios I assumed will be problematic, and they still are, for the reasons I outlined above.
Looking and reading. Would you mind jumping on call and we can get all this work added in Copilotkit
https://calendly.com/suhas-copilotkit/30min
@hejtmii if you don’t mind, can you give this pre-release version a try and tell what if it solves it for you
https://github.com/CopilotKit/CopilotKit/actions/runs/16056255218/job/45311294376#step:7:37
https://github.com/CopilotKit/CopilotKit/pull/2100