CopilotKit: 🐛 Bug: Tool schema missing (incorrect) in the LLM prompt for all tools

♻️ Reproduction Steps

  1. Setup CopilotKit chat with local runtime
  2. Register some MCP server with tools with parameters (and ideally some complex schema)
  3. Use middleware to log LLM request content
  4. Ask the chat to perform some action…

As a result

  1. The action is often retried or fails
  2. When you examine at the LLM request log, it doesn’t contain the tools schema

This is because all tools in LLM prompt are listed with No parameters required instead of the actual schema

I tracked it down to here where it tries to access the parameters via tool.schema.properties https://github.com/CopilotKit/CopilotKit/blob/c3d73e1f6e51943a7f5dae41a83d1ffc67c48ebb/CopilotKit/packages/runtime/src/lib/runtime/mcp-tools-utils.ts#L152

BUT… the code that prepares data for this function maps them to tool.schema.parameters.properties ! so they are never found https://github.com/CopilotKit/CopilotKit/blob/c3d73e1f6e51943a7f5dae41a83d1ffc67c48ebb/CopilotKit/packages/runtime/src/lib/runtime/copilot-runtime.ts#L419

This code was introduced in https://github.com/CopilotKit/CopilotKit/commit/f81a5267e4803d002147e3127df6c5ed3dfa2199 and seems to be released like that since v1.8.9

I tried to downgrade the package to 1.8.8 (prior to this change), but the parameters are still not mapping. Assuming there were mapping problems even before that the commit tried to fix, but failed to do so…

✅ Expected Behavior

  1. MCP tools schema should be properly populated for every tool in LLM request
  2. Tool requests succeed on a first try

❌ Actual Behavior

  1. LLM gets No parameters required for all tools in the prompt
[1]         '- whois_domain: Looksup whois information about the domain\n' +
[1]         '    No parameters required\n' +
[1]         '\n' +
[1]         '- whois_tld: Looksup whois information about the Top Level Domain (TLD)\n' +
[1]         '    No parameters required\n' +
[1]         '\n' +
[1]         '- whois_ip: Looksup whois information about the IP\n' +
[1]         '    No parameters required\n' +
[1]         '\n' +
  1. The request often fails, if you are lucky, it retries and succeeds after several retries

𝌚 CopilotKit Version

1.9.1

📄 Logs (Optional)


About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 17

Commits related to this issue

Most upvoted comments

Hi @suhasdeshpande I tried 1.9.2-next.22, and it still doesn’t work properly even with the new supported schema options

The biggest problem is that the schema sent to the tools parameter of the LLM (already at the level of actionInputs) still fallbacks to string type instead of array in \"terms\":{\"type\":\"string\" so the LLM malforms the type of the tool input…

"{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\",\"description\":\"Display name of the taxonomy group\"},\"codename\":{\"type\":\"string\",\"description\":\"Codename of the taxonomy group (optional, will be generated if not provided)\"},\"external_id\":{\"type\":\"string\",\"description\":\"External ID of the taxonomy group (optional)\"},\"terms\":{\"type\":\"string\",\"description\":\"Hierarchical structure of taxonomy terms Array of objects with properties: name, codename, external_id, terms\"}},\"required\":[\"name\",\"terms\"]}"

The instructions sent to the LLM in prompt seem fine, though (better than before):

- add-taxonomy-group-mapi: Add a new taxonomy group via Management API
    - name* (string) - Display name of the taxonomy group
    - codename (string) - Codename of the taxonomy group (optional, will be generated if not provided)
    - external_id (string) - External ID of the taxonomy group (optional)
    - terms* (array) - Hierarchical structure of taxonomy terms Array of objects with properties: name, codename, external_id, terms

Although I don’t understand why the same adjustment with Array of objects with properties: ... is done in generateMcpToolInstructions when its input already did this transformation earlier via extractParametersFromSchema. This location doesn’t seem to do anything because propDetails.items are never present at this level of execution…

Image

I also briefly checked some of the advanced scenarios I assumed will be problematic, and they still are, for the reasons I outlined above.

Looking and reading. Would you mind jumping on call and we can get all this work added in Copilotkit

https://calendly.com/suhas-copilotkit/30min

@hejtmii if you don’t mind, can you give this pre-release version a try and tell what if it solves it for you

0.0.0-fix-mcp-parameters-extract-20250703165548

https://github.com/CopilotKit/CopilotKit/actions/runs/16056255218/job/45311294376#step:7:37

https://github.com/CopilotKit/CopilotKit/pull/2100