Skip to content

fix(litellm): report unusable tool call arguments instead of raising - #7803

Closed
yasumorishima wants to merge 2 commits into
google:mainfrom
yasumorishima:fix-litellm-arg-parse-errors
Closed

yasumorishima wants to merge 2 commits into
google:mainfrom
yasumorishima:fix-litellm-arg-parse-errors

Conversation

@yasumorishima

@yasumorishima yasumorishima commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
Tool call arguments that cannot become a function call raised out of LiteLlm response parsing instead of being reported as MALFORMED_FUNCTION_CALL. The cases are an unhashable or non-string key, a set or bytes value from the ast.literal_eval repair, an integer over 4,300 digits, and deep nesting. The issue has a reproduction.

Solution:

  • Catch what the parsers raise for these cases. ast.literal_eval documents TypeError, MemoryError and RecursionError beyond SyntaxError and ValueError. json.loads raises a plain ValueError for a huge integer and RecursionError for deep nesting.
  • Check what the arguments parse to before using them: the keys must be strings at every depth and the value must encode as JSON. The function call is serialized back to JSON later, and the streaming path does that straight away. On Python 3.14 a 50,000-level object decodes but does not encode.
  • Apply the same catch to the other scans of model output for JSON: tool calls written into text, DeepSeek arguments, and the streamed completion check. A tool call written into text whose arguments decode but are too deep to encode again stays text.
  • Make _JsonPathTracker, which also serves the Anthropic, OpenAI and Apigee models, stop raising. It already caught JSONDecodeError and RecursionError from decoding. It now returns no partial args for an integer over 4,300 digits or nesting too deep for its recursive diff, and a valueless partial arg for an integer too large for a float (like other values PartialArg cannot hold). After a subtree too deep to diff, that call gets no more partial args; the final call is still complete.

Behaviour that changes besides the crashes:

  • In the non-streaming path, a literal holding a set, bytes, a complex number or ... used to reach the tool as that Python value. It is now reported as malformed; streaming raised on these.
  • In streaming, {1.5: 1}, {None: 1} and {True: 1} used to become {"1.5": 1}, {"null": 1} and {"true": 1}. They are now reported as malformed, as the non-streaming path did by raising.
  • A non-string key inside a nested dict or list, such as {'a': {1: 2}}, is now reported as malformed, instead of being encoded with the key turned into a string.
  • An object nested a couple of thousand levels now gives the same function call in streaming as in the non-streaming path, instead of raising.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

pytest tests/unittests/models/test_litellm.py tests/unittests/utils/test_streaming_utils.py: 884 passed on Python 3.12.14 and 3.14.7. I reverted each of the 17 changed spots one at a time, and every revert fails at least one test on both versions. For the nested key check, dropping tuples from the walk fails 3 tests, and checking keys before the encoding check hangs the self-reference test. The full suite on the fork's CI (3.11 to 3.14) has one failure, test_remote_a2a_agent.py::TestRemoteA2aAgentResolution::test_ensure_resolved_uses_custom_factory. It also fails on main today.

Manual End-to-End (E2E) Tests:

The reproduction in the issue, run against main:

unhashable key: raised TypeError
non-string key: raised ValidationError
huge integer: raised ValueError
deep nesting: raised RecursionError

and against this branch, including the existing warning for a discarded call:

Discarding tool call 'get_weather' with unparseable arguments (7 chars).
Discarding tool call 'get_weather' with unparseable arguments (6 chars).
Discarding tool call 'get_weather' with unparseable arguments (5007 chars).
Discarding tool call 'get_weather' with unparseable arguments (200000 chars).
unhashable key: FinishReason.MALFORMED_FUNCTION_CALL
non-string key: FinishReason.MALFORMED_FUNCTION_CALL
huge integer: FinishReason.MALFORMED_FUNCTION_CALL
deep nesting: FinishReason.MALFORMED_FUNCTION_CALL

The streaming path, driven the same way with a mocked acompletion, now reports MALFORMED_FUNCTION_CALL for these four and for {'a': {1, 2}}. On main, {[]: 1} and {'a': {1, 2}} raised TypeError, the huge integer raised ValueError, the deep nesting raised RecursionError, and {1: 2} became {"1": 2}.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

🤖 Generated with Claude Code

Tool call arguments are parsed with json.loads and repaired with
ast.literal_eval, but only JSONDecodeError and (SyntaxError, ValueError)
were caught. json.loads raises a plain ValueError for an integer over
the int max str digits limit and RecursionError for deep nesting,
literal_eval raises TypeError for an unhashable key such as `{[]: 1}`
(and MemoryError or RecursionError for some inputs), and a repaired
literal can hold what a function call cannot carry, such as a
non-string key, a set or bytes. Each of these raised out of response
parsing instead of being reported as MALFORMED_FUNCTION_CALL.

Catch the errors the parsers raise, and check what the arguments parse
to before using it: top-level keys must be strings and the value must
encode as JSON, since the function call is serialized back to JSON
later (the streaming path does so right away; on Python 3.14 a
50,000-level object decodes but does not encode). The scans for JSON
tool calls written into text, DeepSeek arguments and the streamed
completion check get the same catch, and a tool call written into text
whose arguments cannot be encoded again stays text. _JsonPathTracker,
shared with the other streaming models, returns no partial args for a
huge integer or for nesting too deep for its recursive diff, and a
valueless partial arg for an integer too large for a float.

Fixes google#7802
A Python literal repair can give a non-string key inside a nested dict,
such as {'a': {1: 2}}, and json.dumps turned it into {"a": {"1": 2}}
instead of reporting the call. Check keys at every level, after the
encoding check, which has already ruled out cycles and deep nesting.
copybara-service Bot pushed a commit that referenced this pull request Nov 4, 2026
Merge #7803

Tool call arguments with an unhashable or non-string key, a set or bytes
value from the Python-literal repair, an integer over the int max str digits
limit, or very deep nesting no longer raise out of LiteLlm response parsing;
they are reported as MALFORMED_FUNCTION_CALL like other unparseable
arguments. A Python literal holding a set, bytes, a complex number or `...`
used to reach the tool as that Python value on the non-streaming path and is
now reported as malformed. A non-string key at any depth, such as
`{'a': {1: 2}}`, used to reach the tool unchanged (non-streaming) or as a
string key (streaming), and is now reported as malformed; in streaming this
also covers top-level `{1.5: 1}`, `{None: 1}` and `{True: 1}`. The streaming partial-args tracker used by the LiteLLM, Anthropic,
OpenAI and Apigee models returns no partial args for these inputs instead of
raising.

Fixes #7802

PiperOrigin-RevId: 1013363783
@adk-bot

adk-bot commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Thank you @yasumorishima for your contribution! 🎉

Your changes have been successfully imported and merged via Copybara in commit d5ecb99.

Closing this PR as the changes are now in the main branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merged [Status] This PR is merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LiteLlm raises out of response parsing on some unusable tool call arguments instead of reporting MALFORMED_FUNCTION_CALL

3 participants