Skip to content

Evaluation pairs every ID-less tool call with the last tool response #8132

Description

@1aifanatic

Describe the bug

Evaluation transcripts with missing function-call IDs pair every tool call with the last ID-less response. get_all_tool_calls_with_responses constructs a dictionary keyed by response ID, so all missing IDs collide under None. The incorrect pairs reach the rubric tool-use judge prompt.

Minimal reproduction

from google.adk.evaluation.eval_case import IntermediateData, get_all_tool_calls_with_responses
from google.genai import types

data = IntermediateData(
    tool_uses=[types.FunctionCall(name='search'), types.FunctionCall(name='book')],
    tool_responses=[
        types.FunctionResponse(name='search', response={'result': 'found'}),
        types.FunctionResponse(name='book', response={'result': 'booked'}),
    ],
)
print([(call.name, response.name) for call, response in get_all_tool_calls_with_responses(data)])

Actual: [('search', 'book'), ('book', 'book')].
Expected: [('search', 'search'), ('book', 'book')].

Scope and proposed fix

This affects restored or hand-authored legacy IntermediateData and ID-less InvocationEvents, not normal Runner traces that supply call IDs. Match missing-ID records by name and occurrence, consuming each response once; retain explicit-ID matching. For multiple calls of the same name, recorded order is the only available correlation information.

Reproduced on main at 12aab4aaa, Python 3.12.10, Windows. A JSON-round-tripped Invocation was also evaluated through the real RubricBasedToolUseV1Evaluator with a scripted external judge; its prompt contained the incorrect pairs. No external API credentials are needed.

Prepared with AI assistance; reproduction and validation were executed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions