Skip to content

Commit 10b7162

Browse files
haiyuan-eng-googlecopybara-github
authored andcommitted
fix(plugins): answer failed tool calls in eval traces and skip HITL tools
- A TOOL_ERROR row, which the plugin logs instead of TOOL_COMPLETED when a tool raises, becomes a `function_response` of `{"error": ...}` that holds the error message, or `Tool execution failed` when the row has none. Every tool call that completes or raises now has a response, which comes after the calls that started with it. The transfer and HITL request tools still add no response, as their calls reach the trace as AGENT_TRANSFER and HITL request rows. `tool_error_count` no longer counts the errors of the transfer and HITL request tools. With `eval_view_mode="full"`, TOOL_ERROR joins the events that the eval views require, so denying it or leaving it out of `event_allowlist` is rejected at construction. - A TOOL_STARTING, TOOL_COMPLETED or TOOL_ERROR row that names no tool, as when a custom `content_formatter` fails, adds no event to the trace, and such a TOOL_STARTING is not a tool call. - `adk_request_credential`, `adk_request_confirmation` and `adk_request_input`, the tools that pause a turn for a credential, a confirmation or the user's input, are no longer tool calls: `tool_call_count`, `distinct_tools_used` and the consecutive-call columns skip them, so asking the user the same question three times is not a tool loop. The trace still shows each request as a `function_call`. Calls on either side of a HITL request are not consecutive, even when the turn resumes with the same `invocation_id`. Co-authored-by: Haiyuan Cao <haiyuan@google.com> PiperOrigin-RevId: 1014092677
1 parent 02db0ff commit 10b7162

4 files changed

Lines changed: 635 additions & 71 deletions

File tree

‎src/google/adk/plugins/bigquery_agent_analytics_plugin.py‎

Lines changed: 34 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2658,7 +2658,7 @@ class BigQueryLoggerConfig:
26582658
``content`` column and, with the default ``eval_view_mode`` of
26592659
``"full"``, require ``USER_MESSAGE_RECEIVED``,
26602660
``AGENT_STARTING``, ``LLM_REQUEST``, ``TOOL_STARTING``,
2661-
``TOOL_COMPLETED``, ``TOOL_PAUSED``, ``AGENT_TRANSFER``,
2661+
``TOOL_COMPLETED``, ``TOOL_ERROR``, ``TOOL_PAUSED``, ``AGENT_TRANSFER``,
26622662
``AGENT_RESPONSE``, ``INVOCATION_COMPLETED``, ``HITL_*_REQUEST``, and
26632663
``HITL_*_REQUEST_COMPLETED`` events, so denying ``content`` via
26642664
``payload_column_denylist`` or filtering out any of those events via
@@ -5466,9 +5466,20 @@ def _parse_custom_metadata_allowlist(
54665466
# - USER_MESSAGE_RECEIVED becomes user text; TOOL_STARTING and HITL_*_REQUEST
54675467
# a function_call; AGENT_TRANSFER a ``transfer_to_agent`` function_call; and
54685468
# TOOL_COMPLETED and HITL_*_REQUEST_COMPLETED a function_response whose
5469-
# non-object result is wrapped as ``{"result": ...}``. The transfer and
5470-
# ``adk_request_*`` HITL tools' own TOOL_STARTING and TOOL_COMPLETED rows
5471-
# are skipped: AGENT_TRANSFER, logged after both, carries the handoff, and
5469+
# non-object result is wrapped as ``{"result": ...}``. TOOL_ERROR, which is
5470+
# logged instead of TOOL_COMPLETED when a tool raises, becomes a
5471+
# function_response of ``{"error": ...}`` holding the error message (or
5472+
# ``Tool execution failed`` when it has none), so every tool call that
5473+
# completes or raises has a response. ``after_tool_callback`` also logs
5474+
# TOOL_ERROR, for a result that reports a failure: that row records the
5475+
# error message it synthesized and leaves out the result the model saw, so
5476+
# the response holds that message, not the result. A TOOL_STARTING,
5477+
# TOOL_COMPLETED or TOOL_ERROR row that names no tool, as when a custom
5478+
# ``content_formatter`` fails, adds no event, and a HITL row that names none
5479+
# gets the name ``""``.
5480+
# The transfer and ``adk_request_*`` HITL tools' own
5481+
# TOOL_STARTING, TOOL_COMPLETED and TOOL_ERROR rows are skipped:
5482+
# AGENT_TRANSFER, logged after them, carries the handoff, and
54725483
# HITL_*_REQUEST and HITL_*_REQUEST_COMPLETED carry the HITL request and
54735484
# the completion that resumes it.
54745485
# - Model text comes from AGENT_RESPONSE (which strips thought parts).
@@ -5497,12 +5508,18 @@ def _parse_custom_metadata_allowlist(
54975508
# proto field names only, because unknown fields are rejected.
54985509
# - No output column is named ``status``, avoiding collisions with
54995510
# downstream evaluation output schemas.
5500-
# - Tool metrics (``tool_call_count``, ``distinct_tools_used`` and the
5501-
# consecutive-call columns) skip the ``transfer_to_agent`` tool, whose
5502-
# handoff the trace carries as the AGENT_TRANSFER function_call. Two calls
5511+
# - Tool metrics (``tool_call_count``, ``tool_error_count``,
5512+
# ``distinct_tools_used`` and the consecutive-call columns) skip the
5513+
# ``transfer_to_agent`` tool, whose handoff the trace carries as the
5514+
# AGENT_TRANSFER function_call, and the ``adk_request_credential``,
5515+
# ``adk_request_confirmation`` and ``adk_request_input`` tools, which the
5516+
# framework calls to pause for a credential, a confirmation or the user's
5517+
# input. A TOOL_STARTING row that names no tool is not a tool call, though
5518+
# a TOOL_ERROR row that names none still counts as a tool error. Two calls
55035519
# are identical when their tool name and args match; ``tool_call_island``
55045520
# numbers each run of consecutive identical calls, and ``has_tool_loop``
5505-
# means a run of 3 or more.
5521+
# means a run of 3 or more. A HITL request ends a run, even though the turn
5522+
# that resumes after it keeps its invocation_id.
55065523
# - ``eval_ready`` needs user text, a finished turn and at least one trace
55075524
# event: user input with no text (image-only or offloaded to GCS) leaves
55085525
# the judge without a prompt. In ``eval_session``, a paused turn (one that
@@ -5524,6 +5541,7 @@ def _parse_custom_metadata_allowlist(
55245541
"LLM_REQUEST",
55255542
"TOOL_STARTING",
55265543
"TOOL_COMPLETED",
5544+
"TOOL_ERROR",
55275545
"TOOL_PAUSED",
55285546
"AGENT_TRANSFER",
55295547
"HITL_CREDENTIAL_REQUEST",
@@ -7372,14 +7390,14 @@ async def create_eval_views(self) -> None:
73727390
The views rebuild ``AgentData`` traces from ``content``. When
73737391
``config.eval_view_mode`` is ``"full"`` (default), they require
73747392
``USER_MESSAGE_RECEIVED``, ``AGENT_STARTING``, ``LLM_REQUEST``,
7375-
``TOOL_STARTING``, ``TOOL_COMPLETED``, ``TOOL_PAUSED``, ``AGENT_TRANSFER``,
7376-
``AGENT_RESPONSE``, ``INVOCATION_COMPLETED``, ``HITL_*_REQUEST``, and
7377-
``HITL_*_REQUEST_COMPLETED`` events; when ``config.eval_view_mode`` is
7378-
``"conversation_only"``, they require only the three core conversation
7379-
events (``USER_MESSAGE_RECEIVED``, ``AGENT_RESPONSE``, and
7380-
``INVOCATION_COMPLETED``). Any custom ``content_formatter`` must preserve
7381-
``$.text_summary``, ``$.response``, ``$.system_prompt``, ``$.tool``,
7382-
``$.args``, and ``$.result``.
7393+
``TOOL_STARTING``, ``TOOL_COMPLETED``, ``TOOL_ERROR``, ``TOOL_PAUSED``,
7394+
``AGENT_TRANSFER``, ``AGENT_RESPONSE``, ``INVOCATION_COMPLETED``,
7395+
``HITL_*_REQUEST``, and ``HITL_*_REQUEST_COMPLETED`` events; when
7396+
``config.eval_view_mode`` is ``"conversation_only"``, they require only
7397+
the three core conversation events (``USER_MESSAGE_RECEIVED``,
7398+
``AGENT_RESPONSE``, and ``INVOCATION_COMPLETED``). Any custom
7399+
``content_formatter`` must preserve ``$.text_summary``, ``$.response``,
7400+
``$.system_prompt``, ``$.tool``, ``$.args``, and ``$.result``.
73837401
73847402
The views aggregate per session or turn, so a filter on their
73857403
``session_start_timestamp`` or ``turn_start_timestamp`` column does not

‎src/google/adk/plugins/bigquery_agent_analytics_sql/eval_view_ctes_v1.sql‎

Lines changed: 70 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -71,15 +71,26 @@ WITH
7171
AND invocation_id IS NOT NULL
7272
),
7373
-- Each event with the fields that later steps read parsed out of its JSON
74-
-- payload: its author, the tool name and args, the user text, the model
75-
-- response before it is unwrapped, and the token and latency numbers.
74+
-- payload: its author, the tool name and args, whether the tool is a
75+
-- framework tool, the user text, the model response before it is unwrapped,
76+
-- and the token and latency numbers.
7677
TypedEvents AS (
7778
SELECT
7879
*,
7980
COALESCE(
8081
agent, JSON_VALUE(attributes, '$.root_agent_name'), 'root_agent'
8182
) AS author,
8283
JSON_VALUE(content, '$.tool') AS tool_name,
84+
-- The framework tools are the transfer and the tools that pause the turn
85+
-- for a credential, a confirmation or input from the user. The trace
86+
-- shows their AGENT_TRANSFER and HITL rows instead of their tool rows.
87+
-- A row that names no tool is not a framework tool.
88+
COALESCE(JSON_VALUE(content, '$.tool'), '') IN (
89+
'transfer_to_agent',
90+
'adk_request_credential',
91+
'adk_request_confirmation',
92+
'adk_request_input'
93+
) AS is_framework_tool,
8394
TO_JSON_STRING(JSON_QUERY(content, '$.args')) AS tool_args_json,
8495
CASE
8596
WHEN event_type != 'USER_MESSAGE_RECEIVED' THEN NULL
@@ -167,15 +178,34 @@ WITH
167178
FROM
168179
RawEvents
169180
),
170-
-- Marks the real tool calls and unwraps the text of each model response.
171-
-- A transfer is not a tool call.
181+
-- Marks the real tool calls, the calls of tools other than the framework
182+
-- tools, and unwraps the text of each model response. It also counts the
183+
-- HITL requests so far in each turn.
172184
ResponseEvents AS (
173185
SELECT
174186
*,
187+
-- A TOOL_STARTING row that names no tool, as when a custom
188+
-- content_formatter fails, is not a tool call, since the trace could not
189+
-- pair it with its response.
175190
(
176191
event_type = 'TOOL_STARTING'
177-
AND COALESCE(tool_name, '') != 'transfer_to_agent'
192+
AND NULLIF(tool_name, '') IS NOT NULL
193+
AND NOT is_framework_tool
178194
) AS is_tool_call,
195+
-- The number of HITL requests so far in the turn. A turn that resumes
196+
-- after one keeps its invocation_id, so the count tells the calls before
197+
-- a pause from the calls after it.
198+
COUNTIF(
199+
event_type IN (
200+
'HITL_CREDENTIAL_REQUEST',
201+
'HITL_CONFIRMATION_REQUEST',
202+
'HITL_INPUT_REQUEST'
203+
)
204+
) OVER (
205+
PARTITION BY session_id, invocation_id
206+
ORDER BY timestamp, event_id
207+
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
208+
) AS hitl_request_count,
179209
-- The text parts that _format_content renders become plain text, and a
180210
-- response with any other part has no text.
181211
CASE
@@ -210,7 +240,9 @@ WITH
210240
),
211241
-- Adds two marks to each event: whether its turn has a final model
212242
-- response, which picks the rows that count tokens, and the island number
213-
-- that a run of consecutive identical tool calls shares.
243+
-- that a run of consecutive identical tool calls shares. The second
244+
-- ROW_NUMBER restarts after each HITL request, so the request ends a run,
245+
-- as a different call does.
214246
AnnotatedEvents AS (
215247
SELECT
216248
*,
@@ -222,7 +254,12 @@ WITH
222254
ORDER BY timestamp, event_id
223255
) - ROW_NUMBER() OVER (
224256
PARTITION BY
225-
session_id, invocation_id, is_tool_call, tool_name, tool_args_json
257+
session_id,
258+
invocation_id,
259+
is_tool_call,
260+
tool_name,
261+
tool_args_json,
262+
hitl_request_count
226263
ORDER BY timestamp, event_id
227264
) AS tool_call_island
228265
FROM
@@ -248,7 +285,8 @@ WITH
248285
response_text,
249286
is_function_response,
250287
is_tool_call,
251-
IF(is_tool_call, NULLIF(tool_name, ''), NULL) AS invoked_tool_name,
288+
is_framework_tool,
289+
IF(is_tool_call, tool_name, NULL) AS invoked_tool_name,
252290
IF(
253291
is_tool_call,
254292
COUNT(*) OVER (
@@ -286,6 +324,7 @@ WITH
286324
WHEN
287325
event_type IN (
288326
'TOOL_COMPLETED',
327+
'TOOL_ERROR',
289328
'HITL_CREDENTIAL_REQUEST_COMPLETED',
290329
'HITL_CONFIRMATION_REQUEST_COMPLETED',
291330
'HITL_INPUT_REQUEST_COMPLETED'
@@ -306,14 +345,7 @@ WITH
306345
'parts', JSON_ARRAY(JSON_OBJECT('text', message_text))
307346
)
308347
WHEN
309-
(
310-
is_tool_call
311-
AND COALESCE(tool_name, '') NOT IN (
312-
'adk_request_credential',
313-
'adk_request_confirmation',
314-
'adk_request_input'
315-
)
316-
)
348+
is_tool_call
317349
OR event_type IN (
318350
'HITL_CREDENTIAL_REQUEST',
319351
'HITL_CONFIRMATION_REQUEST',
@@ -352,12 +384,8 @@ WITH
352384
WHEN
353385
(
354386
event_type = 'TOOL_COMPLETED'
355-
AND COALESCE(tool_name, '') NOT IN (
356-
'transfer_to_agent',
357-
'adk_request_credential',
358-
'adk_request_confirmation',
359-
'adk_request_input'
360-
)
387+
AND NULLIF(tool_name, '') IS NOT NULL
388+
AND NOT is_framework_tool
361389
)
362390
OR event_type IN (
363391
'HITL_CREDENTIAL_REQUEST_COMPLETED',
@@ -379,6 +407,24 @@ WITH
379407
)
380408
)
381409
)
410+
WHEN
411+
event_type = 'TOOL_ERROR'
412+
AND NULLIF(tool_name, '') IS NOT NULL
413+
AND NOT is_framework_tool
414+
THEN JSON_OBJECT(
415+
'role', 'user',
416+
'parts', JSON_ARRAY(
417+
JSON_OBJECT(
418+
'function_response', JSON_OBJECT(
419+
'name', tool_name,
420+
'response', JSON_OBJECT(
421+
'error',
422+
COALESCE(NULLIF(error_message, ''), 'Tool execution failed')
423+
)
424+
)
425+
)
426+
)
427+
)
382428
WHEN event_type = 'AGENT_RESPONSE' AND response_text IS NOT NULL
383429
THEN JSON_OBJECT(
384430
'role', 'model',
@@ -408,7 +454,8 @@ WITH
408454
MIN(timestamp) AS turn_start_timestamp,
409455
MAX(timestamp) AS turn_end_timestamp,
410456
COUNTIF(is_tool_call) AS tool_call_count,
411-
COUNTIF(event_type = 'TOOL_ERROR') AS tool_error_count,
457+
COUNTIF(event_type = 'TOOL_ERROR' AND NOT is_framework_tool)
458+
AS tool_error_count,
412459
COUNTIF(event_type = 'LLM_ERROR') AS llm_error_count,
413460
COUNTIF(event_type = 'AGENT_ERROR') AS agent_error_count,
414461
COALESCE(

0 commit comments

Comments
 (0)