Repository navigation
Description
Describe the Bug:
final_response_match_v2 and hallucinations_v1 throw away the judge's reasoning, so a failed eval returns only a score. You can't tell why it failed without calling the judge again by hand.
final_response_match_v2asks the judge for{"reasoning": ..., "is_the_agent_response_valid": ...}, butconvert_auto_rater_response_to_scorekeeps only the label and returnsAutoRaterScore(score=1.0|0.0)with norubric_scores.hallucinations_v1already parses alabel,rationaleand supporting or contradicting excerpts for each sentence (_parse_validation_results), then returnsrubric_scores=[](L801, L838).
The rubric_based_* metrics already return their reasoning in PerInvocationResult.rubric_scores / RubricScore.rationale. Since #7366 (fixing #7350), the Vertex-backed metrics do too. These two ADK-native judges are what's left.
Steps to Reproduce:
pip install "google-adk[eval]"(also reproduced on main atc6586f19ab)- Save the script under Minimal Reproduction Code as
repro.py. It uses a scripted judge registered throughLLMRegistry, so it needs no network calls or credentials. - Run
python repro.py. - Look at
rubric_scoresfor both metrics.
Expected Behavior:
Each result carries the judge's reasoning in rubric_scores, as the rubric-based and Vertex-backed metrics already do:
final_response_match_v2: oneRubricScoreholding the judge'sreasoning, with the same score as the label.hallucinations_v1: oneRubricScoreper sentence holding its label score and rationale.
Scores, statuses and aggregation stay as they are.
Observed Behavior:
Both metrics score correctly (0.0, FAILED), but the reasoning the judge returned is gone:
final_response_match_v2: 0.0 FAILED rubric_scores = None
hallucinations_v1: 0.0 FAILED rubric_scores = []
Environment Details:
- ADK Library Version (pip show google-adk): 2.14.0; also main at
c6586f19ab8a2bb8b53e70041633b10858d9902e - Desktop OS: Linux
- Python Version (python -V): 3.14.0 (on 2.14.0), 3.13 (on main)
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A (scripted judge; the behavior does not depend on the model)
🟡 Optional Information
Regression:
No. These two metrics have never returned the judge's reasoning.
Logs:
N/A, the output above is the whole result.
Screenshots / Video:
N/A
Additional Context:
- fix(eval): retry unscored hallucinations_v1 judge outcomes #8122 (open) adds
RubricScore(rubric_id="error")whenhallucinations_v1cannot score a response. This issue is about the reasoning on responses that are scored, so the two are complementary. - The description of Vertex-backed eval metrics drop rubric verdicts and explanations (multi_turn_*_v1, safety_v1) #7350 says these two judges already return their reasoning in
rubric_scores; on main they don't. - For
final_response_match_v2, the majority vote returns a whole winning sample, so a rationale attached to each sample stays consistent with the reported verdict. - Happy to send a PR if this direction sounds right.
Minimal Reproduction Code:
import asyncio
from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric, HallucinationsCriterion, JudgeModelOptions, LlmAsAJudgeCriterion
from google.adk.evaluation.final_response_match_v2 import FinalResponseMatchV2Evaluator
from google.adk.evaluation.hallucinations_v1 import HallucinationsV1Evaluator
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_response import LlmResponse
from google.adk.models.registry import LLMRegistry
from google.genai import types
class ScriptedJudge(BaseLlm):
@classmethod
def supported_models(cls):
return [r"scripted-judge"]
async def generate_content_async(self, llm_request, stream=False):
prompt = llm_request.contents[-1].parts[0].text
if "is_the_agent_response_valid" in prompt:
text = '{"reasoning": "The agent says Lyon; the reference says Paris.", "is_the_agent_response_valid": "invalid"}'
elif "label:" not in prompt:
text = "<sentence>The capital of France is Lyon.</sentence>"
else:
text = (
"sentence: The capital of France is Lyon.\nlabel: contradictory\n"
"rationale: The context states the capital is Paris.\n"
"supporting_excerpt: null\ncontradicting_excerpt: The capital of France is Paris."
)
yield LlmResponse(content=types.Content(role="model", parts=[types.Part(text=text)]))
LLMRegistry.register(ScriptedJudge)
def invocation(text):
return Invocation(
invocation_id="inv-1",
user_content=types.Content(role="user", parts=[types.Part(text="What is the capital of France?")]),
final_response=types.Content(role="model", parts=[types.Part(text=text)]),
)
async def main():
opts = JudgeModelOptions(judge_model="scripted-judge", num_samples=1)
frm = FinalResponseMatchV2Evaluator(
EvalMetric(metric_name="final_response_match_v2", criterion=LlmAsAJudgeCriterion(threshold=0.5, judge_model_options=opts))
)
r = await frm.evaluate_invocations([invocation("The capital of France is Lyon.")], [invocation("The capital of France is Paris.")])
print("final_response_match_v2:", r.overall_score, r.overall_eval_status.name,
"rubric_scores =", r.per_invocation_results[0].rubric_scores)
hal = HallucinationsV1Evaluator(
EvalMetric(metric_name="hallucinations_v1", criterion=HallucinationsCriterion(threshold=0.5, judge_model_options=opts))
)
r = await hal.evaluate_invocations([invocation("The capital of France is Lyon.")], None)
print("hallucinations_v1:", r.overall_score, r.overall_eval_status.name,
"rubric_scores =", r.per_invocation_results[0].rubric_scores)
asyncio.run(main())How often has this issue occurred?:
- Always (100%)
Activity