Skip to content

Eval judges drop their own reasoning (final_response_match_v2, hallucinations_v1) #8184

Open

Description

@krisztianfekete

Describe the Bug:
final_response_match_v2 and hallucinations_v1 throw away the judge's reasoning, so a failed eval returns only a score. You can't tell why it failed without calling the judge again by hand.

  • final_response_match_v2 asks the judge for {"reasoning": ..., "is_the_agent_response_valid": ...}, but convert_auto_rater_response_to_score keeps only the label and returns AutoRaterScore(score=1.0|0.0) with no rubric_scores.
  • hallucinations_v1 already parses a label, rationale and supporting or contradicting excerpts for each sentence (_parse_validation_results), then returns rubric_scores=[] (L801, L838).

The rubric_based_* metrics already return their reasoning in PerInvocationResult.rubric_scores / RubricScore.rationale. Since #7366 (fixing #7350), the Vertex-backed metrics do too. These two ADK-native judges are what's left.

Steps to Reproduce:

  1. pip install "google-adk[eval]" (also reproduced on main at c6586f19ab)
  2. Save the script under Minimal Reproduction Code as repro.py. It uses a scripted judge registered through LLMRegistry, so it needs no network calls or credentials.
  3. Run python repro.py.
  4. Look at rubric_scores for both metrics.

Expected Behavior:
Each result carries the judge's reasoning in rubric_scores, as the rubric-based and Vertex-backed metrics already do:

  • final_response_match_v2: one RubricScore holding the judge's reasoning, with the same score as the label.
  • hallucinations_v1: one RubricScore per sentence holding its label score and rationale.

Scores, statuses and aggregation stay as they are.

Observed Behavior:
Both metrics score correctly (0.0, FAILED), but the reasoning the judge returned is gone:

final_response_match_v2: 0.0 FAILED rubric_scores = None
hallucinations_v1: 0.0 FAILED rubric_scores = []

Environment Details:

  • ADK Library Version (pip show google-adk): 2.14.0; also main at c6586f19ab8a2bb8b53e70041633b10858d9902e
  • Desktop OS: Linux
  • Python Version (python -V): 3.14.0 (on 2.14.0), 3.13 (on main)

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A (scripted judge; the behavior does not depend on the model)

🟡 Optional Information

Regression:
No. These two metrics have never returned the judge's reasoning.

Logs:
N/A, the output above is the whole result.

Screenshots / Video:
N/A

Additional Context:

Minimal Reproduction Code:

import asyncio

from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric, HallucinationsCriterion, JudgeModelOptions, LlmAsAJudgeCriterion
from google.adk.evaluation.final_response_match_v2 import FinalResponseMatchV2Evaluator
from google.adk.evaluation.hallucinations_v1 import HallucinationsV1Evaluator
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_response import LlmResponse
from google.adk.models.registry import LLMRegistry
from google.genai import types


class ScriptedJudge(BaseLlm):
  @classmethod
  def supported_models(cls):
    return [r"scripted-judge"]

  async def generate_content_async(self, llm_request, stream=False):
    prompt = llm_request.contents[-1].parts[0].text
    if "is_the_agent_response_valid" in prompt:
      text = '{"reasoning": "The agent says Lyon; the reference says Paris.", "is_the_agent_response_valid": "invalid"}'
    elif "label:" not in prompt:
      text = "<sentence>The capital of France is Lyon.</sentence>"
    else:
      text = (
          "sentence: The capital of France is Lyon.\nlabel: contradictory\n"
          "rationale: The context states the capital is Paris.\n"
          "supporting_excerpt: null\ncontradicting_excerpt: The capital of France is Paris."
      )
    yield LlmResponse(content=types.Content(role="model", parts=[types.Part(text=text)]))


LLMRegistry.register(ScriptedJudge)


def invocation(text):
  return Invocation(
      invocation_id="inv-1",
      user_content=types.Content(role="user", parts=[types.Part(text="What is the capital of France?")]),
      final_response=types.Content(role="model", parts=[types.Part(text=text)]),
  )


async def main():
  opts = JudgeModelOptions(judge_model="scripted-judge", num_samples=1)

  frm = FinalResponseMatchV2Evaluator(
      EvalMetric(metric_name="final_response_match_v2", criterion=LlmAsAJudgeCriterion(threshold=0.5, judge_model_options=opts))
  )
  r = await frm.evaluate_invocations([invocation("The capital of France is Lyon.")], [invocation("The capital of France is Paris.")])
  print("final_response_match_v2:", r.overall_score, r.overall_eval_status.name,
        "rubric_scores =", r.per_invocation_results[0].rubric_scores)

  hal = HallucinationsV1Evaluator(
      EvalMetric(metric_name="hallucinations_v1", criterion=HallucinationsCriterion(threshold=0.5, judge_model_options=opts))
  )
  r = await hal.evaluate_invocations([invocation("The capital of France is Lyon.")], None)
  print("hallucinations_v1:", r.overall_score, r.overall_eval_status.name,
        "rubric_scores =", r.per_invocation_results[0].rubric_scores)


asyncio.run(main())

How often has this issue occurred?:

  • Always (100%)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions