Skip to content

fix(eval): treat efficiency metrics as lower is better in min_over_runs - #8166

Open
raulEspaa wants to merge 1 commit into
google:mainfrom
raulEspaa:fix/min-over-runs-efficiency-metrics
Open

raulEspaa wants to merge 1 commit into
google:mainfrom
raulEspaa:fix/min-over-runs-efficiency-metrics

Conversation

@raulEspaa

Copy link
Copy Markdown

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
With cross_run_scoring="min_over_runs", the efficiency metrics (token_usage_v1, inference_call_count_v1, invocation_duration_v1 and tool_call_count_v1) kept the score of their best run. Lower is better for them, so min_over_runs should keep the highest score. _judge_metric only works out the direction from a threshold, and these metrics are informational and can't have one, so higher_is_better stayed True.

Solution:
In _judge_metric, when a metric is informational and is one of _DEFAULT_EFFICIENCY_METRICS, set higher_is_better = False. _compute already takes the highest run score for min_over_runs when lower is better, and the method result gets the same lower is better note that thresholded lower-is-better metrics get. The status stays INFORMATIONAL.

The change only applies to those four metrics. Any other informational metric, and every metric with a threshold, is aggregated as before. I also added one sentence to the eval config guide, where it explains how the direction is worked out.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests in tests/unittests/evaluation/test_eval_case_result_multi_run_aggregator.py:

  • test_min_over_runs_takes_the_highest_score_of_an_efficiency_metric, run once for each of the four metrics.
  • test_min_over_runs_by_metric_takes_the_highest_score_of_an_efficiency_metric, the same with cross_run_scoring_by_metric.
  • test_min_over_runs_takes_the_lowest_score_of_other_informational_metrics, which checks that other informational metrics still keep the lowest score.

Without the fix, the first five fail and the last one passes. With the fix, all of them pass.

$ pytest tests/unittests/evaluation/test_eval_case_result_multi_run_aggregator.py
88 passed in 0.73s

$ pytest tests/unittests/evaluation
1337 passed, 41 warnings in 16.39s

pre-commit run --files passes on the changed files, and mypy reports no issues for the changed module.

Manual End-to-End (E2E) Tests:

I ran the reproduction script from #8165, which aggregates two runs of the same case that used 100 and 5000 tokens.

Before:

token_usage_v1 100.0
response_match_score 0.6

After:

token_usage_v1 5000.0
response_match_score 0.6

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

None.

The efficiency metrics (token_usage_v1, inference_call_count_v1,
invocation_duration_v1 and tool_call_count_v1) are informational and have
no threshold. The multi-run aggregator only works out that lower is better
from a threshold, so min_over_runs kept their lowest run score, which is the
best run rather than the worst.

Treat the metrics in _DEFAULT_EFFICIENCY_METRICS as lower is better when
they have no threshold, and say so in the eval config guide.

Fixes google#8165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

min_over_runs keeps the best run, not the worst, for token_usage_v1 and the other efficiency metrics

2 participants