Repository navigation
Conversation
The efficiency metrics (token_usage_v1, inference_call_count_v1, invocation_duration_v1 and tool_call_count_v1) are informational and have no threshold. The multi-run aggregator only works out that lower is better from a threshold, so min_over_runs kept their lowest run score, which is the best run rather than the worst. Treat the metrics in _DEFAULT_EFFICIENCY_METRICS as lower is better when they have no threshold, and say so in the eval config guide. Fixes google#8165
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
With
cross_run_scoring="min_over_runs", the efficiency metrics (token_usage_v1,inference_call_count_v1,invocation_duration_v1andtool_call_count_v1) kept the score of their best run. Lower is better for them, somin_over_runsshould keep the highest score._judge_metriconly works out the direction from a threshold, and these metrics are informational and can't have one, sohigher_is_betterstayedTrue.Solution:
In
_judge_metric, when a metric is informational and is one of_DEFAULT_EFFICIENCY_METRICS, sethigher_is_better = False._computealready takes the highest run score formin_over_runswhen lower is better, and the method result gets the samelower is betternote that thresholded lower-is-better metrics get. The status staysINFORMATIONAL.The change only applies to those four metrics. Any other informational metric, and every metric with a threshold, is aggregated as before. I also added one sentence to the eval config guide, where it explains how the direction is worked out.
Testing Plan
Unit Tests:
New tests in
tests/unittests/evaluation/test_eval_case_result_multi_run_aggregator.py:test_min_over_runs_takes_the_highest_score_of_an_efficiency_metric, run once for each of the four metrics.test_min_over_runs_by_metric_takes_the_highest_score_of_an_efficiency_metric, the same withcross_run_scoring_by_metric.test_min_over_runs_takes_the_lowest_score_of_other_informational_metrics, which checks that other informational metrics still keep the lowest score.Without the fix, the first five fail and the last one passes. With the fix, all of them pass.
pre-commit run --filespasses on the changed files, andmypyreports no issues for the changed module.Manual End-to-End (E2E) Tests:
I ran the reproduction script from #8165, which aggregates two runs of the same case that used 100 and 5000 tokens.
Before:
After:
Checklist
Additional context
None.