Skip to content

fix(eval): stop passing deprecated plugins= to Runner when no App is given - #8170

Open
jayanti-prajapati wants to merge 1 commit into
google:mainfrom
jayanti-prajapati:fix/eval-runner-deprecated-plugins
Open

jayanti-prajapati wants to merge 1 commit into
google:mainfrom
jayanti-prajapati:fix/eval-runner-deprecated-plugins

Conversation

@jayanti-prajapati

Copy link
Copy Markdown

Link to Issue or Description of Change

No existing issue; small fix.

Problem:
When an eval runs without an App, _build_eval_runner_kwargs in evaluation/evaluation_generator.py builds the Runner with agent= and the deprecated plugins= argument. Every such run therefore emits:

DeprecationWarning: The `plugins` argument is deprecated. Please use the `app` argument to provide plugins instead.

It shows up in the warnings summary of any pytest run that uses AgentEvaluator.evaluate with a bare agent.

Solution:
Wrap the bare agent in an App that holds the internal eval plugins and pass app= instead.

  • It uses App.model_construct(...), which is exactly what Runner._resolve_app builds for the legacy (app_name, agent, plugins) path. Runner behaviour is unchanged; only the warning goes away.
  • model_construct rather than App(...) keeps eval app names working when they are not valid App names (eval app names come from the eval session, e.g. session_input.app_name), matching the legacy path.
  • The App-provided path is untouched. Two kwarg constants that became unused (_AGENT_KEY, _PLUGINS_KEY) are removed.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.

  • All unit tests pass locally.

  • test_runner_falls_back_to_bare_agent_when_no_app asserted the legacy agent=/plugins= shape. Renamed to test_runner_wraps_bare_agent_in_app_when_no_app; it now asserts app= with the eval plugins and no agent=/plugins=.

  • New TestBuildEvalRunnerKwargs::test_no_app_builds_runner_without_deprecation_warning builds a real Runner from the kwargs with DeprecationWarning turned into an error, for my_app and my-agent.evalset. Both cases fail before the fix and pass after.

pytest tests/unittests/evaluation tests/unittests/test_runners.py tests/unittests/cli
3249 passed, 5 skipped, 1 xfailed

Manual End-to-End (E2E) Tests:

A small offline AgentEvaluator.evaluate test (bare LlmAgent, scripted BaseLlm, no API key), run with pytest -rw:

  • before: tests pass, warnings summary includes the plugins deprecation warning
  • after: tests pass, no plugins deprecation warning

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

…s given

When an eval runs without an App, `_build_eval_runner_kwargs` built the
Runner with `agent=` and the deprecated `plugins=` argument, so every such
run emitted "The `plugins` argument is deprecated" (visible in pytest runs
that use AgentEvaluator).

Wrap the bare agent in an App holding the internal eval plugins instead.
Like Runner's own wrapping of a bare agent, this uses `App.model_construct`
so eval app names that are not valid App names keep working. Runner
behaviour is otherwise unchanged: both paths resolve to the same App.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants