Repository navigation
fix(eval): stop passing deprecated plugins= to Runner when no App is given - #8170
Open
jayanti-prajapati wants to merge 1 commit into
Open
jayanti-prajapati wants to merge 1 commit into
jayanti-prajapati wants to merge 1 commit into
Conversation
…s given When an eval runs without an App, `_build_eval_runner_kwargs` built the Runner with `agent=` and the deprecated `plugins=` argument, so every such run emitted "The `plugins` argument is deprecated" (visible in pytest runs that use AgentEvaluator). Wrap the bare agent in an App holding the internal eval plugins instead. Like Runner's own wrapping of a bare agent, this uses `App.model_construct` so eval app names that are not valid App names keep working. Runner behaviour is otherwise unchanged: both paths resolve to the same App.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
No existing issue; small fix.
Problem:
When an eval runs without an
App,_build_eval_runner_kwargsinevaluation/evaluation_generator.pybuilds theRunnerwithagent=and the deprecatedplugins=argument. Every such run therefore emits:It shows up in the warnings summary of any pytest run that uses
AgentEvaluator.evaluatewith a bare agent.Solution:
Wrap the bare agent in an
Appthat holds the internal eval plugins and passapp=instead.App.model_construct(...), which is exactly whatRunner._resolve_appbuilds for the legacy(app_name, agent, plugins)path. Runner behaviour is unchanged; only the warning goes away.model_constructrather thanApp(...)keeps eval app names working when they are not valid App names (eval app names come from the eval session, e.g.session_input.app_name), matching the legacy path.App-provided path is untouched. Two kwarg constants that became unused (_AGENT_KEY,_PLUGINS_KEY) are removed.Testing Plan
Unit Tests:
I have added or updated unit tests for my change.
All unit tests pass locally.
test_runner_falls_back_to_bare_agent_when_no_appasserted the legacyagent=/plugins=shape. Renamed totest_runner_wraps_bare_agent_in_app_when_no_app; it now assertsapp=with the eval plugins and noagent=/plugins=.New
TestBuildEvalRunnerKwargs::test_no_app_builds_runner_without_deprecation_warningbuilds a realRunnerfrom the kwargs withDeprecationWarningturned into an error, formy_appandmy-agent.evalset. Both cases fail before the fix and pass after.Manual End-to-End (E2E) Tests:
A small offline
AgentEvaluator.evaluatetest (bareLlmAgent, scriptedBaseLlm, no API key), run withpytest -rw:pluginsdeprecation warningpluginsdeprecation warningChecklist