Repository navigation
Conversation
parse_factory split the arguments of a factory call on every comma, so model=my_model(layers=[1,2,3]) reached parse_cli_args as 'layers=[1', '2' and '3]', and a quoted value such as activation='a,b' was cut in half the same way. The stray pieces were treated as positional arguments after a keyword argument, and the whole value then failed with "Unsupported type". Use PythonicParser._split_constructor_args, which already tracks brackets and quotes for dict()/list() constructors. Empty pieces are still dropped, so a trailing ", " or "( )" keeps working as before. Signed-off-by: Dmitry Voropaev <dy.voropaev@gmail.com>
|
/ok to test 2fd1b2a |
|
/review-approve |
|
/review-approve |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
One correctness regression remains in factory argument splitting: a supported scalar string can now silently disappear, causing the factory to use its default instead. Reviewed both changed files and traced the shared splitter and downstream type parsing. Validation was static; tests were not executed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| if args_str: | ||
| cli_args = [arg.strip() for arg in args_str.split(",") if arg.strip()] | ||
| # Split on top-level commas only, so lists and quoted strings stay whole. | ||
| cli_args = [arg for arg in PythonicParser()._split_constructor_args(args_str) if arg] |
There was a problem hiding this comment.
[P2] Preserve an unfinished scalar tail instead of silently dropping it
Failure: Using layered_model and func from the added test, call:
result = parse_cli_args(func, ["model=layered_model(activation=relu[)"])The expected activation is "relu[": string parsing accepts unquoted text, and the base implementation passed it through. Now the unmatched literal bracket keeps the splitter at nonzero depth through its synthetic final comma. It returns [], so the factory runs without the override and silently returns activation "relu". Unfinished quotes or unmatched closing delimiters can likewise discard the tail.
Correction: Make _split_constructor_args lossless at EOF, retaining the empty-piece filter here. A small suggested addition immediately before its return parts is:
if current_arg:
parts.append(current_arg[:-1].strip())The slice removes exactly the synthetic comma introduced by text = args + ",". This restores the invariant that a supplied tail reaches downstream type parsing rather than becoming an omitted argument. Balanced arguments still flush through the existing branch. Please verify this suggestion before adopting it; it has not been executed.
Preservation and acceptance: Add the invocation above and assert result.model.activation == "relu[". Separately, model=layered_model(layers=[1) must raise a parsing error rather than silently use the default layers: literal delimiters are valid string data, but an incomplete list must remain invalid. Retain the new comma-containing cases and their hidden == 6, hidden == 9, activation == "a,b", and trailing-comma hidden == 3 assertions. Also retain the shared constructor regressions, especially list(it's fine, 2) == ["it's fine", 2] and list(r'a,b', 2) == ["a,b", 2]. These are proposed/static acceptance checks, not reported test results.
There was a problem hiding this comment.
Right, the splitter dropped whatever was left when the last comma was not at level zero. I made it flush the tail at the end, and an unmatched closing bracket no longer pushes the level below zero, so activation=relu), layers=[2] splits the way the plain comma split did. layered_model(activation=relu[) now gives "relu[" and layered_model(layers=[1) raises a ParseError, both are in the test now.
_split_constructor_args only flushed a piece on a comma at nesting level zero. An unclosed bracket or quote, or an unmatched closing bracket, kept the final comma from splitting, so everything after the last split was dropped. Through parse_factory that meant layered_model(activation=relu[) ran the factory without the override and layered_model(layers=[1) silently used the default layers. Flush whatever is left at the end, without the comma added for the last split, and do not let an unmatched closing bracket push the level below zero. Balanced input splits as before. Signed-off-by: Dmitry Voropaev <dy.voropaev@gmail.com>
Head branch was pushed to by a user without write access
A factory argument that contains a comma can't be passed on the command line today. parse_factory splits whatever sits inside the parentheses on every comma, so
model=my_model(layers=[1,2,3])goes to parse_cli_args aslayers=[1,2and3], andmodel=my_model(activation='a,b')becomesactivation='aandb'. The extra pieces look like positional arguments after a keyword argument. parse_cli_args catches that exception from parse_factory and falls back to parse_value, so the run ends with "Unsupported type: <class 'main.Model'>". The real cause only shows up as one logged "Positional argument found after keyword argument" line above it. Calls likemy_model(hidden_size=1024,activation='tanh')from examples/entrypoint/README.md work only because none of their values contain a comma.The fix reuses PythonicParser._split_constructor_args for the split. It already tracks brackets and quotes for dict()/list() constructors (quote handling came in #618). I kept the filter that drops empty pieces, because the splitter returns an empty string for a trailing
,or for( ), and both of those parse fine on main.The new test in test/cli/test_cli_parser.py registers a small factory in
__main__and checks a list argument, a quoted string with a comma, both together, and a trailing,. It fails on main and passes with the change.pytest test/cligives 367 passed and 1 failed. The failure is test_run_context_execute_task_with_repl, which needs IPython and is not installed in my environment. I also ran a smallrun.cli.mainscript with--yes --directand the argumentsmodel=my_model(layers=[1,2,3])andmodel=my_model(activation='a,b'). On main both fail as described above. With the change they printModel(hidden=256, layers=[1, 2, 3], activation='relu')andModel(hidden=256, layers=[], activation='a,b'), and the README call still works. ruff 0.6.1 check and format are clean.