Repository navigation
Conversation
When launcher.nsys_profile is set, package() wraps the role with nsys and always appends the postfix returned by get_nsys_entrypoint(). The base executor returns an empty postfix, and so does SlurmExecutor unless nsys_gpu_metrics is set. Executors that join the args into a shell string only end up with a trailing space, but the local scheduler hands the args list straight to subprocess, so the user's program receives a real empty argument. An argparse script run through LocalExecutor with Launcher(nsys_profile=True) then exits with "unrecognized arguments". Only append the postfix when it is non-empty. The Slurm GPU metrics case, which needs the closing quote, is unchanged. The NVCRE scheduler stripped the empty entry itself; it now strips the postfix only when there is one, so a trailing empty argument the user passed on purpose is kept. Signed-off-by: Dmitry Voropaev <dy.voropaev@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
Launcher(nsys_profile=True),package()wraps the role with nsys and then always appends the postfix fromget_nsys_entrypoint(). That postfix is an empty string for the base executor, and forSlurmExecutorunlessnsys_gpu_metricsis set. Slurm, Docker and the other shell-based executors join the args into one string, so the extra''is only a trailing space there. The local scheduler passes the args list tosubprocess.Popenas is, so the script gets a real empty argument. The Kubeflow executor without a workdir PVC also puts the list straight into the container command. For example,run.run(run.Script(path="train.py", args=["--lr", "0.1"], entrypoint="python"), executor=run.LocalExecutor(launcher=Launcher(nsys_profile=True)))gives train.pysys.argv[1:] == ['--lr', '0.1', ''], and argparse exits with "unrecognized arguments".The fix appends the postfix only when it is non-empty, so the Slurm GPU metrics case that needs the closing
'behaves as before. The NVCRE scheduler already worked around the empty entry by dropping the last arg when it equaled the postfix. With the empty entry gone, that check would drop a trailing''the user passed on purpose, so it now runs only when the postfix is non-empty. The NVCRE test that asserted the old empty last argument is updated to match, and a new NVCRE test checks that a Script ending in''keeps it. That test passes on main and fails if only the packaging change is applied.I added two tests in
test/run/torchx_backend/test_packaging.py. One packages a Script withLocalExecutor(launcher=Launcher(nsys_profile=True))and checks that the args end with the user's command and contain no''. It fails on main and passes with the change. The other checks that a non-empty postfix is still appended.pytest test/run/torchx_backend/test_packaging.py test/run/torchx_backend/schedulers/test_nvcre.py test/core/execution/test_nvcre.py test/core/execution/test_slurm.py test/core/execution/test_slurm_templates.py test/core/execution/test_docker.py test/core/execution/test_local.py test/run/test_job.py test/run/test_experiment.pygives 476 passed.ruff checkandruff format --diff(0.9.10, the version in uv.lock) are clean.I also ran the example above end to end on a Mac with a stub
nsyson PATH that drops its own flags and execs the rest. On main the script printedargv: ['--lr', '0.1', '']and then the argparse error. With the change it printedargv: ['--lr', '0.1']andparsed: Namespace(lr=0.1).