Skip to content

fix(packaging): don't append an empty nsys postfix argument - #677

Open
v0ropaev wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
v0ropaev:fix/nsys-empty-postfix
Open

v0ropaev wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
v0ropaev:fix/nsys-empty-postfix

Conversation

@v0ropaev

Copy link
Copy Markdown
Contributor

With Launcher(nsys_profile=True), package() wraps the role with nsys and then always appends the postfix from get_nsys_entrypoint(). That postfix is an empty string for the base executor, and for SlurmExecutor unless nsys_gpu_metrics is set. Slurm, Docker and the other shell-based executors join the args into one string, so the extra '' is only a trailing space there. The local scheduler passes the args list to subprocess.Popen as is, so the script gets a real empty argument. The Kubeflow executor without a workdir PVC also puts the list straight into the container command. For example, run.run(run.Script(path="train.py", args=["--lr", "0.1"], entrypoint="python"), executor=run.LocalExecutor(launcher=Launcher(nsys_profile=True))) gives train.py sys.argv[1:] == ['--lr', '0.1', ''], and argparse exits with "unrecognized arguments".

The fix appends the postfix only when it is non-empty, so the Slurm GPU metrics case that needs the closing ' behaves as before. The NVCRE scheduler already worked around the empty entry by dropping the last arg when it equaled the postfix. With the empty entry gone, that check would drop a trailing '' the user passed on purpose, so it now runs only when the postfix is non-empty. The NVCRE test that asserted the old empty last argument is updated to match, and a new NVCRE test checks that a Script ending in '' keeps it. That test passes on main and fails if only the packaging change is applied.

I added two tests in test/run/torchx_backend/test_packaging.py. One packages a Script with LocalExecutor(launcher=Launcher(nsys_profile=True)) and checks that the args end with the user's command and contain no ''. It fails on main and passes with the change. The other checks that a non-empty postfix is still appended. pytest test/run/torchx_backend/test_packaging.py test/run/torchx_backend/schedulers/test_nvcre.py test/core/execution/test_nvcre.py test/core/execution/test_slurm.py test/core/execution/test_slurm_templates.py test/core/execution/test_docker.py test/core/execution/test_local.py test/run/test_job.py test/run/test_experiment.py gives 476 passed. ruff check and ruff format --diff (0.9.10, the version in uv.lock) are clean.

I also ran the example above end to end on a Mac with a stub nsys on PATH that drops its own flags and execs the rest. On main the script printed argv: ['--lr', '0.1', ''] and then the argparse error. With the change it printed argv: ['--lr', '0.1'] and parsed: Namespace(lr=0.1).

When launcher.nsys_profile is set, package() wraps the role with nsys and
always appends the postfix returned by get_nsys_entrypoint(). The base
executor returns an empty postfix, and so does SlurmExecutor unless
nsys_gpu_metrics is set. Executors that join the args into a shell string
only end up with a trailing space, but the local scheduler hands the args
list straight to subprocess, so the user's program receives a real empty
argument. An argparse script run through LocalExecutor with
Launcher(nsys_profile=True) then exits with "unrecognized arguments".

Only append the postfix when it is non-empty. The Slurm GPU metrics case,
which needs the closing quote, is unchanged. The NVCRE scheduler stripped
the empty entry itself; it now strips the postfix only when there is one,
so a trailing empty argument the user passed on purpose is kept.

Signed-off-by: Dmitry Voropaev <dy.voropaev@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants