Skip to content

fix(experiment): keep job dependencies across Experiment.reset - #691

Open
udsy19 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
udsy19:fix/experiment-reset-dependencies
Open

udsy19 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
udsy19:fix/experiment-reset-dependencies

Conversation

@udsy19

@udsy19 udsy19 commented Oct 31, 2026

Copy link
Copy Markdown

What

Experiment.reset() rebuilds the experiment by re-adding each saved job with self.add(...), passing the task/executor, name and tail_logs but not dependencies. The reloaded Job / JobGroup objects still hold their dependencies (they're serialized and restored correctly), but every rebuilt job ends up with dependencies == [].

So relaunching a finished experiment with dependent jobs (e.g. train → eval, Experiment.from_id(...).reset().run()) flattens the dependency graph to one level and launches every job at once, with no error or warning.

with Experiment("exp") as exp:
    j1 = exp.add(run.Partial(f), name="job1")
    exp.add(run.Partial(g), name="job2", dependencies=[j1])
...
r = Experiment.from_id(exp._id); r.reset()
r.jobs[1].dependencies   # [] (expected ["job1"])

Fix

Pass dependencies=job.dependencies in both reset() branches (single jobs and job groups). Jobs are re-added in their saved order, so each dependency already exists when add() validates it.

Tests

test_reset_preserves_job_dependencies in test/run/test_experiment.py covers a single job and a job group that depend on another job. It fails on main (assert [] == ['job1']) and passes with the fix; test/run/test_experiment.py and test/run/test_job.py pass (129).

Experiment.reset() rebuilds the job list by calling self.add() for every
saved job, but it only forwarded the task, executor, name and tail_logs.
The dependencies stored on each Job/JobGroup were dropped, so a relaunched
experiment treated every job as independent and launched dependent jobs in
parallel instead of after the jobs they depend on.

Pass each job's dependencies through to add(). Jobs are re-added in their
saved order, so every dependency id is already present when it is checked.

Signed-off-by: Udaya Tejas <udayatejas2004@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants