Skip to content

[DMD2 10/10, view on #372] feat(dmd2): train_dmd2.py and --dmd2-critic-scoring original - #375

Open
zhihengy wants to merge 5 commits into
dmd2/06b-h3-dmd2-e2efrom
dmd2/07b-view-on-372
Open

zhihengy wants to merge 5 commits into
dmd2/06b-h3-dmd2-e2efrom
dmd2/07b-view-on-372

Conversation

@zhihengy

Copy link
Copy Markdown
Collaborator

View-only copy of #351 against its real base, #372 (dmd2/08-teacher-in-trunk), so the diff shows this PR's five commits alone. GitHub refuses to retarget #351 (it is inside the #330..#343 stack), whose diff against dmd2/06-h3-recipe includes #367, #368 and #372 until they merge. Review and merge happen on #351; this PR is closed once that diff is clean.

The branch dmd2/07b-view-on-372 is the same commits as dmd2/07-original-scoring (a6e8635). The description of the change is on #351.

@zhihengy
zhihengy force-pushed the dmd2/07b-view-on-372 branch from a6e8635 to 36222f1 Compare November 11, 2026 11:54
@Rockdu
Rockdu force-pushed the dmd2/07b-view-on-372 branch 2 times, most recently from 8732fa8 to 676119c Compare November 11, 2026 13:16
@Rockdu
Rockdu changed the base branch from dmd2/08-teacher-in-trunk to dmd2/06b-h3-dmd2-e2e November 11, 2026 13:16
@Rockdu
Rockdu added this pull request to stack #374 November 11, 2026 13:41
@Rockdu
Rockdu removed this pull request from stack #374 November 11, 2026 15:54
@Rockdu
Rockdu added this pull request to stack #379 November 11, 2026 15:55
@Rockdu
Rockdu force-pushed the dmd2/07b-view-on-372 branch from 676119c to 0992d73 Compare November 11, 2026 17:00
zhihengy and others added 5 commits November 11, 2026 09:01
A group that trains and then scores in the same wake would wake twice, and waking a frozen model twice replaces its
pinned host copies with its GPU parameters, so the next sleep would leave the teacher on the GPU.

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
`train(..., optim_steps=(lo, hi))` runs windows lo..hi-1 of the rollout's micro-batch schedule with the micro-batch
ids of the full run, so a range draws the same noise as the whole rollout would; a range outside the schedule, or
one under --diffusion-recompute-old-log-prob, is rejected. The EMA steps once per rollout, by the call whose range
ends at the last step. `RayTrainGroup.async_train` passes the range through.

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
`FSDPTrainRayActor.compute(rollout_id, rollout_data_ref, external_data, hooks_key)` runs the loss hub's
`(--loss-type, hooks_key)` hooks without grad: their train-input selection, one forward per micro-batch through the
trunk (`_forward_train_pair_batch` takes the hooks to run instead of the group's), numbered past the rollout's
training micro-batches so no training forward shares a draw, then their train-output hook, boxed as train's output
is. The models are resharded after; the optimizer, scheduler, EMA and step count stay. `RayTrainGroup.async_compute`
dispatches it per rank like `async_train`.

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
…ate scoring forward

`reuse-critic-step` (default, no behavior change): the critic's last training step draws in the scoring range, asks
for the teacher and keeps both x0 as the score. `original`: that step is plain SFT and keeps nothing; the new
`("dmd2", "critic-scoring")` hooks, run forward-only by `FSDPTrainRayActor.compute`, score the last step's pairs
(`last_step_pairs`) at the scoring draw with the critic before its last update, as tianweiy/DMD2 does. Both modes share
`prepare_scoring_batch` and `keep_scores`, so they score the same function of the same draw.

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
…actor within a rollout

`--loss-type dmd2` runs through its own entrypoint, as async NFT does: under `reuse-critic-step` the critic trains
and the actor trains on its scores; under `original` the critic trains steps 0..K-2, scores forward-only, the actor
trains on the scores and the critic trains step K-1. A group saves after its last train of the rollout and, under
--offload-train, sleeps before another group wakes. `train_diffusion.py` sends dmd2 here; the H3 recipe passes
`train_script="train_dmd2.py"` (its reference evals keep train_diffusion.py).

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
@Rockdu
Rockdu force-pushed the dmd2/07b-view-on-372 branch from 0992d73 to a48de7f Compare November 11, 2026 17:01
@Rockdu Rockdu added the run-ci-e2e Run e2e metric-regression tests on this PR label Nov 17, 2026
@Rockdu

Rockdu commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR! One thought on original: could the scoring forward live inside the critic's last step instead of a separate compute phase? Same pattern as prepare asking for the teacher: on step K-1, also build the scoring batch (microbatch_id + m, m = micro-batches per step), and the trunk runs the teacher and the critic no-grad on it before the training forward. Weights, draw ids and the step K-1 training stay the same, so I'd expect it to match the current original bit for bit (a GPU test could confirm).

Since the critic's last step doesn't depend on the actor, the order could then be critic → actor like reuse, and compute/async_compute, optim_steps, the wake_up no-op and train_dmd2.py might not be needed anymore. Also a small thing: compute doesn't call materialize_stored_tensors, so original with --use-output-store would read unmaterialized refs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci-e2e Run e2e metric-regression tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants