Repository navigation
Conversation
a6e8635 to
36222f1
Compare
8732fa8 to
676119c
Compare
676119c to
0992d73
Compare
A group that trains and then scores in the same wake would wake twice, and waking a frozen model twice replaces its pinned host copies with its GPU parameters, so the next sleep would leave the teacher on the GPU. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
`train(..., optim_steps=(lo, hi))` runs windows lo..hi-1 of the rollout's micro-batch schedule with the micro-batch ids of the full run, so a range draws the same noise as the whole rollout would; a range outside the schedule, or one under --diffusion-recompute-old-log-prob, is rejected. The EMA steps once per rollout, by the call whose range ends at the last step. `RayTrainGroup.async_train` passes the range through. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
`FSDPTrainRayActor.compute(rollout_id, rollout_data_ref, external_data, hooks_key)` runs the loss hub's `(--loss-type, hooks_key)` hooks without grad: their train-input selection, one forward per micro-batch through the trunk (`_forward_train_pair_batch` takes the hooks to run instead of the group's), numbered past the rollout's training micro-batches so no training forward shares a draw, then their train-output hook, boxed as train's output is. The models are resharded after; the optimizer, scheduler, EMA and step count stay. `RayTrainGroup.async_compute` dispatches it per rank like `async_train`. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
…ate scoring forward
`reuse-critic-step` (default, no behavior change): the critic's last training step draws in the scoring range, asks
for the teacher and keeps both x0 as the score. `original`: that step is plain SFT and keeps nothing; the new
`("dmd2", "critic-scoring")` hooks, run forward-only by `FSDPTrainRayActor.compute`, score the last step's pairs
(`last_step_pairs`) at the scoring draw with the critic before its last update, as tianweiy/DMD2 does. Both modes share
`prepare_scoring_batch` and `keep_scores`, so they score the same function of the same draw.
Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
…actor within a rollout `--loss-type dmd2` runs through its own entrypoint, as async NFT does: under `reuse-critic-step` the critic trains and the actor trains on its scores; under `original` the critic trains steps 0..K-2, scores forward-only, the actor trains on the scores and the critic trains step K-1. A group saves after its last train of the rollout and, under --offload-train, sleeps before another group wakes. `train_diffusion.py` sends dmd2 here; the H3 recipe passes `train_script="train_dmd2.py"` (its reference evals keep train_diffusion.py). Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
0992d73 to
a48de7f
Compare
|
Thanks for the PR! One thought on Since the critic's last step doesn't depend on the actor, the order could then be critic → actor like reuse, and |
View-only copy of #351 against its real base, #372 (
dmd2/08-teacher-in-trunk), so the diff shows this PR's five commits alone. GitHub refuses to retarget #351 (it is inside the #330..#343 stack), whose diff againstdmd2/06-h3-recipeincludes #367, #368 and #372 until they merge. Review and merge happen on #351; this PR is closed once that diff is clean.The branch
dmd2/07b-view-on-372is the same commits asdmd2/07-original-scoring(a6e8635). The description of the change is on #351.