Skip to content

fix(fsdp): preserve FA3 autograd and deterministic dispatch - #259

Open
Odysseusailoon wants to merge 10 commits into
radixark:mainfrom
Odysseusailoon:codex/fa3-deterministic-ulysses
Open

Odysseusailoon wants to merge 10 commits into
radixark:mainfrom
Odysseusailoon:codex/fa3-deterministic-ulysses

Conversation

@Odysseusailoon

@Odysseusailoon Odysseusailoon commented Sep 24, 2026 •

Copy link
Copy Markdown

Summary

With --fsdp-attention-backend _flash_3, the pinned Diffusers implementation wraps FA3 in a custom op without a registered backward. It also passes an explicit deterministic=False, which can override Miles' existing partial(..., deterministic=True).

This change registers a dense FA3 adapter that calls FA3's public autograd function directly, preserving backward support and forwarding the effective deterministic request to the kernel.

  • Install the adapter when FA3 is explicitly selected, with deterministic mode enabled or disabled.
  • Preserve actor-level, per-call, and PyTorch-global deterministic requests.
  • Validate the selected flash entry point: an installed FA2 kernel no longer satisfies FA3 validation.
  • Preserve Miles' existing pure-Ulysses communication path and reject unsupported attention configurations.

Validation

  • CPU: 317 tests passed on macOS using the repository's CPU stubs. All nine pre-commit hooks and train_diffusion.py --help passed.
  • H100 attention regression: 48 focused CPU tests and four strict GPU cases passed in the complete Miles image. GPU coverage includes BF16 SP=1/2/4 and FP16 SP=1 at S=1024, checking outputs and gradients against direct FA3 and independent FP32 references, 20 bitwise-repeatable evaluations per fixed topology, and a projection-layer SGD update.
  • Krea2 short E2E: four NFT training updates completed at DP=2/SP=1, including OCR rewards, EMA updates, and LoRA IPC synchronization. Tracing confirmed actual FA3 execution; all 256 LoRA tensors changed between the first and fourth checkpoints.

Model-level SP=2 validation

The tiny Wan FSDP2 regression passed on two H100s through the normal pytest entry point: 1 passed, 0 skipped/errors/failures. It compares DP=2/SP=1 with DP=1/SP=2 at fixed global batch 2 for two SGD updates, with gradient checkpointing disabled and enabled. The model is randomly initialized and uses synthetic inputs; it reuses Miles' existing Wan sequence-parallel plan.

Each rank passed 2,304 numerical comparisons, including 1,152 bitwise-equal repeat checks, covering all 69 trainable parameter tensors. All 96 traced real FA3 calls per rank returned successfully, including 32 checkpoint recomputations. Across SP=1/2, full-gradient and full-parameter-delta relative L2 errors were at most 0.001843 and 0.001701; every per-parameter check also passed the fixed, predeclared limits. Checkpoint on/off results were bitwise equal in this run. The native two-rank CPU control also passed.

Scope

This establishes the tested tiny Wan architecture's FSDP2 + FA3 + Ulysses training integration. It does not establish pretrained Wan quality, Krea2 SP=2, model-level SP=4, frozen rollout/train numerical parity, or whole-run repeatability. It does not add Ring, FA3 varlen, or compile support. Deterministic behavior remains dependent on the underlying kernel, hardware, and input configuration; recorded GPU validation is scoped to H100.

See the validation guide for reproduction commands and test scope.

@Odysseusailoon
Odysseusailoon marked this pull request as ready for review September 28, 2026 03:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant