[mmshad]

Summary

• CheckpointManager._cleanup now removes an old checkpoint with _remove_tree, which retries shutil.rmtree when it raises ENOTEMPTY, EBUSY or ENOENT while the directory still exists. All the removals of one cleanup share waits of 0.05, 0.1, 0.2 and 0.4 s, so a cleanup requests at most 0.75 s of retry sleep. On NFS, deleting a file written a moment earlier can briefly leave a .nfs file behind, and the prune failed with [Errno 39] Directory not empty and left the directory for the next cleanup (Checkpoint pruning on NFS fails with Errno 39 right after a write and can leave a directory behind #527).
• Any other removal error, or one that outlasts the waits, gives the same warning as before, and the next cleanup retries the directory. On a server that keeps the .nfs file longer (one of 317 failed removals on an Isilon server kept it 4.0 s), old directories can remain, also when the run ends. Each attempt holds the sidecar lock; the waits do not.
• docs/checkpointing/auto-resume.md describes the retries.
• Show child tracebacks in test failures and fix NFS and isolation flakes #496 makes the tests of test_a_registrar_that_raises_costs_only_the_registration fails when NFS refuses to prune step_1 #494 and test_cleanup_never_deletes_latest_or_in_flight fails when a shared filesystem refuses a prune #372 tolerate a refused removal; this PR retries the removal itself, which also made those tests pass in the NFS runs on the review. The two merge cleanly together. test_record_amends_one_record_atomically_through_a_temp_file fails on NFS when the replaced metadata.json is kept as a .nfs entry #531, from os.replace, is a separate NFS failure this PR does not change.

Testing

• On netscratch (NFSv3 over RDMA), at 0fd1bd1: the two unit tests from Checkpoint pruning on NFS fails with Errno 39 right after a write and can leave a directory behind #527 passed in 10 of 10 runs (on main one or both failed in 2 of 5). Saving every step with keep_last_n = 1, three runs each on 1 GPU and on 4 GPUs with async saves removed all 39 old directories with no prune warning and kept only step_40 (on main 11 and 4 prunes failed, and the 1-GPU run left step_39). A resumed 4-GPU async run saving every 10 steps kept step_50 and step_60 with no warning. Lustre gave the same results.
• At the head, on 4 H200s: on netscratch the retention tests and the tests of Checkpoint pruning on NFS fails with Errno 39 right after a write and can leave a directory behind #527 and test_cleanup_never_deletes_latest_or_in_flight fails when a shared filesystem refuses a prune #372 passed in 10 of 10 runs; saving every step with keep_last_n = 1 on 1 GPU and on 4 GPUs with async saves removed all 39 old directories with no prune warning; tests/unit with the GPUs visible, temp dirs on NFS 3633 passed, 4 skipped, 5 xfailed and on Lustre the same; with temp dirs on NFS, tests/integration (not slow) 123 passed, tests/distributed 129 passed, 2 skipped, 1 xfailed, and tests/e2e --e2e 42 passed, 1 skipped, 2 xfailed. ruff clean; pyright 0 errors with mlflow and databricks-sdk installed.
• New tests: test_cleanup_retries_a_directory_nfs_briefly_keeps_non_empty (ENOTEMPTY, EBUSY and ENOENT, then success; each attempt holds the lock and the waits do not), test_cleanup_tries_other_errors_once (a permission error, and a directory already gone, each tried once) and test_a_cleanup_waits_at_most_0_75_s_in_all (the removals of one cleanup share the waits). test_cleanup_retries_a_checkpoint_it_could_not_remove checks the five tries, each under the lock, and the waits; with the lock taken off the tries, it and two other tests fail. test_sidecar_writes_amends_and_removals_hold_the_sidecar_lock records each removal once per lock state, so an attempt without the lock still fails it.

Closes #527

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
kempnerforge/checkpoint/manager.py	99.47% <100.00%> (+0.01%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amazloumi]

Approve. The retries also make the NFS failures of #494 and #372 pass (4 of 8 runs each on main, none here), which #496 closes with test-side changes, so it would help to say how the two PRs relate. One more NFS-only test failure, from os.replace rather than a removal, is filed as #531.

[amazloumi]

Non-blocking: every old checkpoint whose removal keeps failing now adds up to 0.75 s of sleep to each cleanup, with no limit across directories, and every rank waits for it at the _agree barrier that ends save(), wait() and flush_pending_save(). With 10 old directories that kept raising ENOTEMPTY (as when a process on the same node holds their files open), one _cleanup slept 7.5 s over 50 rmtree calls, against 0 s and 10 calls on main.

[mmshad]

Capped in a9e7c51: the removals of one cleanup share the 0.05, 0.1, 0.2 and 0.4 s waits, so a cleanup requests at most 0.75 s of retry sleep however many directories keep failing. With your ten directories that is fourteen rmtree calls.

[amazloumi]

Non-blocking: the 0.75 s budget fits the windows measured on netscratch but not every NFS server, so a run can still end with a leftover directory when its last prune misses. On a second NFS server (Isilon), writing a directory the way a save does and removing it at once left a .nfs entry that outlived 0.75 s in 1 of 317 failed removals (it lasted 4.0 s); the other 316 cleared within 0.38 s.

[mmshad]

Kept at 0.75 s per cleanup, since every rank waits for it at the barrier. The next cleanup still retries a directory that outlasts it, but on such a server old directories can remain, also when the run ends. The docs and the description now say so.

[amazloumi]

The user docs still describe the old behavior: docs/checkpointing/auto-resume.md:158-160 says a .nfs placeholder makes the removal fail and leaves the checkpoint for the next cleanup, with no mention of the retries added here.

[mmshad]

Updated docs/checkpointing/auto-resume.md in a9e7c51 and 9ccb65c: it describes the retries, the 0.75 s of retry sleep a cleanup can request, and the directories a slower server can leave.

[amazloumi]

Nit: no test checks that the fifth and last attempt holds the sidecar lock. With the lock dropped from the final rmtree (manager.py:421), all nine cleanup and sidecar tests still pass, since this test reaches the fifth attempt but records only the directory name and the NFS test stops after four.

[mmshad]

test_cleanup_retries_a_checkpoint_it_could_not_remove now records the lock state of all five tries and the four waits (a9e7c51), so a try without the lock fails it. On GPU, with the lock taken off the tries, it and two other tests fail.

[mmshad]

#496 makes the tests of #494 and #372 tolerate a refused removal; this PR retries the removal itself, which also made those tests pass in your NFS runs. The two merge cleanly together, though a bounded retry cannot make every prune succeed. #531, from os.replace, is outside this change. The description now says how they relate.