test(two-node): simulator models the takeover read on a frozen peer's devices (F23) and F24

[schmidt-scaled]

Test code only: the two-node simulator now models the failure the c11 bed run found in ultra #219 (F24), and corrects the 02c model (F23).

What c11 showed

• 02c (frozen primary sn2): Adding lvol id to lvol events #219 isolated sn2's devices at the solo grant by closing them, but their status still read online.
  • The follower survivor sn1's IO-triggered takeover read the lvstore super block from a closed device and failed (ParityOneRead::sNormalRead_Begin res=-6).
  • The lvstore update on failover then aborted the survivor, and the client saw 160 IO errors.
• Why the simulator missed it: the F23 model (test(two-node): simulate F23, survivor writes waiting on a frozen peer's data devices #1501) only covered the leader's writes (02a).

Model changes

• Takeover read on a frozen peer (F23): the takeover's lvstore metadata read now waits on a still-frozen peer's data devices until their connections close after the peer's abort on wake. That matches bed c9/c10 02c (9 s), unless the node isolates them at the grant (stoplist_peer_devices_on_grant).
  • Only while the peer is still frozen: after its abort, the hublvol redirect itself fails only once the connections close, so 02b stays at 3 s as on the bed.
• Closed devices still read as online (F24): with isolation but without the new capability isolate_marks_devices_unavailable, a solo follower's takeover read aborts the survivor. That capability models Adding lvol id to lvol events #219's F24 commit, which marks the devices unavailable before resetting them.
• New scenario F24-frozen-primary-takeover-read: expected to fail on main for F23, and passes with the pending stack's capabilities.
  • test_f24_needs_its_own_fix shows it fails with "F24" under Adding lvol id to lvol events #219 as run in c11.
• Pending fix sets: the frozen-primary scenarios (F19, F20, F21) and F23's now include the new capability. In the simulator the client also writes vol0, whose LVS the survivor takes over in 02a.

Tests

Run locally in WSL against real FoundationDB, filtered on F24, F23, F21, F19, F20, 02a, 02b and 02c: 12 passed, 6 expected failures. The fast and full tiers run in CI. ruff is clean.

The ultra side is simplyblock/ultra#219, commit 04c38403.

🤖 Generated with Claude Code