Repository navigation
Conversation
Add a workload that creates the conditions for the "missing context slot" corruption from crucible#1906. For every ZFS record (recordsize, default 128 KiB) of block data in every extent, write W1 (64 KiB) at the start of the record, then W2 (32 KiB) directly after it with no flush between them, then send one flush per full pass. Both writes default to at least zfs_immediate_write_sz, so the downstairs data writes are logged WR_INDIRECT in the ZIL, which is the setup required for a crash during zil_commit to replay W2's data without its matching context slot write. A prober task watches the repair server of each downstairs target and pauses all new IO within about a second when a downstairs stops responding, limiting how far the surviving downstairs drift from the dead one. IO does not resume until the upstairs reports all downstairs clients are Active again, so every fault is injected from a clean all-ACTIVE state.
test_one906.sh runs on the host with the three downstairs VMs. It starts crutest with the one906 workload, then loops: arm the panic trigger in the downstairs 0 VM over ssh, wait for the VM to die, destroy the VMM and snapshot the dataset backing its disk to preserve the pre-ZIL-replay state, then boot the VM again. A downstairs that comes back online has self-verified its region, so the snapshot is destroyed once all three downstairs are ACTIVE and the loop continues. A downstairs that fails to start with "missing context slot" means crucible#1906 was reproduced: crutest is stopped, the snapshot is preserved, and the test exits. dtrace/one906-panic.d runs inside the downstairs VM. Once a dmu_sync is seen (a zil_commit is committing a WR_INDIRECT record), it panics the guest after the Nth following ZIL log block write completes, cutting the ZIL chain inside the window where replay can restore data without its matching context slot write. The harness randomizes N on every arming.
Route the one906 geometry, pass, probe, and pause/resume messages through the crutest task logger instead of println, matching the rest of crutest's output. Pause events log at WARN. The epoch timestamp helper is no longer needed since slog timestamps every line.
The scp of the DTrace trigger to the VM did not share the ssh options, so a setup needing -i prompted for a password. Add SSH_KEY, used by both ssh and scp when set.
Arming is now the first thing each test loop does; there is no reason to let IO run for a random delay first. The randomized panic point in the trigger itself already varies where the ZIL chain is cut.
pkill -f one906-panic matched the remote shell running the pkill itself, since its argv contains the pattern and illumos pkill does not exclude ancestor processes. That killed the ssh session before dtrace could start, so arming always failed. Use the [o]ne906-panic bracket pattern, which still matches a running trigger but not the command line that contains the bracketed pattern.
The script ignored propolis-cli errors when restarting the VM after a panic, so a propolis-server that was not back yet left the test stuck waiting for a downstairs that could never return. Retry the instance create up to VM_START_RETRIES times, five seconds apart, and if the VM still cannot be created or run, stop crutest, preserve the snapshot, and exit with an error.
The one906 workload only checked for SIGUSR1 between passes, but a pass can never finish while the prober has IO paused for a dead downstairs, so a shutdown request was ignored exactly when the harness needs it. Add a stop flag checked by the workers and all pause gates, and poll the signal channel while paused and while waiting for the workers, so SIGUSR1 stops the test promptly in any state.
Escalate when stopping crutest: SIGUSR1 for a clean shutdown, SIGTERM after 15 seconds, then SIGKILL, so the harness can not block forever waiting for crutest to exit. Also pass --quit to crutest so it exits its final work queue loop once the workload stops.
seq is not in the default PATH on illumos. Use bash C-style for loops instead.
The prober resumed when active_count reached num_targets, but after a silent downstairs death the upstairs keeps the dead connection Active (and buffers IO to it) until its ~45s inactivity timeout, so active_count is a stale read during that window. Resuming then pushed a full IO load into a cluster about to fault and live-repair, which could cascade into a lost quorum. Resume instead on a positive check. While a client is not Active the upstairs is still faulting or repairing, so wait and send nothing. Once all clients are Active, send exactly one read+flush probe per fault episode and wait for the work queue to drain: a flush only retires once it is terminal on all three clients, so ds_count returning to zero means every downstairs actually processed the probe, not merely that the repair port answers or that a stale Active lingers. The probe is not re-sent while it is in flight.
Wait for crutest to log that it resumed IO (its one-shot read+flush liveness probe is the authority on real recovery) instead of re-checking dsstate, which shows a stale Active after a silent VM death. Also watch crutest during the trigger wait and the recovery wait so the harness stops at once if crutest exits, rather than blocking for the full timeout.
Fill every block (and verify the fill) before the test pattern begins, so all three downstairs hold known data everywhere from the start. After a downstairs faults and rejoins, verify every block before the next write pass. This confirms the repaired downstairs is consistent with the others and catches a downstairs that rejoined with wrong data even when it opened its region without error. A failed verify exits the test.
The startup wait polled downstairs state at 5s then 30s intervals, so "all downstairs active" was reported up to 30s after it actually happened, well behind crutest's own log. Poll every 2s and print only when the state changes, and watch crutest during startup too. Tighten the recovery-wait poll from 5s to 2s for the same reason.
The one906 prober polled get_work four times a second, which calls show_work and takes the downstairs lock, contending with real IO. The probe only needs to know whether the repair server answers, so switch it to get_region_mode, which returns a static field from the server's context without touching the downstairs.
The test reproducer pushes crutest harder than the three VM setup can handle, and we were getting false errors. Back off a little during which we expect live repair to be running.
|
@claude review |
| }) | ||
| }; | ||
|
|
||
| // The write log is shared with the workers; put it back in | ||
| // DiskInfo when the test is done so verify works. | ||
| let write_log = Arc::new(std::sync::Mutex::new(std::mem::replace( | ||
| &mut di.write_log, | ||
| WriteLog::new(0), | ||
| ))); | ||
|
|
||
| // Set when a shutdown has been requested; the workers and pause |
There was a problem hiding this comment.
🔴 After a downstairs rejoins, the post-recovery full-volume verify will almost never run, so a bad repair goes unnoticed. was_paused is only set by the wait loop at the top of each pass. A fault in the middle of a pass is absorbed by one906_pause_gate inside the workers, and by the pre-flush wait loop. By the next loop top, paused is already false. Fix: record that a pause happened wherever it is observed and run the verify before the next pass. For example, keep a sticky AtomicBool or episode counter that the prober sets when it pauses, then test and clear it at the loop top. [also at: crutest/src/main.rs:495 - After a downstairs rejoins, the post-recovery whole-volume verify usually never runs, so a bad repair can go unnoticed. was_paused is only set if paused is…]
Why this was flagged
The harness (tools/test_one906.sh) kills downstairs 0 while the one906 workers are mid-pass, and that is the normal case. Workers park in one906_pause_gate and the end-of-pass flush waits in the while paused.load(...) loop. Neither sets was_paused. The pass only finishes after the prober clears paused ("resuming"). So when the next iteration reaches the top-of-loop wait, paused is false, was_paused stays false, and the "one906: downstairs rejoined, verifying all blocks" path is skipped. The repaired downstairs is never checked against the WriteLog, despite what the workload's comments and description say. The harness also treats the "resuming" log line as recovery and destroys the snapshot, so a divergent repair is neither detected nor preserved. This is new code, so there is no base behaviour to compare against.
Verification: normal. Trigger: any downstairs fault or not-Active episode that starts and clears inside a pass, i.e. while workers are writing or before the end-of-pass flush. That is the harness's own case, and also covers live repair or a congestion fault noticed by the prober at main.rs:3803. Mechanism, verified in crutest/src/main.rs:
let mut was_paused = false;is set to true only inside the top-of-loopwhile paused.load(...)(3890-3895). The verify at 3900-3908 runs only when that flag is true.
| exit 1 | ||
| fi | ||
| resume_now=$(grep -c "resuming" "$CRUTEST_LOG" 2>/dev/null) | ||
| if [[ "$resume_now" -gt "$resume_before" ]]; then | ||
| recovered=1 | ||
| break | ||
| fi | ||
| sleep 2 | ||
| done | ||
|
|
||
| if [[ $recovered -eq 0 ]]; then | ||
| msg "crutest did not resume IO in $DS_WAIT seconds" | ||
| stop_crutest | ||
| msg "snapshot left in place at $SNAP" | ||
| exit 1 | ||
| fi | ||
| msg "crutest resumed IO, all downstairs serving again" | ||
|
|
||
| # Region verified and repaired, this loop found nothing. | ||
| msg "Destroying snapshot $SNAP" | ||
| if ! zfs destroy "$SNAP"; then |
There was a problem hiding this comment.
🟡 (optional) Operators can lose the pre-replay ZFS snapshot when crutest's post-rejoin verify finds bad data, because the harness destroys it before the verify finishes. Recovery is declared as soon as a new "resuming" line shows up (line 367). The snapshot is then destroyed at line 385 with no check that crutest is still alive. The verify at crutest/src/main.rs:3904 only starts after that line, so a failure there is seen only after the evidence is gone. Fix: before zfs destroy, wait for "one906: post-recovery verify passed" (or the next pass line) and confirm crutest is still running; if it exited, keep the snapshot.
Why this was flagged
The prober clears paused and logs "one906: all {num_targets} downstairs serving IO again, resuming" at crutest/src/main.rs:3852-3855. Only after that does the main loop run verify_volume when was_paused is set (main.rs:3899-3907), and that reads the whole volume, which takes seconds or more. The harness polls every 2s. Once the resume count goes up (tools/test_one906.sh:367-369), it goes straight to zfs destroy "$SNAP" at line 385. If the verify then fails, crutest exits with "post-recovery verify failed". The harness only notices at the next loop's kill -0 check (line 206-207), after the pre-ZIL-replay snapshot of the downstairs-0 dataset is already gone. That snapshot is the evidence for a downstairs that rejoined with wrong data. No safeguard covers this: the recovery loop checks kill -0 only before it sees the resume line, never after. This is a new tool, so there is no base behavior.
Verification: The harness counts it as recovered as soon as a new "resuming" line appears (tools/test_one906.sh:367-370). It then goes straight to zfs destroy "$SNAP" at line 385. The verify only runs after that (main.rs:3900-3907). If it fails, crutest exits. The harness only notices on the next iteration (line 206, "crutest is gone"), and by then the snapshot is gone.
| } | ||
| Ok(SignalAction::Verify) => { | ||
| warn!(log, "one906: ignoring verify request mid-pass"); | ||
| } |
There was a problem hiding this comment.
🟡 (optional) In --continuous mode, verify requests sent during a pass are almost always thrown away. The end-of-pass SignalAction::Verify branch is effectively dead. one906_check_signals takes Verify off shutdown_rx and only logs 'ignoring verify request mid-pass'. It runs every 250ms while workers are running and on every pause poll, so it nearly always takes the signal first. Fix: record a pending-verify flag in one906_check_signals instead of dropping it, and run verify_volume after the pass's flush when it is set.
Why this was flagged
An operator runs crutest one906 --continuous and sends the verify signal (SignalAction::Verify) to check the volume. Passes take seconds or more. During a pass the main loop polls one906_check_signals every 250ms (main.rs ~3946-3953). It also polls during each pause-wait loop (~3896, ~3964). Each of those calls shutdown_rx.try_recv() and drops Verify with a warn (main.rs ~3553-3556). The post-flush match at ~3994-4002 only sees signals that arrive in the tiny window after the workers finish. So the requested verify never runs, while other crutest workloads using WhenToQuit::Signal do honor it.
Verification: SIGUSR2 maps to SignalAction::Verify at main.rs:749-750. one906_check_signals runs every 250ms in the worker-wait loop (main.rs:3953-3959). On Ok(SignalAction::Verify) it only logs warn!(log, "one906: ignoring verify request mid-pass") (main.rs:3554-3556). It consumes the message without recording it anywhere. Most SIGUSR2 requests are therefore dropped and no verify runs.
| // all of it. The test run with downstairs on VMs runs | ||
| // into trouble during repair, so don't try to send IO | ||
| // during that window. | ||
| let disturbed = ports_down || active != num_targets; |
There was a problem hiding this comment.
🟡 (optional) Running one906 from a VCR that has a read-only parent stalls forever after the fill, because the prober sees the cluster as permanently disturbed. active comes from query_work_queue, which adds the read-only parent's Active clients. num_targets counts only the sub_volume targets, so active != num_targets is always true. Fix: reject volumes with a read_only_parent in the setup checks, or compare against the active count of the sub_volume alone. Either way the resume condition must match the targets being probed.
Why this was flagged
An operator runs crutest one906 --vcr-file with a VCR that has a read_only_parent. The sub_volume check at crutest/src/main.rs:3618-3622 passes, because volume_extent_info (upstairs/src/volume.rs:712-739) lists only sub_volumes. vcr.targets() (crucible-client-types/src/lib.rs:43-47) returns only the sub_volume targets, so num_targets = 3 at main.rs:3758. Volume::query_work_queue (upstairs/src/volume.rs:831-836) adds the read-only parent's active_count, so active = 6 at main.rs:3790. disturbed at main.rs:3803 is then always true. The prober sets paused on its first cycle and never clears it, and the pass loop waits at the pause gate forever. In count mode that is an endless hang with no error. In --continuous mode the run only ends on SIGUSR1. No guard catches this: the setup checks never look at read_only_parent. The base branch has no one906 workload.
Verification: nit. Trigger: one906 is run with --vcr-file pointing at a VCR whose Volume has a Region read_only_parent. The candidate's description of the mechanism is accurate.
- Setup check passes. The check at crutest/src/main.rs:3618 (
di.volume_info.volumes.len() != 1) usesvolume_extent_info(upstairs/src/volume.rs:712-746). That function walks onlyself.sub_volumesand has the note// TODO: add support for read only parents, so a read-only parent is never counted. num_targetscounts sub_volume targets only.targetscomes fromvcr.targets()(main.rs:816).
A test script, an updated crutest, and (hopefully) a reproduction from crucible 1906.
To run this, you need a system that can run three crucible downstairs, each in a VM.
Have the host running the test have the ability to SSH into the VMs.
I'm not sure if we want to keep this test, but it's worth looking at.
Claude wrote most of the crutest test, I did some cleanup of it (and reworded a bunch) but
it could probably use a bit more cleanup. But, until I know for sure we want it forever
I'm hesitant to spend time on polishing it.