Skip to content

A test for crucible1906 - #1990

Open
leftwo wants to merge 19 commits into
mainfrom
alan/one906-test
Open

leftwo wants to merge 19 commits into
mainfrom
alan/one906-test

Conversation

@leftwo

@leftwo leftwo commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

A test script, an updated crutest, and (hopefully) a reproduction from crucible 1906.

To run this, you need a system that can run three crucible downstairs, each in a VM.
Have the host running the test have the ability to SSH into the VMs.

I'm not sure if we want to keep this test, but it's worth looking at.
Claude wrote most of the crutest test, I did some cleanup of it (and reworded a bunch) but
it could probably use a bit more cleanup. But, until I know for sure we want it forever
I'm hesitant to spend time on polishing it.

leftwo added 17 commits October 6, 2026 17:15
Add a workload that creates the conditions for the "missing context
slot" corruption from crucible#1906.  For every ZFS record (recordsize,
default 128 KiB) of block data in every extent, write W1 (64 KiB) at
the start of the record, then W2 (32 KiB) directly after it with no
flush between them, then send one flush per full pass.  Both writes
default to at least zfs_immediate_write_sz, so the downstairs data
writes are logged WR_INDIRECT in the ZIL, which is the setup required
for a crash during zil_commit to replay W2's data without its matching
context slot write.

A prober task watches the repair server of each downstairs target and
pauses all new IO within about a second when a downstairs stops
responding, limiting how far the surviving downstairs drift from the
dead one.  IO does not resume until the upstairs reports all
downstairs clients are Active again, so every fault is injected from a
clean all-ACTIVE state.
test_one906.sh runs on the host with the three downstairs VMs.  It
starts crutest with the one906 workload, then loops: arm the panic
trigger in the downstairs 0 VM over ssh, wait for the VM to die,
destroy the VMM and snapshot the dataset backing its disk to preserve
the pre-ZIL-replay state, then boot the VM again.  A downstairs that
comes back online has self-verified its region, so the snapshot is
destroyed once all three downstairs are ACTIVE and the loop continues.
A downstairs that fails to start with "missing context slot" means
crucible#1906 was reproduced: crutest is stopped, the snapshot is
preserved, and the test exits.

dtrace/one906-panic.d runs inside the downstairs VM.  Once a dmu_sync
is seen (a zil_commit is committing a WR_INDIRECT record), it panics
the guest after the Nth following ZIL log block write completes,
cutting the ZIL chain inside the window where replay can restore data
without its matching context slot write.  The harness randomizes N on
every arming.
Route the one906 geometry, pass, probe, and pause/resume messages
through the crutest task logger instead of println, matching the rest
of crutest's output.  Pause events log at WARN.  The epoch timestamp
helper is no longer needed since slog timestamps every line.
The scp of the DTrace trigger to the VM did not share the ssh options,
so a setup needing -i prompted for a password.  Add SSH_KEY, used by
both ssh and scp when set.
Arming is now the first thing each test loop does; there is no reason
to let IO run for a random delay first.  The randomized panic point in
the trigger itself already varies where the ZIL chain is cut.
pkill -f one906-panic matched the remote shell running the pkill
itself, since its argv contains the pattern and illumos pkill does not
exclude ancestor processes.  That killed the ssh session before dtrace
could start, so arming always failed.  Use the [o]ne906-panic bracket
pattern, which still matches a running trigger but not the command
line that contains the bracketed pattern.
The script ignored propolis-cli errors when restarting the VM after a
panic, so a propolis-server that was not back yet left the test stuck
waiting for a downstairs that could never return.  Retry the instance
create up to VM_START_RETRIES times, five seconds apart, and if the VM
still cannot be created or run, stop crutest, preserve the snapshot,
and exit with an error.
The one906 workload only checked for SIGUSR1 between passes, but a
pass can never finish while the prober has IO paused for a dead
downstairs, so a shutdown request was ignored exactly when the harness
needs it.  Add a stop flag checked by the workers and all pause gates,
and poll the signal channel while paused and while waiting for the
workers, so SIGUSR1 stops the test promptly in any state.
Escalate when stopping crutest: SIGUSR1 for a clean shutdown, SIGTERM
after 15 seconds, then SIGKILL, so the harness can not block forever
waiting for crutest to exit.  Also pass --quit to crutest so it exits
its final work queue loop once the workload stops.
seq is not in the default PATH on illumos.  Use bash C-style for loops
instead.
The prober resumed when active_count reached num_targets, but after a
silent downstairs death the upstairs keeps the dead connection Active
(and buffers IO to it) until its ~45s inactivity timeout, so
active_count is a stale read during that window.  Resuming then pushed
a full IO load into a cluster about to fault and live-repair, which
could cascade into a lost quorum.

Resume instead on a positive check.  While a client is not Active the
upstairs is still faulting or repairing, so wait and send nothing.
Once all clients are Active, send exactly one read+flush probe per
fault episode and wait for the work queue to drain: a flush only
retires once it is terminal on all three clients, so ds_count
returning to zero means every downstairs actually processed the probe,
not merely that the repair port answers or that a stale Active
lingers.  The probe is not re-sent while it is in flight.
Wait for crutest to log that it resumed IO (its one-shot read+flush
liveness probe is the authority on real recovery) instead of
re-checking dsstate, which shows a stale Active after a silent VM
death.  Also watch crutest during the trigger wait and the recovery
wait so the harness stops at once if crutest exits, rather than
blocking for the full timeout.
Fill every block (and verify the fill) before the test pattern begins,
so all three downstairs hold known data everywhere from the start.

After a downstairs faults and rejoins, verify every block before the
next write pass.  This confirms the repaired downstairs is consistent
with the others and catches a downstairs that rejoined with wrong data
even when it opened its region without error.  A failed verify exits
the test.
The startup wait polled downstairs state at 5s then 30s intervals, so
"all downstairs active" was reported up to 30s after it actually
happened, well behind crutest's own log.  Poll every 2s and print only
when the state changes, and watch crutest during startup too.  Tighten
the recovery-wait poll from 5s to 2s for the same reason.
@leftwo
leftwo requested a review from jmpesp October 8, 2026 18:27
The one906 prober polled get_work four times a second, which calls
show_work and takes the downstairs lock, contending with real IO.  The
probe only needs to know whether the repair server answers, so switch
it to get_region_mode, which returns a static field from the server's
context without touching the downstairs.
The test reproducer pushes crutest harder than the three VM
setup can handle, and we were getting false errors.  Back off
a little during which we expect live repair to be running.
@leftwo

leftwo commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread crutest/src/main.rs
Comment on lines +3871 to +3881
})
};

// The write log is shared with the workers; put it back in
// DiskInfo when the test is done so verify works.
let write_log = Arc::new(std::sync::Mutex::new(std::mem::replace(
&mut di.write_log,
WriteLog::new(0),
)));

// Set when a shutdown has been requested; the workers and pause

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 After a downstairs rejoins, the post-recovery full-volume verify will almost never run, so a bad repair goes unnoticed. was_paused is only set by the wait loop at the top of each pass. A fault in the middle of a pass is absorbed by one906_pause_gate inside the workers, and by the pre-flush wait loop. By the next loop top, paused is already false. Fix: record that a pause happened wherever it is observed and run the verify before the next pass. For example, keep a sticky AtomicBool or episode counter that the prober sets when it pauses, then test and clear it at the loop top. [also at: crutest/src/main.rs:495 - After a downstairs rejoins, the post-recovery whole-volume verify usually never runs, so a bad repair can go unnoticed. was_paused is only set if paused is…]

Why this was flagged

The harness (tools/test_one906.sh) kills downstairs 0 while the one906 workers are mid-pass, and that is the normal case. Workers park in one906_pause_gate and the end-of-pass flush waits in the while paused.load(...) loop. Neither sets was_paused. The pass only finishes after the prober clears paused ("resuming"). So when the next iteration reaches the top-of-loop wait, paused is false, was_paused stays false, and the "one906: downstairs rejoined, verifying all blocks" path is skipped. The repaired downstairs is never checked against the WriteLog, despite what the workload's comments and description say. The harness also treats the "resuming" log line as recovery and destroys the snapshot, so a divergent repair is neither detected nor preserved. This is new code, so there is no base behaviour to compare against.

Verification: normal. Trigger: any downstairs fault or not-Active episode that starts and clears inside a pass, i.e. while workers are writing or before the end-of-pass flush. That is the harness's own case, and also covers live repair or a congestion fault noticed by the prober at main.rs:3803. Mechanism, verified in crutest/src/main.rs:

  • let mut was_paused = false; is set to true only inside the top-of-loop while paused.load(...) (3890-3895). The verify at 3900-3908 runs only when that flag is true.

Comment thread tools/test_one906.sh
Comment on lines +365 to +385
exit 1
fi
resume_now=$(grep -c "resuming" "$CRUTEST_LOG" 2>/dev/null)
if [[ "$resume_now" -gt "$resume_before" ]]; then
recovered=1
break
fi
sleep 2
done

if [[ $recovered -eq 0 ]]; then
msg "crutest did not resume IO in $DS_WAIT seconds"
stop_crutest
msg "snapshot left in place at $SNAP"
exit 1
fi
msg "crutest resumed IO, all downstairs serving again"

# Region verified and repaired, this loop found nothing.
msg "Destroying snapshot $SNAP"
if ! zfs destroy "$SNAP"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Operators can lose the pre-replay ZFS snapshot when crutest's post-rejoin verify finds bad data, because the harness destroys it before the verify finishes. Recovery is declared as soon as a new "resuming" line shows up (line 367). The snapshot is then destroyed at line 385 with no check that crutest is still alive. The verify at crutest/src/main.rs:3904 only starts after that line, so a failure there is seen only after the evidence is gone. Fix: before zfs destroy, wait for "one906: post-recovery verify passed" (or the next pass line) and confirm crutest is still running; if it exited, keep the snapshot.

Why this was flagged

The prober clears paused and logs "one906: all {num_targets} downstairs serving IO again, resuming" at crutest/src/main.rs:3852-3855. Only after that does the main loop run verify_volume when was_paused is set (main.rs:3899-3907), and that reads the whole volume, which takes seconds or more. The harness polls every 2s. Once the resume count goes up (tools/test_one906.sh:367-369), it goes straight to zfs destroy "$SNAP" at line 385. If the verify then fails, crutest exits with "post-recovery verify failed". The harness only notices at the next loop's kill -0 check (line 206-207), after the pre-ZIL-replay snapshot of the downstairs-0 dataset is already gone. That snapshot is the evidence for a downstairs that rejoined with wrong data. No safeguard covers this: the recovery loop checks kill -0 only before it sees the resume line, never after. This is a new tool, so there is no base behavior.

Verification: The harness counts it as recovered as soon as a new "resuming" line appears (tools/test_one906.sh:367-370). It then goes straight to zfs destroy "$SNAP" at line 385. The verify only runs after that (main.rs:3900-3907). If it fails, crutest exits. The harness only notices on the next iteration (line 206, "crutest is gone"), and by then the snapshot is gone.

Comment thread crutest/src/main.rs
Comment on lines +3553 to +3556
}
Ok(SignalAction::Verify) => {
warn!(log, "one906: ignoring verify request mid-pass");
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) In --continuous mode, verify requests sent during a pass are almost always thrown away. The end-of-pass SignalAction::Verify branch is effectively dead. one906_check_signals takes Verify off shutdown_rx and only logs 'ignoring verify request mid-pass'. It runs every 250ms while workers are running and on every pause poll, so it nearly always takes the signal first. Fix: record a pending-verify flag in one906_check_signals instead of dropping it, and run verify_volume after the pass's flush when it is set.

Why this was flagged

An operator runs crutest one906 --continuous and sends the verify signal (SignalAction::Verify) to check the volume. Passes take seconds or more. During a pass the main loop polls one906_check_signals every 250ms (main.rs ~3946-3953). It also polls during each pause-wait loop (~3896, ~3964). Each of those calls shutdown_rx.try_recv() and drops Verify with a warn (main.rs ~3553-3556). The post-flush match at ~3994-4002 only sees signals that arrive in the tiny window after the workers finish. So the requested verify never runs, while other crutest workloads using WhenToQuit::Signal do honor it.

Verification: SIGUSR2 maps to SignalAction::Verify at main.rs:749-750. one906_check_signals runs every 250ms in the worker-wait loop (main.rs:3953-3959). On Ok(SignalAction::Verify) it only logs warn!(log, "one906: ignoring verify request mid-pass") (main.rs:3554-3556). It consumes the message without recording it anywhere. Most SIGUSR2 requests are therefore dropped and no verify runs.

Comment thread crutest/src/main.rs
// all of it. The test run with downstairs on VMs runs
// into trouble during repair, so don't try to send IO
// during that window.
let disturbed = ports_down || active != num_targets;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Running one906 from a VCR that has a read-only parent stalls forever after the fill, because the prober sees the cluster as permanently disturbed. active comes from query_work_queue, which adds the read-only parent's Active clients. num_targets counts only the sub_volume targets, so active != num_targets is always true. Fix: reject volumes with a read_only_parent in the setup checks, or compare against the active count of the sub_volume alone. Either way the resume condition must match the targets being probed.

Why this was flagged

An operator runs crutest one906 --vcr-file with a VCR that has a read_only_parent. The sub_volume check at crutest/src/main.rs:3618-3622 passes, because volume_extent_info (upstairs/src/volume.rs:712-739) lists only sub_volumes. vcr.targets() (crucible-client-types/src/lib.rs:43-47) returns only the sub_volume targets, so num_targets = 3 at main.rs:3758. Volume::query_work_queue (upstairs/src/volume.rs:831-836) adds the read-only parent's active_count, so active = 6 at main.rs:3790. disturbed at main.rs:3803 is then always true. The prober sets paused on its first cycle and never clears it, and the pass loop waits at the pause gate forever. In count mode that is an endless hang with no error. In --continuous mode the run only ends on SIGUSR1. No guard catches this: the setup checks never look at read_only_parent. The base branch has no one906 workload.

Verification: nit. Trigger: one906 is run with --vcr-file pointing at a VCR whose Volume has a Region read_only_parent. The candidate's description of the mechanism is accurate.

  • Setup check passes. The check at crutest/src/main.rs:3618 (di.volume_info.volumes.len() != 1) uses volume_extent_info (upstairs/src/volume.rs:712-746). That function walks only self.sub_volumes and has the note // TODO: add support for read only parents, so a read-only parent is never counted.
  • num_targets counts sub_volume targets only. targets comes from vcr.targets() (main.rs:816).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant