Repository navigation
Conversation
|
I've started on removing the "replace before reconcile" ability for this PR. |
With 2/3 activation possible, we can no longer allow a downstairs to be replaced before we have gone through reconciliation. Added a new crutest test for 2/3 activation. Removed or changed a few tests that attempted to replace a downstairs before activation.
|
I've removed the tests and ability to replace a downstairs before activation, which I believe is what we want now that we have 2/3 activation. I also added a specific crutest test for 2/3 activation. I'll most likely build on that test and roll something into the nightly test set to do a 2/3 activation test. |
|
Before this goes in (or before propolis/omicron point to it) we must have oxidecomputer/omicron#8528 sorted. |
|
@claude review |
There was a problem hiding this comment.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
upstairs/src/downstairs.rs— When the third Downstairs is offline, min-quorum activation can never finish if repair takes longer than one reconnect cycle (about 10-20s). Every failed reconnect by the non-participating client callsDownstairs::reinitialize. That aborts reconciliation whenever any other client is inReconcile(downstairs.rs:735-745). Fix: abort only when the client being reinitialized is one ofreconcile.participating, or is itself inReconcile, matching the participant-only check already added inon_reconciliation_ack.Why this was flagged
Setup: GoActive with DS0 and DS1 in WaitQuorum and DS2 unreachable. After
NEGOTIATION_DELAY,on_min_quorum(upstairs.rs:2142-2165) starts reconciliation for DS0 and DS1 only and marks DS2 Faulted. DS2's IO task keeps failing to connect (ClientRunResult::ConnectionFailed/ConnectionTimeout, client.rs:2387/2405). It retries everyCLIENT_RECONNECT_DELAY(10s). Each failure arrives asClientAction::TaskStoppedand goes toon_client_task_stopped, thenDownstairs::reinitialize(DS2). The special case at downstairs.rs:735-745 sees DS0/DS1 inReconcileand callsabort_reconciliation, which halts both participants withFailedReconcile. They renegotiate, the 500ms timer fires again, and the cycle repeats. Any reconciliation longer than DS2's retry interval never completes, so activation hangs. This is exactly the one-Downstairs-down case RFD 542 targets. On base, activation simply waited for all three. The participant-only check added inon_reconciliation_ack(downstairs.rs:1954) does not help, because this abort path never consultsparticipating.Verification:
Downstairs::reinitialize(downstairs.rs:735-745) callsabort_reconciliation(up_state)if ANY client is inNegotiationState::Reconcile. It does not check whether the client being reinitialized is inreconcile.participating. On the base branch reconciliation needed all three clients, so a client that couldn't connect never overlapped with a running reconciliation.
Updated a test that needed a slightly different starting point since things on main have moved on since this PR started.
A new tests to verify that if we activate with 2/3, the third downstairs does not get marked as been active.
This plumbs in the BeenActive status of a downstairs into the min-quorum decision making process. To avoid having a replacement downstairs count as one of two in the min-quorum activation, we added the BeenActive status for each downstairs. Only BeenActive downstairs can count toward min-quorum. This also adds a bunch of tests around this new feature.
|
I'm going to fix #2072 before I merge more into |
# Conflicts: # crutest/src/main.rs
This reverts commit 16c0b32. This reverts the runtime restriction from "Remove replacement before reconciliation." That restriction kept a new, empty replacement from being one of the two downstairs in a min-quorum activation. The been_active check now prevents that, because a new region has never been active. The restriction also stopped a read-only parent from having a target replaced after its scrub. Restore the replace-before-active and replace-reconcile tests and test_replace_special.sh in the nightly run. Keep the two-of-three crutest test.
When two downstairs activate through min-quorum, the third downstairs may be stopping, for example because it is being replaced. on_min_quorum set its connection mode to faulted, and on_reconciliation_done panicked on a downstairs in the Stopping state. If the third downstairs is stopping because of a replacement, a failed negotiation, or a fault, leave it alone in both places. It picks its connection mode when it restarts, and since we will be active by then, it comes back through live-repair. Add upstairs tests that replace the third downstairs before the min-quorum deadline, with and without reconciliation work, and during min-quorum reconciliation.
We need to be sure (for some tests) that our downstairs has had time to write out that it has been active, as a follow on action may expect it to be so. Update the tests to confirm that this has been done. This avoids test flakes.
Replace the crutest replace-before-active workload with min_quorum_replace_before_active, which replaces an offline Downstairs before activation and checks that the new region, which has never been active, can't make up a min-quorum with one that has. Make NEGOTIATION_DELAY public so the min-quorum integration tests base their waits on it.
After starting new_ds, the test waited for downrev_ds to be running instead of new_ds. Wait for new_ds, and fix the missing dsc endpoint error message, which named the replace reconcile test.
With min-quorum, a loop of replace_while_reconcile can take more than one path and the test will pass whichever one it takes. Instead of sleeping and checking that the upstairs is not active, poll volume info until reconciliation is in progress before replacing the downstairs. If the upstairs goes active first, count it as a miss and try again and give up after five misses in a row. We record which path each loop takes, whether the reconciliation was three-way or two-way (with or without the downstairs being replaced), whether the upstairs was already active right after the replace, and whether the new downstairs joined through reconciliation or live-repair. Logs are printed on each loop's path, marking whether it was the intended one, and report how many loops took the intended path and which other paths were taken. Also clean up some comments in crutest, integration_tests, and the upstairs min-quorum code.
If all three downstairs need live-repair while the upstairs is active, we switch to reconciliation. If reconciliation looks at all the extents and sees nothing to do, all three will still be in LiveRepairReady when reconciliation finishes. on_reconciliation_done used to set every client active. The min-quorum change made it set only clients in WaitQuorum or Reconcile active, and mark any other client that are still connecting as faulted. That handles the downstairs left out of a min-quorum activation, but it missed this case. All three clients fell through to the faulted branch and were never set active. Now we set them active when all three are in LiveRepairReady. A single downstairs left out of a min-quorum activation can also be in LiveRepairReady when reconciliation finishes, and it must still go through live-repair. It is never set active here, because the two downstairs that took part in the min-quorum reconciliation are in WaitQuorum or Reconcile.
The tests in test_up.sh were re-using the downstairs from previous tests, and this was causing problems when a previous test had shifted around which downstairs were part of which region set. Fixed the order and added a comment.
When a downstairs task stops while reconciliation is running, we abort reconciliation. With min-quorum, the third downstairs is not part of the reconciliation. If the third downstairs is offline, its task fails to connect every 10 to 20 seconds, and each failure aborted reconciliation, so a min-quorum reconciliation that took longer than that could never finish. A downstairs left out of the reconciliation now lets reconciliation keep running, and the left-out downstairs restarts in the Faulted mode so it comes back through live-repair. Without that, after a failed negotiation the left-out downstairs would restart in the New mode and could reach WaitQuorum while reconciliation is running. A downstairs being replaced still aborts reconciliation. The replacement's new region has a different UUID, and a client in the Faulted mode panics on a UUID change once the upstairs is active.
Do a better job of waiting for clients to come back using how long the client task will wait itself as input into the wait time.
|
@claude review |
There was a problem hiding this comment.
🔴 After an aborted min-quorum reconciliation, the left-out Downstairs stays on the live-repair path, so activation can hang even when a usable quorum is present. on_min_quorum (upstairs.rs:2213) sets its mode to Faulted. abort_reconciliation only restarts clients in Reconcile, and the upstairs stays GoActive. That client then sits in LiveRepairReady, which on_wait_quorum never counts. Fix: when reconciliation is aborted before going active, put every non-participating client back on the ConnectionMode::New path (restart it if it is already in LiveRepairReady), so it can reach WaitQuorum again.
Why this was flagged
In on_min_quorum's Ok(true) branch (upstairs.rs:2213), and in reinitialize (downstairs.rs:779), the unready client C gets set_connection_mode_faulted(). Then participant A sends ExtentError, or A is replaced or disconnects mid-reconcile. abort_reconciliation (downstairs.rs:2072-2093) runs abort_negotiation only on clients in Reconcile, so C keeps ConnectionMode::Faulted, and the upstairs stays GoActive. C finishes negotiation into LiveRepairReady (client.rs:1662). on_wait_quorum counts only WaitQuorum clients. If A stays down, B and C have both been active, but the count is 1 and activation never happens. If A is replaced by a new region (been_active false), min_quorum_allowed fails and full quorum needs C in WaitQuorum, so activation hangs until the upstairs is deactivated. On the base, C was never forced to Faulted during GoActive, so it reached WaitQuorum and full-quorum activation went ahead.
Verification: nit. Trigger: a participant (A) in a running min-quorum reconciliation disconnects, sends ExtentError, or is replaced, and then stays unavailable. abort_reconciliation (downstairs.rs:2077-2093) calls abort_negotiation only on clients in NegotiationState::Reconcile, so the left-out client C keeps mode: Faulted. on_wait_quorum counts only WaitQuorum clients (upstairs.rs:2040-2048).
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🟡
crutest/src/main.rs— The replace-reconcile crutest fails on a path it calls "fine": a min-quorum reconciliation that includes the old downstairs. There the replace is rejected and the loop bails with "Failed replace". The third downstairs left out of min-quorum is inConnectionMode::Faulted, andDownstairs::replace(downstairs.rs:2862-2876) rejects any replace while another client is Faulted. Fix: when the loop seesReconcileKind::TwoWayWithOld(or an active min-quorum volume with one client still faulted), skip the replace or wait for the left-out downstairs to rejoin before replacing.Why this was flagged
The upstairs reconciles min-quorum when, among three running been-active downstairs, one reaches WaitQuorum more than
NEGOTIATION_DELAY(500ms) after the other two. On a loaded CI host this can happen to a downstairs other thanold_ds. The poll loop classifies that asReconcileKind::TwoWayWithOld(crutest/src/main.rs:2810), then callsvolume.replace_downstairs(old, new).on_min_quorum(upstairs.rs:2213) has already put the left-out client intoConnectionMode::Faulted.Downstairs::replaceloops over the other clients and returns aReplaceRequestInvaliderror forConnecting { mode: Faulted, .. }(downstairs.rs:2862-2876). Thex => bail!("Failed replace: {:?}", x)arm at main.rs:2865 aborts the test, which makestest_up.shfail. The newReplaceReconcilePathsdocs (main.rs:2605-2606, 2642) say these paths are fine and get counted, but the replace step can never succeed on them. On the base branch activation always needed all three downstairs, so this rejection could not happen.Verification: nit (test-only flake).
on_min_quorum(upstairs/src/upstairs.rs:2207-2213) sets the left-out client toConnectionMode::Faulted. crutest classifies the loop asReconcileKind::TwoWayWithOld(crutest/src/main.rs:2803-2813) and callsvolume.replace_downstairs(old, new).Downstairs::replace(downstairs.rs:2862-2880) rejects it, andbail!("Failed replace: {:?}", x)(main.rs:2864-2866) fails the workload.
| DsState::Stopping( | ||
| ClientStopReason::Replacing | ||
| | ClientStopReason::NegotiationFailed(..) | ||
| | ClientStopReason::Fault(..) | ||
| ) | ||
| ) { | ||
| info!( | ||
| self.log, | ||
| "{unready_ds} is stopping, it will pick its mode \ | ||
| when it restarts" | ||
| ); | ||
| } else { | ||
| info!( | ||
| self.log, | ||
| "Requiring {unready_ds} to connect through live-repair" | ||
| ); | ||
| self.downstairs.clients[unready_ds] | ||
| .set_connection_mode_faulted(); |
There was a problem hiding this comment.
🔴 A downstairs replaced while activation is in progress can crash the Upstairs once it connects after min-quorum activation. The replaced client restarts in New mode and waits CLIENT_RECONNECT_DELAY (10s) before connecting. During that wait, on_min_quorum or on_reconciliation_done switches it to Faulted mode, and it still holds the old region_uuid. Fix: the client must not fail the UUID check after a replacement whatever its mode, e.g. clear region_uuid in DownstairsClient::replace, and do not overwrite a replaced client's mode with Faulted. [also at: upstairs/src/upstairs.rs:2203 - A Downstairs replaced before activation can crash the Upstairs once min-quorum activates without it. On the base branch the replacement joined through full-quorum WaitQuorum.]
Why this was flagged
The trigger is BlockOp::ReplaceDownstairs arriving while the Upstairs is GoActive and the other two been-active downstairs are reaching WaitQuorum. DownstairsClient::replace (client.rs:902-910) changes only target_addr, so region_uuid keeps the old region's UUID. After TaskStopped, reinitialize gives ConnectionMode::New (client.rs:606-613) and the client waits 10s in Start. When the 500ms NEGOTIATION_DELAY fires, on_min_quorum sees the client is not Stopping and calls set_connection_mode_faulted (upstairs.rs:2213). When the client finally connects, RegionInfo returns a new UUID, the mode is Faulted rather than Replaced, and up_state is Active. client.rs:1546-1557 then panics with "New client:... does not match existing". The new guards cover only the Stopping(Replacing) window (upstairs.rs:2196-2205), not the much longer reconnect-delay window. On the base branch, activation needed all three in WaitQuorum, so this client could not be left behind in Faulted mode.
Verification: DownstairsClient::replace (client.rs:902-910) changes only target_addr and stats. A client in Connecting{Start, New} gets set_connection_mode_faulted(). When the new downstairs finally sends RegionInfo, client.rs:1523-1556 sees uuid != region_def.uuid(). The mode is Faulted, not Replaced, and up_state is Active, so it reaches the panic arm and the Upstairs aborts.
(This is a revival of #1733, rebased onto
main, and then merged with the V15 protocol bump)This implements RFD 542 and closes #1690.This implements RFD 542 and closes #1690. It builds on the been-active
negotiation (#2050, #2052, #2068, #2070) and the dsc region recreate support
for crutest (#2073, #2078), which are on main.
Min-quorum activation
Today, the upstairs needs all three downstairs in WaitQuorum before it can
reconcile and go active. With this PR:
we schedule an event to fire after
NEGOTIATION_DELAY(500 ms).full-quorum reconciliation as on main.
them. The third downstairs is set to Faulted, so it must rejoin through
a live-repair.
replacement) doesn't count toward the two. A new region set still needs
all three downstairs for its first activation. The work to support this
came in with the over the wire protocol bump to V14.
Reconciliation now tracks which downstairs are taking part, and only sends
reconciliation work to those.
The downstairs left out of min-quorum
If the third downstairs is offline or has other problems while min-quorum
reconciliation is running, reconciliation keeps going. The offline
downstairs restarts in Faulted mode, so it can't reach WaitQuorum while
reconciliation is running. Replacing an offline downstairs during
reconciliation will still abort reconciliation. By aborting the whole
reconciliation, we will reset what we expect from the replaced downstairs.
set_connection_mode_faultedno longer requires the downstairs to beOffline; it works for any connecting downstairs.
Finishing reconciliation
on_reconciliation_doneused to set every downstairs active. It now setsactive only the downstairs that took part (those in WaitQuorum or
Reconcile), and sets any other connecting downstairs to the Faulted mode.
The one exception is when all three needed live-repair: the upstairs
switches to reconciliation, and if that reconciliation finds nothing to do,
all three downstairs are still in LiveRepairReady. In this case we can
allow all three to go active.
Tests
Upstairs unit tests for min-quorum: activating with two been-active
downstairs, not activating when one hasn't been active, cancelling when
the third arrives, the third downstairs being replaced, offline, or
failing negotiation before and during reconciliation, and all three
downstairs needing live-repair with nothing to reconcile.
New integration tests:
connect_two_ds_then_deactivate,connect_two_ds_then_another,min_quorum_live_repair,min_quorum_cancel, andmin_quorum_replace_before_active.min_quorum_replace_before_activereplaces an offline downstairs beforeactivation, and checks that the new region can't make up a min-quorum with
one that has been active. It replaces the crutest replace-before-active
workload, which is removed (along with its
test_up.shstep).Integration test helpers:
activate_all_onceactivates all threedownstairs once and waits until each has recorded that it has been
active, so later activations can use min-quorum.
stopreturns thedownstairs ports so a test can restart it on the same ones.
NEGOTIATION_DELAYandCLIENT_RECONNECT_DELAYare now public. Thisallows integration tests to use them to base their waits on.
crutest and scripts
A new crutest two-of-three workload: it stops one downstairs, writes, then
activates with two (one of which needs reconciliation), and checks the
third comes back through live-repair.
test_up.shruns it beforereplace-reconcile, because replace-reconcile moves regions between
sub_volumes and this makes reuse of the regions for future tests fragile.
For crutest replace-reconcile: we can't control exactly which path each loop
takes, since that depends on timing inside the upstairs, but we can tell
which one it took. The test now waits until reconciliation is actually
in progress before replacing (instead of a fixed sleep), and reports how
many loops took the intended path (replace during a three-way
reconciliation, with the new downstairs joining through reconciliation)
and which other paths were taken. In 100 local loops, all 100 took the
intended path.
test_repair.shwaits for the downstairs to start before verifying.Before merging
before propolis/omicron point to it).