Skip to content

Min-quorum negotiation (again) - #1908

Open
mkeeter wants to merge 23 commits into
mainfrom
mkeeter/min-quorum-negotiation-redux
Open

mkeeter wants to merge 23 commits into
mainfrom
mkeeter/min-quorum-negotiation-redux

Conversation

@mkeeter

@mkeeter mkeeter commented Mar 17, 2026 •

Copy link
Copy Markdown
Contributor

(This is a revival of #1733, rebased onto main, and then merged with the V15 protocol bump)


This implements RFD 542 and closes #1690.This implements RFD 542 and closes #1690. It builds on the been-active
negotiation (#2050, #2052, #2068, #2070) and the dsc region recreate support
for crutest (#2073, #2078), which are on main.

Min-quorum activation

Today, the upstairs needs all three downstairs in WaitQuorum before it can
reconcile and go active. With this PR:

  • Once two downstairs are in WaitQuorum, and both have been active before,
    we schedule an event to fire after NEGOTIATION_DELAY (500 ms).
  • If the third downstairs reaches WaitQuorum before the event fires, we do
    full-quorum reconciliation as on main.
  • Otherwise, we reconcile with just the two downstairs and go active with
    them. The third downstairs is set to Faulted, so it must rejoin through
    a live-repair.
  • A downstairs that has never been active (for example, a new region, or a
    replacement) doesn't count toward the two. A new region set still needs
    all three downstairs for its first activation. The work to support this
    came in with the over the wire protocol bump to V14.

Reconciliation now tracks which downstairs are taking part, and only sends
reconciliation work to those.

The downstairs left out of min-quorum

If the third downstairs is offline or has other problems while min-quorum
reconciliation is running, reconciliation keeps going. The offline
downstairs restarts in Faulted mode, so it can't reach WaitQuorum while
reconciliation is running. Replacing an offline downstairs during
reconciliation will still abort reconciliation. By aborting the whole
reconciliation, we will reset what we expect from the replaced downstairs.

set_connection_mode_faulted no longer requires the downstairs to be
Offline; it works for any connecting downstairs.

Finishing reconciliation

on_reconciliation_done used to set every downstairs active. It now sets
active only the downstairs that took part (those in WaitQuorum or
Reconcile), and sets any other connecting downstairs to the Faulted mode.
The one exception is when all three needed live-repair: the upstairs
switches to reconciliation, and if that reconciliation finds nothing to do,
all three downstairs are still in LiveRepairReady. In this case we can
allow all three to go active.

Tests

Upstairs unit tests for min-quorum: activating with two been-active
downstairs, not activating when one hasn't been active, cancelling when
the third arrives, the third downstairs being replaced, offline, or
failing negotiation before and during reconciliation, and all three
downstairs needing live-repair with nothing to reconcile.

New integration tests: connect_two_ds_then_deactivate,
connect_two_ds_then_another, min_quorum_live_repair,
min_quorum_cancel, and min_quorum_replace_before_active.
min_quorum_replace_before_active replaces an offline downstairs before
activation, and checks that the new region can't make up a min-quorum with
one that has been active. It replaces the crutest replace-before-active
workload, which is removed (along with its test_up.sh step).

Integration test helpers: activate_all_once activates all three
downstairs once and waits until each has recorded that it has been
active, so later activations can use min-quorum. stop returns the
downstairs ports so a test can restart it on the same ones.

NEGOTIATION_DELAY and CLIENT_RECONNECT_DELAY are now public. This
allows integration tests to use them to base their waits on.

crutest and scripts

A new crutest two-of-three workload: it stops one downstairs, writes, then
activates with two (one of which needs reconciliation), and checks the
third comes back through live-repair. test_up.sh runs it before
replace-reconcile, because replace-reconcile moves regions between
sub_volumes and this makes reuse of the regions for future tests fragile.

For crutest replace-reconcile: we can't control exactly which path each loop
takes, since that depends on timing inside the upstairs, but we can tell
which one it took. The test now waits until reconciliation is actually
in progress before replacing (instead of a fixed sleep), and reports how
many loops took the intended path (replace during a three-way
reconciliation, with the new downstairs joining through reconciliation)
and which other paths were taken. In 100 local loops, all 100 took the
intended path.

test_repair.sh waits for the downstairs to start before verifying.

Before merging

@mkeeter
mkeeter requested review from jmpesp and leftwo March 17, 2026 13:54
@AlejandroME AlejandroME modified the milestone: 20 Mar 24, 2026
Comment thread crutest/src/main.rs Resolved
@leftwo

leftwo commented May 14, 2026

Copy link
Copy Markdown
Contributor

I've started on removing the "replace before reconcile" ability for this PR.
More changes coming soon

With 2/3 activation possible, we can no longer allow a downstairs
to be replaced before we have gone through reconciliation.

Added a new crutest test for 2/3 activation.
Removed or changed a few tests that attempted to replace a
downstairs before activation.
@leftwo
leftwo self-requested a review May 14, 2026 20:39
@leftwo

leftwo commented May 14, 2026

Copy link
Copy Markdown
Contributor

I've removed the tests and ability to replace a downstairs before activation, which I believe is what we want now that we have 2/3 activation.

I also added a specific crutest test for 2/3 activation. I'll most likely build on that test and roll something into the nightly test set to do a 2/3 activation test.

Comment thread crutest/src/main.rs Resolved
@leftwo
leftwo marked this pull request as ready for review May 18, 2026 17:27
@leftwo

leftwo commented May 18, 2026

Copy link
Copy Markdown
Contributor

Before this goes in (or before propolis/omicron point to it) we must have oxidecomputer/omicron#8528 sorted.

@leftwo

leftwo commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🔴 upstairs/src/downstairs.rs — When the third Downstairs is offline, min-quorum activation can never finish if repair takes longer than one reconnect cycle (about 10-20s). Every failed reconnect by the non-participating client calls Downstairs::reinitialize. That aborts reconciliation whenever any other client is in Reconcile (downstairs.rs:735-745). Fix: abort only when the client being reinitialized is one of reconcile.participating, or is itself in Reconcile, matching the participant-only check already added in on_reconciliation_ack.

    Why this was flagged

    Setup: GoActive with DS0 and DS1 in WaitQuorum and DS2 unreachable. After NEGOTIATION_DELAY, on_min_quorum (upstairs.rs:2142-2165) starts reconciliation for DS0 and DS1 only and marks DS2 Faulted. DS2's IO task keeps failing to connect (ClientRunResult::ConnectionFailed/ConnectionTimeout, client.rs:2387/2405). It retries every CLIENT_RECONNECT_DELAY (10s). Each failure arrives as ClientAction::TaskStopped and goes to on_client_task_stopped, then Downstairs::reinitialize(DS2). The special case at downstairs.rs:735-745 sees DS0/DS1 in Reconcile and calls abort_reconciliation, which halts both participants with FailedReconcile. They renegotiate, the 500ms timer fires again, and the cycle repeats. Any reconciliation longer than DS2's retry interval never completes, so activation hangs. This is exactly the one-Downstairs-down case RFD 542 targets. On base, activation simply waited for all three. The participant-only check added in on_reconciliation_ack (downstairs.rs:1954) does not help, because this abort path never consults participating.

    Verification: Downstairs::reinitialize (downstairs.rs:735-745) calls abort_reconciliation(up_state) if ANY client is in NegotiationState::Reconcile. It does not check whether the client being reinitialized is in reconcile.participating. On the base branch reconciliation needed all three clients, so a client that couldn't connect never overlapped with a running reconciliation.

Comment thread upstairs/src/upstairs.rs Resolved
Comment thread upstairs/src/downstairs.rs Resolved
Comment thread upstairs/src/upstairs.rs Resolved
Updated a test that needed a slightly different starting point
since things on main have moved on since this PR started.
A new tests to verify that if we activate with 2/3, the third
downstairs does not get marked as been active.
This plumbs in the BeenActive status of a downstairs into the
min-quorum decision making process.  To avoid having a replacement
downstairs count as one of two in the min-quorum activation, we
added the BeenActive status for each downstairs.  Only BeenActive
downstairs can count toward min-quorum.

This also adds a bunch of tests around this new feature.
@leftwo

leftwo commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

I'm going to fix #2072 before I merge more into
this PR, as we need that fixed to properly test min-quorum.

This reverts commit 16c0b32.

This reverts the runtime restriction from "Remove replacement before
reconciliation."  That restriction kept a new, empty replacement
from being one of the two downstairs in a min-quorum activation.  The
been_active check now prevents that, because a new region has never
been active.  The restriction also stopped a read-only parent from
having a target replaced after its scrub.

Restore the replace-before-active and replace-reconcile tests and
test_replace_special.sh in the nightly run.  Keep the two-of-three
crutest test.
When two downstairs activate through min-quorum, the third downstairs may
be stopping, for example because it is being replaced.  on_min_quorum set
its connection mode to faulted, and on_reconciliation_done panicked on a
downstairs in the Stopping state.

If the third downstairs is stopping because of a replacement, a failed
negotiation, or a fault, leave it alone in both places.  It picks its
connection mode when it restarts, and since we will be active by then, it
comes back through live-repair.

Add upstairs tests that replace the third downstairs before the min-quorum
deadline, with and without reconciliation work, and during min-quorum
reconciliation.
We need to be sure (for some tests) that our downstairs
has had time to write out that it has been active, as
a follow on action may expect it to be so.  Update the
tests to confirm that this has been done.  This avoids
test flakes.
Replace the crutest replace-before-active workload with
min_quorum_replace_before_active, which replaces an offline Downstairs
before activation and checks that the new region, which has never been
active, can't make up a min-quorum with one that has.

Make NEGOTIATION_DELAY public so the min-quorum integration tests base
their waits on it.
After starting new_ds, the test waited for downrev_ds to be running
instead of new_ds.  Wait for new_ds, and fix the missing dsc endpoint
error message, which named the replace reconcile test.
With min-quorum, a loop of replace_while_reconcile can take more than
one path and the test will pass whichever one it takes.  Instead of
sleeping and checking that the upstairs is not active, poll volume
info until reconciliation is in progress before replacing the downstairs.
If the upstairs goes active first, count it as a miss and try again and
give up after five misses in a row.

We record which path each loop takes, whether the reconciliation was
three-way or two-way (with or without the downstairs being replaced),
whether the upstairs was already active right after the replace, and
whether the new downstairs joined through reconciliation or live-repair.

Logs are printed on each loop's path, marking whether it was the
intended one, and report how many loops took the intended path and which
other paths were taken.

Also clean up some comments in crutest, integration_tests, and the
upstairs min-quorum code.
If all three downstairs need live-repair while the upstairs is active,
we switch to reconciliation.  If reconciliation looks at all the extents
and sees nothing to do, all three will still be in LiveRepairReady when
reconciliation finishes.

on_reconciliation_done used to set every client active.  The min-quorum
change made it set only clients in WaitQuorum or Reconcile active, and
mark any other client that are still connecting as faulted.  That handles
the downstairs left out of a min-quorum activation, but it missed this
case.  All three clients fell through to the faulted branch and were
never set active.

Now we set them active when all three are in LiveRepairReady.  A single
downstairs left out of a min-quorum activation can also be in
LiveRepairReady when reconciliation finishes, and it must still go
through live-repair.  It is never set active here, because the two
downstairs that took part in the min-quorum reconciliation are in
WaitQuorum or Reconcile.
The tests in test_up.sh were re-using the downstairs from
previous tests, and this was causing problems when a previous
test had shifted around which downstairs were part of which
region set.  Fixed the order and added a comment.
When a downstairs task stops while reconciliation is running, we abort
reconciliation.  With min-quorum, the third downstairs is not part of the
reconciliation.  If the third downstairs is offline, its task fails to
connect every 10 to 20 seconds, and each failure aborted reconciliation,
so a min-quorum reconciliation that took longer than that could never
finish.

A downstairs left out of the reconciliation now lets reconciliation keep
running, and the left-out downstairs restarts in the Faulted mode so it
comes back through live-repair.  Without that, after a failed
negotiation the left-out downstairs would restart in the New mode and
could reach WaitQuorum while reconciliation is running.

A downstairs being replaced still aborts reconciliation.  The
replacement's new region has a different UUID, and a client in the
Faulted mode panics on a UUID change once the upstairs is active.
Do a better job of waiting for clients to come back using
how long the client task will wait itself as input into
the wait time.
@leftwo

leftwo commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@claude review

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 After an aborted min-quorum reconciliation, the left-out Downstairs stays on the live-repair path, so activation can hang even when a usable quorum is present. on_min_quorum (upstairs.rs:2213) sets its mode to Faulted. abort_reconciliation only restarts clients in Reconcile, and the upstairs stays GoActive. That client then sits in LiveRepairReady, which on_wait_quorum never counts. Fix: when reconciliation is aborted before going active, put every non-participating client back on the ConnectionMode::New path (restart it if it is already in LiveRepairReady), so it can reach WaitQuorum again.

Why this was flagged

In on_min_quorum's Ok(true) branch (upstairs.rs:2213), and in reinitialize (downstairs.rs:779), the unready client C gets set_connection_mode_faulted(). Then participant A sends ExtentError, or A is replaced or disconnects mid-reconcile. abort_reconciliation (downstairs.rs:2072-2093) runs abort_negotiation only on clients in Reconcile, so C keeps ConnectionMode::Faulted, and the upstairs stays GoActive. C finishes negotiation into LiveRepairReady (client.rs:1662). on_wait_quorum counts only WaitQuorum clients. If A stays down, B and C have both been active, but the count is 1 and activation never happens. If A is replaced by a new region (been_active false), min_quorum_allowed fails and full quorum needs C in WaitQuorum, so activation hangs until the upstairs is deactivated. On the base, C was never forced to Faulted during GoActive, so it reached WaitQuorum and full-quorum activation went ahead.

Verification: nit. Trigger: a participant (A) in a running min-quorum reconciliation disconnects, sends ExtentError, or is replaced, and then stays unavailable. abort_reconciliation (downstairs.rs:2077-2093) calls abort_negotiation only on clients in NegotiationState::Reconcile, so the left-out client C keeps mode: Faulted. on_wait_quorum counts only WaitQuorum clients (upstairs.rs:2040-2048).

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🟡 crutest/src/main.rs — The replace-reconcile crutest fails on a path it calls "fine": a min-quorum reconciliation that includes the old downstairs. There the replace is rejected and the loop bails with "Failed replace". The third downstairs left out of min-quorum is in ConnectionMode::Faulted, and Downstairs::replace (downstairs.rs:2862-2876) rejects any replace while another client is Faulted. Fix: when the loop sees ReconcileKind::TwoWayWithOld (or an active min-quorum volume with one client still faulted), skip the replace or wait for the left-out downstairs to rejoin before replacing.

    Why this was flagged

    The upstairs reconciles min-quorum when, among three running been-active downstairs, one reaches WaitQuorum more than NEGOTIATION_DELAY (500ms) after the other two. On a loaded CI host this can happen to a downstairs other than old_ds. The poll loop classifies that as ReconcileKind::TwoWayWithOld (crutest/src/main.rs:2810), then calls volume.replace_downstairs(old, new). on_min_quorum (upstairs.rs:2213) has already put the left-out client into ConnectionMode::Faulted. Downstairs::replace loops over the other clients and returns a ReplaceRequestInvalid error for Connecting { mode: Faulted, .. } (downstairs.rs:2862-2876). The x => bail!("Failed replace: {:?}", x) arm at main.rs:2865 aborts the test, which makes test_up.sh fail. The new ReplaceReconcilePaths docs (main.rs:2605-2606, 2642) say these paths are fine and get counted, but the replace step can never succeed on them. On the base branch activation always needed all three downstairs, so this rejection could not happen.

    Verification: nit (test-only flake). on_min_quorum (upstairs/src/upstairs.rs:2207-2213) sets the left-out client to ConnectionMode::Faulted. crutest classifies the loop as ReconcileKind::TwoWayWithOld (crutest/src/main.rs:2803-2813) and calls volume.replace_downstairs(old, new). Downstairs::replace (downstairs.rs:2862-2880) rejects it, and bail!("Failed replace: {:?}", x) (main.rs:2864-2866) fails the workload.

Comment thread upstairs/src/upstairs.rs
Comment on lines +2196 to +2213
DsState::Stopping(
ClientStopReason::Replacing
| ClientStopReason::NegotiationFailed(..)
| ClientStopReason::Fault(..)
)
) {
info!(
self.log,
"{unready_ds} is stopping, it will pick its mode \
when it restarts"
);
} else {
info!(
self.log,
"Requiring {unready_ds} to connect through live-repair"
);
self.downstairs.clients[unready_ds]
.set_connection_mode_faulted();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 A downstairs replaced while activation is in progress can crash the Upstairs once it connects after min-quorum activation. The replaced client restarts in New mode and waits CLIENT_RECONNECT_DELAY (10s) before connecting. During that wait, on_min_quorum or on_reconciliation_done switches it to Faulted mode, and it still holds the old region_uuid. Fix: the client must not fail the UUID check after a replacement whatever its mode, e.g. clear region_uuid in DownstairsClient::replace, and do not overwrite a replaced client's mode with Faulted. [also at: upstairs/src/upstairs.rs:2203 - A Downstairs replaced before activation can crash the Upstairs once min-quorum activates without it. On the base branch the replacement joined through full-quorum WaitQuorum.]

Why this was flagged

The trigger is BlockOp::ReplaceDownstairs arriving while the Upstairs is GoActive and the other two been-active downstairs are reaching WaitQuorum. DownstairsClient::replace (client.rs:902-910) changes only target_addr, so region_uuid keeps the old region's UUID. After TaskStopped, reinitialize gives ConnectionMode::New (client.rs:606-613) and the client waits 10s in Start. When the 500ms NEGOTIATION_DELAY fires, on_min_quorum sees the client is not Stopping and calls set_connection_mode_faulted (upstairs.rs:2213). When the client finally connects, RegionInfo returns a new UUID, the mode is Faulted rather than Replaced, and up_state is Active. client.rs:1546-1557 then panics with "New client:... does not match existing". The new guards cover only the Stopping(Replacing) window (upstairs.rs:2196-2205), not the much longer reconnect-delay window. On the base branch, activation needed all three in WaitQuorum, so this client could not be left behind in Faulted mode.

Verification: DownstairsClient::replace (client.rs:902-910) changes only target_addr and stats. A client in Connecting{Start, New} gets set_connection_mode_faulted(). When the new downstairs finally sends RegionInfo, client.rs:1523-1556 sees uuid != region_def.uuid(). The mode is Faulted, not Replaced, and up_state is Active, so it reaches the panic arm and the Upstairs aborts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support activation with 2/3 read/write downstairs present.

3 participants