Skip to content

feat(converge): deploy the newest commit that passed CI, not only the tip (#1026) - #1032

Merged
Gerrrt merged 3 commits into
mainfrom
gerrrt/converge-newest-green
Oct 7, 2026
Merged

Gerrrt merged 3 commits into
mainfrom
gerrrt/converge-newest-green

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Refs #1026. I'd close the issue after the first live fallback shows in a journal; see the last section.

What

When main's tip is still in CI, converge.sh no longer just waits. It walks main's first-parent history down from the tip, newest first, and deploys the first commit that:

  1. is a fast-forward from what the host runs,
  2. carries a good signature from the pinned key (verify_commit), and
  3. is green by ci_verdict.
  • A red commit along the way is skipped. A newer green commit carries its changes, and its own run says whether the result passed. A red tip still refuses, exactly as before, and --allow-red is unchanged.
  • A commit that doesn't verify is skipped, whatever its CI says. --allow-unsigned covers the tip a person is looking at, so it doesn't extend to commits the walk finds.
  • The walk is capped at HOMELAB_CONVERGE_WALK commits (default 10). ci_fetch is anonymous, which allows 60 calls an hour per address, and both converging hosts share one address. An ask that fails stops the walk, since the likeliest cause is that limit.
  • The run that already matches the tip is unchanged: no walk and no API call.

Metrics and the alert

  • New gauge homelab_deploy_ci_fallback: 1 on a run that applied a commit below the tip. It is set only after make up succeeds, so a dry run or report-only mode never claims it.
  • behind_commits now records how far the host still is from the tip after a fallback.
  • DeployBehind becomes behind_commits > 0 unless on(instance) ci_fallback == 1. A host that moves every hour on a busy main no longer pages. Three hours without progress still fires, because a run that couldn't move records 0 and restarts the for: timer. A host whose converge.sh predates the gauge has no matching series, so it still alerts as before.

Why

On 2026-10-07 trinity sat on f7afe9c from 12:29 to past 15:00 UTC, 23 commits behind. Every hourly run found a tip still in CI, while e4cf127, newer than what was deployed, had already passed. #966's Postgres re-pin and #1022's Caddy and Vaultwarden security bumps were held back the whole time.

Verification

  • converge.sh --self-test: 86 fixtures pass, 17 of them new. Each runs end to end against a real repository, real signatures and a stubbed CI API that now answers per commit:
    • deploys the newest green commit below a running tip, then stays put on the next run
    • skips a red commit in the middle and deploys the green one below it
    • skips a green commit that doesn't verify
    • stops at the walk cap, short of a green commit (and counts the API calls)
    • stops when an ask fails, without asking further
    • a dry run reports the fallback but applies and records nothing
  • The fixtures were mutation-checked. Dropping the walk's signature check, ignoring the cap, or setting the gauge before make up each fails the matching fixture.
  • promtool: a new DeployBehind block covers three hosts: busy-but-moving stays silent, stopped-moving fires 3h after it stopped, and a host without the gauge fires as before. Removing the unless fails it.
  • make validate passes. shellcheck is clean.

After merge

This deploys through converge like everything else. The new script takes effect on the run after the one that deploys it. The next time a tip is still in CI while an older commit has passed, the journal should show CI passed on <sha>, N commit(s) below the tip — deploying that, and the homelab-deploy line should show ci_fallback=1. That is the last item before #1026 can be closed.

🤖 Generated with Claude Code

… tip (#1026)

converge.sh asked CI about main's tip alone and waited while it ran.
On a busy main a new tip usually lands before the last one's CI
finishes, so every hourly run waited. On 2026-10-07 trinity sat on
f7afe9c from 12:29 to past 15:00, 23 commits behind, while e4cf127
had already passed.

A tip still in CI now starts a walk down main's first-parent history
(ci_newest_green). It deploys the newest commit that is a fast-forward
from HEAD, verifies against the pinned key, and is green by ci_verdict.

- A red commit on the way is skipped: a newer green one carries its
  changes. A red tip still refuses, as before.
- The walk asks about at most HOMELAB_CONVERGE_WALK commits (default
  10), because the API calls are anonymous, and it stops at the first
  ask that fails.

homelab_deploy_ci_fallback is 1 on a run that applied such a commit.
It is set only after make up, so a dry run or report-only mode never
claims it. DeployBehind leaves those runs alone (`unless`), so it fires
on 3h without progress rather than on 3h of a busy main. A host whose
converge.sh predates the gauge still alerts as before.

Fixtures:
- converge --self-test (86, 17 new): the fallback, then staying put; a
  red commit skipped; an unverified commit skipped; the walk cap; an
  ask failing mid-walk; a dry run.
- promtool: busy-but-moving, stopped-moving, and pre-gauge hosts.

Mutation-checked: dropping the walk's signature check, its cap, or the
after-make-up timing of the gauge each fails the matching fixture.

Refs #1026

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 15:50
@Gerrrt Gerrrt added feature A new capability or a change observability stacks/observability on prometheus sensitive stacks/sensitive on trinity (VLAN 99) labels Oct 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The walk cap and API-failure handling are incorrect, and failed fallback deployments may not be retried.

3 open findings
What changed in this PR

Adds CI-aware fallback deployment so convergence can advance to the newest signed, green commit below a still-running tip.

Changes:

  • Walks bounded first-parent history for deployable commits.
  • Records fallback deployment metrics and updates alert behavior.
  • Adds self-tests, Prometheus tests, and operational documentation.
File Description
scripts/​converge.sh Implements fallback selection, metrics, and fixtures.
stacks/​observability/​prometheus/​rules/​deploy.rules.yaml Suppresses behind alerts while fallback deployments progress.
stacks/​observability/​prometheus/​tests/​deploy.test.yaml Tests fallback-aware alert timing.
docs/​runbooks/​converge-the-host.md Documents fallback behavior and metrics.
docs/​changelog.md Records the new convergence behavior.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/converge.sh Resolved
Comment thread scripts/converge.sh Resolved
Comment thread scripts/converge.sh Resolved
Gerrrt and others added 2 commits October 7, 2026 16:21
…d at, and don't walk after a failed ask (#1026)

From review on the walk:

- A fallback whose make up failed was never retried. The next run's
  walk started above HEAD, found nothing newer green, and stayed. The
  walk now offers the unapplied HEAD as its last candidate, the same
  way a failed deploy of the tip was always re-applied.
- The cap counted only commits it asked CI about, so any number of
  skipped commits could be walked past. It now counts every commit
  looked at.
- A tip whose own ask failed still started the walk, spending more of
  the anonymous API limit that likely caused the failure. It now stays
  put.

Three new cases in converge --self-test (94 checks, all pass), and each
fails when its fix is reverted.

Refs #1026

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit bb73262 into main Oct 7, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/converge-newest-green branch October 7, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature A new capability or a change observability stacks/observability on prometheus sensitive stacks/sensitive on trinity (VLAN 99)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants