Repository navigation
feat(converge): deploy the newest commit that passed CI, not only the tip (#1026) - #1032
Merged
Merged
Conversation
… tip (#1026) converge.sh asked CI about main's tip alone and waited while it ran. On a busy main a new tip usually lands before the last one's CI finishes, so every hourly run waited. On 2026-10-07 trinity sat on f7afe9c from 12:29 to past 15:00, 23 commits behind, while e4cf127 had already passed. A tip still in CI now starts a walk down main's first-parent history (ci_newest_green). It deploys the newest commit that is a fast-forward from HEAD, verifies against the pinned key, and is green by ci_verdict. - A red commit on the way is skipped: a newer green one carries its changes. A red tip still refuses, as before. - The walk asks about at most HOMELAB_CONVERGE_WALK commits (default 10), because the API calls are anonymous, and it stops at the first ask that fails. homelab_deploy_ci_fallback is 1 on a run that applied such a commit. It is set only after make up, so a dry run or report-only mode never claims it. DeployBehind leaves those runs alone (`unless`), so it fires on 3h without progress rather than on 3h of a busy main. A host whose converge.sh predates the gauge still alerts as before. Fixtures: - converge --self-test (86, 17 new): the fallback, then staying put; a red commit skipped; an unverified commit skipped; the walk cap; an ask failing mid-walk; a dry run. - promtool: busy-but-moving, stopped-moving, and pre-gauge hosts. Mutation-checked: dropping the walk's signature check, its cap, or the after-make-up timing of the gauge each fails the matching fixture. Refs #1026 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The walk cap and API-failure handling are incorrect, and failed fallback deployments may not be retried.
3 open findings
What changed in this PR
Adds CI-aware fallback deployment so convergence can advance to the newest signed, green commit below a still-running tip.
Changes:
- Walks bounded first-parent history for deployable commits.
- Records fallback deployment metrics and updates alert behavior.
- Adds self-tests, Prometheus tests, and operational documentation.
| File | Description |
|---|---|
scripts/converge.sh |
Implements fallback selection, metrics, and fixtures. |
stacks/observability/prometheus/rules/deploy.rules.yaml |
Suppresses behind alerts while fallback deployments progress. |
stacks/observability/prometheus/tests/deploy.test.yaml |
Tests fallback-aware alert timing. |
docs/runbooks/converge-the-host.md |
Documents fallback behavior and metrics. |
docs/changelog.md |
Records the new convergence behavior. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…t-green # Conflicts: # docs/changelog.md
…d at, and don't walk after a failed ask (#1026) From review on the walk: - A fallback whose make up failed was never retried. The next run's walk started above HEAD, found nothing newer green, and stayed. The walk now offers the unapplied HEAD as its last candidate, the same way a failed deploy of the tip was always re-applied. - The cap counted only commits it asked CI about, so any number of skipped commits could be walked past. It now counts every commit looked at. - A tip whose own ask failed still started the walk, spending more of the anonymous API limit that likely caused the failure. It now stays put. Three new cases in converge --self-test (94 checks, all pass), and each fails when its fix is reverted. Refs #1026 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Refs #1026. I'd close the issue after the first live fallback shows in a journal; see the last section.
What
When
main's tip is still in CI,converge.shno longer just waits. It walksmain's first-parent history down from the tip, newest first, and deploys the first commit that:verify_commit), andci_verdict.--allow-redis unchanged.--allow-unsignedcovers the tip a person is looking at, so it doesn't extend to commits the walk finds.HOMELAB_CONVERGE_WALKcommits (default 10).ci_fetchis anonymous, which allows 60 calls an hour per address, and both converging hosts share one address. An ask that fails stops the walk, since the likeliest cause is that limit.Metrics and the alert
homelab_deploy_ci_fallback:1on a run that applied a commit below the tip. It is set only aftermake upsucceeds, so a dry run or report-only mode never claims it.behind_commitsnow records how far the host still is from the tip after a fallback.DeployBehindbecomesbehind_commits > 0 unless on(instance) ci_fallback == 1. A host that moves every hour on a busymainno longer pages. Three hours without progress still fires, because a run that couldn't move records0and restarts thefor:timer. A host whoseconverge.shpredates the gauge has no matching series, so it still alerts as before.Why
On 2026-10-07 trinity sat on
f7afe9cfrom 12:29 to past 15:00 UTC, 23 commits behind. Every hourly run found a tip still in CI, whilee4cf127, newer than what was deployed, had already passed. #966's Postgres re-pin and #1022's Caddy and Vaultwarden security bumps were held back the whole time.Verification
converge.sh --self-test: 86 fixtures pass, 17 of them new. Each runs end to end against a real repository, real signatures and a stubbed CI API that now answers per commit:make upeach fails the matching fixture.DeployBehindblock covers three hosts: busy-but-moving stays silent, stopped-moving fires 3h after it stopped, and a host without the gauge fires as before. Removing theunlessfails it.make validatepasses. shellcheck is clean.After merge
This deploys through converge like everything else. The new script takes effect on the run after the one that deploys it. The next time a tip is still in CI while an older commit has passed, the journal should show
CI passed on <sha>, N commit(s) below the tip — deploying that, and thehomelab-deployline should showci_fallback=1. That is the last item before #1026 can be closed.🤖 Generated with Claude Code