Skip to content

fix(deploy-agent): prove the log path with a marker line, not by waiting - #908

Merged
Gerrrt merged 3 commits into
mainfrom
gerrrt/deploy-agent-log-canary
Oct 4, 2026
Merged

Gerrrt merged 3 commits into
mainfrom
gerrrt/deploy-agent-log-canary

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Oct 4, 2026

Copy link
Copy Markdown
Owner

The gap

deploy-agent.sh's arrival check asked Loki for any line from the host newer than the deploy. On 2026-10-04, golem, an idle Proxmox Backup Server guest, logged nothing during the three-minute wait. The script reported Loki has no line newer than the deploy and exited non-zero.

The agent was fine:

  • Its own counters showed 324 entries sent, two pushes answered 204, and nothing dropped.
  • A hand-sent logger line reached Loki within seconds, by all three paths golem ships: the journal, /var/log/syslog and /var/log/user.log.

The fix

Once the agent passes its checks, the script sends a unique marker on the target:

logger -t homelab-deploy-check 'homelab-deploy-check <host> <epoch>-<pid>-<random>'

It then looks for exactly that text: {host="<host>"} |= "<marker>".

  • It proves the right thing. The marker goes through the host's own logging (journald, and rsyslog where it runs), the same path the host's real logs take. If it arrives, they do; if it doesn't, they don't.
  • It doesn't depend on the host being busy. That's the false failure golem hit.
  • Clock skew can't hide it. It's matched by text, so the search starts ten minutes before the deploy.
  • There's a fallback. If logger can't run on the target, the old "any newer line" check stands in, with a warning that a quiet host can fail it. A failed marker check prints the marker, so it can be searched for by hand.

It also corrects a comment that called ARRIVED_AFTER the agent host's clock. It's the machine running the script.

Verified

  • bash -n and shellcheck are clean.
  • The marker and its LogQL render as {host="golem"} |= "homelab-deploy-check golem <epoch>-<pid>-<random>". That's the same shape as the manual query that found golem's test line.
  • Not yet run end to end. The acceptance test is re-running deploy-agent.sh against golem after merge; it should now pass the Loki check.

🤖 Generated with Claude Code

The arrival check asked Loki for any line from the host newer than the deploy.
golem, an idle PBS guest, logged nothing in the three-minute wait, so a
working agent read as broken: its logs were reaching Loki by journal, syslog
and user.log, which a hand-sent logger line proved.

The script now sends a unique marker with logger(1) on the target once the
agent is up, and looks for that exact text. It travels the host's own logging
(journald, and rsyslog where it runs), so it arrives exactly when the host's
logs are shipped. It is found by text, not time, so the search starts ten
minutes early and no clock skew short of that hides it. If logger cannot run,
the old any-newer-line check stands in, with a warning that says why.

Also corrects the comment that called ARRIVED_AFTER the agent host's clock: it
is this machine's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 4, 2026 14:22
@Gerrrt Gerrrt added bug Something isn't working observability stacks/observability on prometheus labels Oct 4, 2026
@Gerrrt
Gerrrt enabled auto-merge (squash) October 4, 2026 14:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The Loki query still fails for forward clock skew, and fallback failures incorrectly claim the marker was logged.

Review effort: Balanced
Findings: 1 Medium severity · 2 Low severity

Open (3)
What changed in this PR

Adds an explicit log marker so idle hosts can prove end-to-end Loki delivery without waiting for unrelated activity.

Changes:

  • Emits a unique marker through the target’s logging system.
  • Queries Loki for that marker with a fallback to the prior freshness check.
  • Documents the deployment-check behavior and incident.
File Description
scripts/​deploy-agent.sh Adds marker generation, logging, querying, and fallback handling.
docs/​changelog.md Records the new verification approach.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/deploy-agent.sh Resolved
Comment thread docs/changelog.md Resolved
Comment thread scripts/deploy-agent.sh Resolved
Gerrrt and others added 2 commits October 4, 2026 07:40
…er only if sent

Review on #908:
- query_range's end defaulted to Loki's now, while Loki stamps the marker with
  the target's clock, so a target running fast put the marker past the
  window. The search now ends ten minutes after the wait as well as starting
  ten minutes before the deploy. The script comment and changelog say what is
  tolerated (ten minutes either way) instead of claiming skew cannot matter.
- When logger failed and the fallback query ran, the failure message still
  said the marker was logged and printed one that does not exist. It names
  the marker only when one was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit ed5236f into main Oct 4, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/deploy-agent-log-canary branch October 4, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working observability stacks/observability on prometheus

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants