Repository navigation
fix(deploy-agent): prove the log path with a marker line, not by waiting - #908
Merged
Merged
Conversation
The arrival check asked Loki for any line from the host newer than the deploy. golem, an idle PBS guest, logged nothing in the three-minute wait, so a working agent read as broken: its logs were reaching Loki by journal, syslog and user.log, which a hand-sent logger line proved. The script now sends a unique marker with logger(1) on the target once the agent is up, and looks for that exact text. It travels the host's own logging (journald, and rsyslog where it runs), so it arrives exactly when the host's logs are shipped. It is found by text, not time, so the search starts ten minutes early and no clock skew short of that hides it. If logger cannot run, the old any-newer-line check stands in, with a warning that says why. Also corrects the comment that called ARRIVED_AFTER the agent host's clock: it is this machine's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt
enabled auto-merge (squash)
October 4, 2026 14:22
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The Loki query still fails for forward clock skew, and fallback failures incorrectly claim the marker was logged.
Review effort: Balanced
Findings: 1
Open (3)
What changed in this PR
Adds an explicit log marker so idle hosts can prove end-to-end Loki delivery without waiting for unrelated activity.
Changes:
- Emits a unique marker through the target’s logging system.
- Queries Loki for that marker with a fallback to the prior freshness check.
- Documents the deployment-check behavior and incident.
| File | Description |
|---|---|
scripts/deploy-agent.sh |
Adds marker generation, logging, querying, and fallback handling. |
docs/changelog.md |
Records the new verification approach. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…er only if sent Review on #908: - query_range's end defaulted to Loki's now, while Loki stamps the marker with the target's clock, so a target running fast put the marker past the window. The search now ends ten minutes after the wait as well as starting ten minutes before the deploy. The script comment and changelog say what is tolerated (ten minutes either way) instead of claiming skew cannot matter. - When logger failed and the fallback query ran, the failure message still said the marker was logged and printed one that does not exist. It names the marker only when one was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


The gap
deploy-agent.sh's arrival check asked Loki for any line from the host newer than the deploy. On 2026-10-04, golem, an idle Proxmox Backup Server guest, logged nothing during the three-minute wait. The script reportedLoki has no line newer than the deployand exited non-zero.The agent was fine:
204, and nothing dropped.loggerline reached Loki within seconds, by all three paths golem ships: the journal,/var/log/syslogand/var/log/user.log.The fix
Once the agent passes its checks, the script sends a unique marker on the target:
It then looks for exactly that text:
{host="<host>"} |= "<marker>".loggercan't run on the target, the old "any newer line" check stands in, with a warning that a quiet host can fail it. A failed marker check prints the marker, so it can be searched for by hand.It also corrects a comment that called
ARRIVED_AFTERthe agent host's clock. It's the machine running the script.Verified
bash -nand shellcheck are clean.{host="golem"} |= "homelab-deploy-check golem <epoch>-<pid>-<random>". That's the same shape as the manual query that found golem's test line.deploy-agent.shagainst golem after merge; it should now pass the Loki check.🤖 Generated with Claude Code