Repository navigation
fix(protocol): never sweep or evict this node's own node-table row - #65
Merged
Merged
Conversation
The 2-minute staleness sweep retires any connected, service-less row whose
last_seen is older than peerDisconnectAfter (2 min) and deletes it after
peerStaleAfter (10 min). The 30-second liveness tick deliberately skips
self, and a service-less head never runs ReannounceLocalServices (it only
fires on service events), so the head's own row — written once at startup
by MarkSelfAsBootstrap — keeps its last_seen frozen at boot time. Every
head therefore disconnected itself from its own table ~2 minutes after
starting and deleted the row ~12 minutes in, silently emptying
/v1/dnt/bootstraps until the next restart re-registered it and repeated
the cycle.
Skip the local node's own row in both maintenance passes:
- clock.go 2-min sweep: always keep our own row regardless of staleness
- probeUnreachableServicePeers: never probe/evict ourselves; a relay
reservation hiccup must not retire this node from its own table
Remote peers still refresh our row on their side via their liveness pings,
so no real staleness exemption is lost by this change.
Observed in production on the ocf-1 head (v0.2.6, 2026-10-10): the row
registered at startup was swept within minutes and /v1/dnt/bootstraps
returned {"bootstraps":[]} from then on. With this change the row survives
past the full disconnect+delete window and the list stays populated.
Test Coverage Report 📊Click to view detailed coverageSummary: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/v1/dnt/bootstrapskept going empty in production within minutes of every head-node restart, despiteMarkSelfAsBootstrapcorrectly re-registering the head's row (withpublic_address) at startup.Root cause
Three behaviors compose into a self-deleting row on every service-less head:
ReannounceLocalServicesonly fires on service/reconnect events — so a head's own row keeps thelast_seenit got at startup, forever.decideSweepActionwithprobeUnknown) marks any connected, service-less row whoselast_seenis older thanpeerDisconnectAfter(2 min) as disconnected, then deletes it afterpeerStaleAfter(10 min).MarkSelfAsBootstrapruns once, fromP2PListener).Net effect: every head disconnects itself from its own table ~2 min after boot, deletes its row ~12 min in, and
/v1/dnt/bootstraps(which only advertises rows carrying apublic_address) returns{"bootstraps":[]}until the next restart — which repeats the cycle.Note
POST /v1/dnt/_nodecannot rescue it: the CRDTPutHookdeliberately ignores the node's own key, so that path never reaches the in-memory table.Fix
Skip the local node's own row in both maintenance passes:
clock.go— the 2-min sweep always keeps our own row (our own liveness is not in question; remote peers refresh our row on their side via their liveness pings).liveness.go—probeUnreachableServicePeersnever probes/evicts ourselves: a relay-reservation hiccup must not retire this node from its own table.Evidence (production, ocf-1 head @ 140.238.223.116)
Registering as bootstrap node(row written with public address) → row absent from/v1/dnt/tableand bootstraps empty shortly after.3b48011-sweepfix): head row alive and connected >15 min after restart (past the full kill window),NRestarts=0, and/v1/dnt/bootstrapsstays populated: https://bootstraps.opentela.ai/v1/dnt/bootstrapsTesting
go vet ./internal/protocol/clean;go test ./internal/protocol/passes.