You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
oracle's SmartDriveBadSectors silence expires 2026-10-08 with nothing behind it, and smaug's boot disk is next: decide what a static reallocated count means #572
Alertmanager holds two silences that end on 2026-10-08. One is HostBatteryHealthLow for oracle, and #531 is behind it. The other has no issue:
alertname=SmartDriveBadSectors host=oracle device=/dev/sda
Known finding from #351, 2026-09-07: oracle /dev/sda (ST500LT012, 5400rpm, 3138 power-on hours) has 32 reallocated sectors …
Thirty-two reallocated sectors, static since the collector first read the drive on 2026-09-07. SmartDriveBadSectorsGrowing is the live signal and is not silenced, which is the right split. On 2026-10-08 the static count pages again, with nothing to say except what the silence's comment already says — the shape #531 was filed to remove for the battery, arriving a second time for the disk.
hardware.md already promises a third: smaug's Intel DC S3520 boot disk has four reallocated sectors, and "this drive will trip SmartDriveBadSectors on the day SMART collection reaches smaug" (#483). So the question is not whether to renew one silence.
The decision
A static reallocated count is a fact about a drive, not an event. Three ways to treat it:
Renew the silence with an owner and an expiry, every thirty days, per drive. Cheapest; it is what happened once, and it is a calendar nobody keeps.
A baseline the rule reads. The rule fires on homelab_smart_reallocated_sectors > baseline where the baseline is a recorded per-drive value — a textfile series the collector emits from a checked-in table, or a recording rule. Then SmartDriveBadSectors means more than the number we wrote down, SmartDriveBadSectorsGrowing keeps meaning moving now, and a new drive with zero sectors is covered by the default of zero. The table is the place a known count lives, in git, with the date it was read.
Drop the static rule, keep only the growth rule. Loses the first-sight page on a drive nobody has looked at, which is what found oracle's 32 in the first place.
The second is the one that scales to three drives and the next NAS disk; the first is what the estate does today.
Done when
The rule set says what a known static count means, oracle's 32 and smaug's 4 are recorded wherever that is, and no SmartDriveBadSectors silence exists in Alertmanager — deleted, not expired, as #531 asks for its neighbour.
The mechanism side of "silence with an owner and a date" is #575, filed 2026-09-20: a gauge per active silence with the owning issue parsed from its comment, and rules for "expires within seven days" and "names no issue". This issue stays the decision about what a static count means; when it deletes the silence, #575's gauge is what proves it stayed deleted.
SmartDriveBadSectors went from firing (oracle, under the silence) to inactive on the first evaluation with the baseline present. SmartDriveBadSectorsGrowing is loaded unchanged and inactive.
Activity
Gerrrt commentedon Sep 20, 2026
The mechanism side of "silence with an owner and a date" is #575, filed 2026-09-20: a gauge per active silence with the owning issue parsed from its comment, and rules for "expires within seven days" and "names no issue". This issue stays the decision about what a static count means; when it deletes the silence, #575's gauge is what proves it stayed deleted.
Merge pull request #583 from Gerrrt/gerrrt/smart-reallocated-baseline…
Gerrrt commentedon Sep 20, 2026
Deployed and verified 2026-09-20, via #583.
make convergeon the monitoring host took it to 9d2cb5d and reloaded Prometheus;make smart-staterenderedsmart-baselines.promwith the two rows.homelab_smart_reallocated_sectors_baseline{host="oracle",device="/dev/sda"} 32and{host="smaug",device="/dev/sdc"} 4(smaug's row is inert until smaug has no SMART and no patch state, and the mechanism every other host uses is a push it cannot make #483 emits the series;/dev/sdcis whatnode_disk_infonames the one non-rotational disk there today, to be confirmed then).SmartDriveBadSectorswent fromfiring(oracle, under the silence) toinactiveon the first evaluation with the baseline present.SmartDriveBadSectorsGrowingis loaded unchanged and inactive.b06d032c-8150-47c1-8f46-ed7a8ed52b2edeleted (DELETE /api/v2/silence/…→ 200). Alertmanager now holds one active silence, oracle's laptop cell reads 72 % and its alert is silenced to 2026-10-08: buy and fit it, second in line after prometheus #531'sHostBatteryHealthLow, and noSmartDrive*alert is firing. Silences have no hygiene: both active ones expire 2026-10-08, one had no owner, and nothing warns before an expiry #575's gauge is what proves it stays that way.ADR-0046 records the decision;
docs/runbooks/schedule-maintenance.mdsays how to record the next count.fix(smart): the collector passes its JSON by path, renders one series…
fix(smart): the collector could not run on TrueNAS, rendered a duplic…
docs(nas): §6.4 is done, and its done block records the two things th…