Skip to content

oracle's SmartDriveBadSectors silence expires 2026-10-08 with nothing behind it, and smaug's boot disk is next: decide what a static reallocated count means #572

Closed

Description

@Gerrrt

What is true

Alertmanager holds two silences that end on 2026-10-08. One is HostBatteryHealthLow for oracle, and #531 is behind it. The other has no issue:

alertname=SmartDriveBadSectors host=oracle device=/dev/sda
Known finding from #351, 2026-09-07: oracle /dev/sda (ST500LT012, 5400rpm, 3138 power-on hours) has 32 reallocated sectors …

Thirty-two reallocated sectors, static since the collector first read the drive on 2026-09-07. SmartDriveBadSectorsGrowing is the live signal and is not silenced, which is the right split. On 2026-10-08 the static count pages again, with nothing to say except what the silence's comment already says — the shape #531 was filed to remove for the battery, arriving a second time for the disk.

hardware.md already promises a third: smaug's Intel DC S3520 boot disk has four reallocated sectors, and "this drive will trip SmartDriveBadSectors on the day SMART collection reaches smaug" (#483). So the question is not whether to renew one silence.

The decision

A static reallocated count is a fact about a drive, not an event. Three ways to treat it:

  • Renew the silence with an owner and an expiry, every thirty days, per drive. Cheapest; it is what happened once, and it is a calendar nobody keeps.
  • A baseline the rule reads. The rule fires on homelab_smart_reallocated_sectors > baseline where the baseline is a recorded per-drive value — a textfile series the collector emits from a checked-in table, or a recording rule. Then SmartDriveBadSectors means more than the number we wrote down, SmartDriveBadSectorsGrowing keeps meaning moving now, and a new drive with zero sectors is covered by the default of zero. The table is the place a known count lives, in git, with the date it was read.
  • Drop the static rule, keep only the growth rule. Loses the first-sight page on a drive nobody has looked at, which is what found oracle's 32 in the first place.

The second is the one that scales to three drives and the next NAS disk; the first is what the estate does today.

Done when

The rule set says what a known static count means, oracle's 32 and smaug's 4 are recorded wherever that is, and no SmartDriveBadSectors silence exists in Alertmanager — deleted, not expired, as #531 asks for its neighbour.

Before 2026-10-08. Refs #351, #483, #531.

Activity

  1. added this to the Nothing blocks these milestone on Sep 20, 2026
  2. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    The mechanism side of "silence with an owner and a date" is #575, filed 2026-09-20: a gauge per active silence with the owning issue parsed from its comment, and rules for "expires within seven days" and "names no issue". This issue stays the decision about what a static count means; when it deletes the silence, #575's gauge is what proves it stayed deleted.

  3. added a commit that references this issue on Sep 20, 2026
  4. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    Deployed and verified 2026-09-20, via #583.

    ADR-0046 records the decision; docs/runbooks/schedule-maintenance.md says how to record the next count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a changeobservabilitystacks/observability on prometheus

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions