Skip to content

Key the SMART baseline on something a reboot cannot move: smaug's /dev/sdX changes across boots #745

Closed

Description

@Gerrrt

What happened

On 2026-09-29 smaug's pool moved onto the chipset's AHCI ports (ADR-0052, #558). After that, the boot SSD's letter moved twice in one day:

Boot ZVTLQEZ7 ZVTBS4NL INTEL S3520 (boot)
After the disk swap sda sdb sdc
After the memory install (no disk changed) sda sdc sdb

scripts/render-smart-baselines.sh keys its rows on host + device, and SmartDriveBadSectors joins on(host, device). A row naming a letter the drive no longer has matches nothing, so the drive falls back to > 0 and pages on its four recorded reallocated sectors. That is the loud failure the script was designed to have, and it happened on 2026-09-29 at 21:06 UTC, from a textfile written under the pre-swap letters. #599's PR moves the row to /dev/sdb. That lasts until the next boot that reorders the disks.

The rows for oracle and for Saruman's iLO-labelled drives are not affected in practice. Only smaug has shown the drift.

The constraint

collect-smart-state.sh deliberately emits no serial (its header: "NO SERIAL NUMBERS … device plus model is enough to act on"). So the stable key has to be something other than the serial.

Options

  1. host + model. The S3520's model is unique on smaug. The reallocated series carries only host, device, but homelab_smart_healthy carries model, so the rule could join through it, or the collector could add model to the reallocated series. It breaks if two drives of one model ever both carry a baseline, which the Exos pair could.
  2. A by-path label (/dev/disk/by-path/pci-0000:00:17.0-ata-6). Stable while the cable stays in its port, it identifies no drive, and it survives reboots. It changes when the drive changes port, which is arguably right: that is a hardware change.
  3. Leave device as it is, and have the collector sort or relabel by something stable before emitting.

Option 2 looks closest to the serial rule's intent. Decide it here, then change the collector, the baseline script, the rule's join and host.test.yaml in one PR.

Done when

A reboot that reorders smaug's letters does not fire SmartDriveBadSectors, a test proves it, and the row no longer has to be edited by hand.

Refs #599, #572, #744 (the vdev textfile has the same letter problem and should use the same answer), ADR-0047.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    mediastacks/media on smaug (VLAN 40)observabilitystacks/observability on prometheus

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions