You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Saruman's thin pools are invisible to HostDiskCritical: a full Large_data or pve/data makes every guest read-only with no alert #538
Saruman now has two LVM-thin pools: pve/data on the HDD mirror, holding alexander, and Large_data/Large_data on the SSD logical drive made on 2026-09-19 (#527). A thin pool is not a filesystem, so node_filesystem_* never sees either one — the host reports only /, /boot/efi and /etc/pve. HostDiskCritical and HostDiskWillFillIn24h in stacks/observability/prometheus/rules/host.rules.yaml therefore cannot fire for a pool, and a thin pool that fills makes every guest on it read-only, at once, with no warning from anything in this estate.
docs/runbooks/fit-the-saruman-ssds.md §7 records the gap and says it "deserves its own issue, not a step here". This is that issue. It is the same shape as #351: a reading the host has and the stack never asked for.
What closes it
A textfile collector beside scripts/collect-guest-state.sh — same unit and timer shape, same --print / --self-test contract — that emits, per pool, lvs --noheadings -o vg_name,lv_name,data_percent,metadata_percent,lv_size --select 'lv_attr=~^t' as gauges: homelab_thin_pool_data_percent{vg,pool}, homelab_thin_pool_metadata_percent{vg,pool}, and the pool size. lvs needs root, which the agent collectors already run as.
Two rules with unit tests: one on data or metadata use above a threshold for a sustained window (warning), one on predict_linear of data use reaching 100 % inside 24 h (critical, since the consequence is every guest on the pool going read-only). Metadata full is worse than data full and the threshold should be lower.
A staleness check in the shape of GuestStateStopped, or the existing one extended, so a collector that stops does not look like a pool that is fine.
A line in docs/observability.md naming why disk alerts on Saruman come in two kinds — filesystems from node_exporter, pools from the collector — and a row in the runbook's verdict table.
Done when
Both pools have data_percent and metadata_percent series here, the two rules pass make check-rules with fixtures for both pools, and the runbook's §7 blind-spot paragraph points at the collector rather than at this issue.
Done-when met on 2026-09-23. The collector from #639 is installed on Saruman, and Prometheus now has both pools:
pool
data_percent
metadata_percent
pve/data
1.73
0.29
large_data/large_data
7.9
0.42
node_textfile_mtime_seconds{file=".../thin-pool-state.prom", instance="Saruman"} is current. No ThinPool* alert is firing, which is correct at these levels. The rules and their unit tests merged with #639, and §7 of fit-the-saruman-ssds.md points at the collector. Closing.
Activity
Merge origin/main (#637) into the #538 branch
Gerrrt commentedon Sep 23, 2026
Done-when met on 2026-09-23. The collector from #639 is installed on
Saruman, and Prometheus now has both pools:data_percentmetadata_percentpve/datalarge_data/large_datanode_textfile_mtime_seconds{file=".../thin-pool-state.prom", instance="Saruman"}is current. NoThinPool*alert is firing, which is correct at these levels. The rules and their unit tests merged with #639, and §7 offit-the-saruman-ssds.mdpoints at the collector. Closing.Generated by Claude Code