From 968af725c5d57ac04c93fc3ac3d81201769f1f8d Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 11 Nov 2026 03:12:36 +0000 Subject: [PATCH] feat(switch): poll each CRS326 port's PVID and alert on a change (#1387) Walk Q-BRIDGE-MIB dot1qPvid and BRIDGE-MIB dot1dBasePortIfIndex in the crs326 module, chained to ifDescr so each port's untagged VLAN carries the same port name as the IF-MIB series. Measured first on neo over v3: 24 bridge ports, bridge port N is ifIndex N+1, and the values match the port map. dot1qVlanStaticName is not implemented, so it is not polled. SwitchPortVlanChanged fires for an hour after any port's PVID changes. snmp-mibs.sh fetches the seven MIBs the two columns need. The regeneration also picks up the v0.31.0 generator's enum-label change for three unread iLO cpqSm2 index labels. Refs #1387. Refs #482. Refs #481. Co-Authored-By: Claude Opus 5.5 --- README.md | 4 +- docs/changelog.md | 26 ++++ docs/observability.md | 10 +- docs/runbooks/deploy-stack.md | 2 +- docs/security.md | 2 +- scripts/snmp-mibs.sh | 27 ++++ stacks/observability/README.md | 2 +- .../prometheus/rules/network.rules.yaml | 25 +++ .../prometheus/tests/network.test.yaml | 38 +++++ .../snmp-exporter/generator.yaml | 41 ++++- stacks/observability/snmp-exporter/snmp.yaml | 142 ++++++++++++------ 11 files changed, 253 insertions(+), 66 deletions(-) diff --git a/README.md b/README.md index 82484206..0a316865 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and - 181 alert rules — 160 metric-based in Prometheus, 21 log-based in Loki — sharing + 182 alert rules — 161 metric-based in Prometheus, 21 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -201,7 +201,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology . ├── stacks/ │ ├── observability/ # the estate's stack on prometheus — ten services -│ │ ├── prometheus/ # config, file_sd targets, 160 alert rules +│ │ ├── prometheus/ # config, file_sd targets, 161 alert rules │ │ ├── alertmanager/ # routing and inhibition │ │ ├── loki/ # single-binary config + 21 LogQL rules │ │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/changelog.md b/docs/changelog.md index 659dc5a4..db37e3c6 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -17,6 +17,32 @@ roadmap as it read that day, and the *Done* entries keep the shape they had there. `check_docs.py` does not check this file, for the reason its module docstring gives: it is a record, not a claim about now. +## 2026-11-11 + +- **[#1387](https://github.com/Gerrrt/HomeLab/issues/1387): each switch + port's VLAN is in Prometheus as `dot1qPvid`, and `SwitchPortVlanChanged` + fires for an hour after any port's PVID changes.** The drift check used + to read the MokerLink's `vlan.cgi` page for this, and that page went with + the swap ([#444](https://github.com/Gerrrt/HomeLab/issues/444)). + [#482](https://github.com/Gerrrt/HomeLab/issues/482) had asked for it over + SNMP and was closed because the MokerLink had no Q-BRIDGE-MIB. The CRS326 + has one. + - **Measured on `neo` over v3 before the module changed:** `dot1qPvid` + has 24 rows, one per bridge port, and the values match the port map: + port 1 is 7, `ether17` (`smaug`) is 40, `ether20` is 30. Bridge port N + is ifIndex N+1, so the series gets the same `ifDescr` port name as the + IF-MIB ones. `dot1qVlanStaticName` returns no rows, so VLAN names are + not polled. + - **The generator needs seven more MIBs** for those two columns, most of + them an import chain down to TOKEN-RING-RMON-MIB. + `scripts/snmp-mibs.sh` fetches the four that net-snmp lacks from the + observium commit already pinned for UPS-MIB. + - **The regeneration also carries in a change from the v0.31.0 + generator** that #1303 did not pick up (snmp_exporter #1624). Three iLO + index labels in the `cpqSm2` tables (`cpqSm2CommPort`, + `cpqSm2NicLocation` and `cpqSm2NicStatsLocation`) now hold the enum + name instead of the number. No rule or dashboard reads them. + ## 2026-11-10 - **[#1388](https://github.com/Gerrrt/HomeLab/issues/1388): Home Assistant diff --git a/docs/observability.md b/docs/observability.md index cc7d0d6d..614533f8 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -50,7 +50,7 @@ it is not closed by anything in this document. | syslog, `/var/log/*.log` | Alloy | 60s poll | Everything else. The `*.log` source skips `auth.log` and the facility files rsyslog also writes to `syslog` (`kern`, `user`, `mail`, `cron`, `daemon`), so a line is not stored once per copy ([#909](https://github.com/Gerrrt/HomeLab/issues/909)) | | pfSense | snmp-exporter | 60s | pf state table, counters, interface stats | | pfSense logs | syslog-ng → Alloy on 6514, TLS with a client certificate ([#1049](https://github.com/Gerrrt/HomeLab/issues/1049)) | stream | `filterlog` decisions, `suricata` alerts, `kea-dhcp4` leases | -| CRS326 switch | snmp-exporter, SNMPv3 | 60s | Interface status, 64-bit octet counters, RouterOS release | +| CRS326 switch | snmp-exporter, SNMPv3 | 60s | Interface status, 64-bit octet counters, each port's PVID, RouterOS release | | APC UPS | snmp-exporter | 60s | Charge, runtime, load, voltage, alarms | | ProLiant iLO | snmp-exporter | 60s | Temperature, PSU, drive and battery health | | The stack itself | Prometheus | 15s | Every component scrapes itself | @@ -352,7 +352,7 @@ separates a quiet stream from a stopped one. ## Alerting -181 rules in total: 160 metric-based in `prometheus/rules/`, and 21 log-based in +182 rules in total: 161 metric-based in `prometheus/rules/`, and 21 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -513,12 +513,12 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -160 rules across twelve files in `prometheus/rules/`: +161 rules across twelve files in `prometheus/rules/`: | File | Covers | | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and a guest tagged `backup` that no enabled backup job selects, as `GuestNotInBackupJob`, after the domain's six fell out of `golem-nightly` through purged tofu destroys (2026-10-10); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) | -| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below | +| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, a switch port whose VLAN changed (`SwitchPortVlanChanged`, from `dot1qPvid`, [#1387](https://github.com/Gerrrt/HomeLab/issues/1387)), iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | | `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. `AlloyDown` (2026-10-07) is the one alert per host for the agent itself stopping. It keys on the `-alloy` self-scrape and fires five minutes before `RemoteWriteJobStale` would, and an inhibit rule in `alertmanager.yaml` then holds back that host's per-job alerts. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose | @@ -549,7 +549,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is 160 rules of 160 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all twelve +against the broken rule too. Coverage is 161 rules of 161 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all twelve in `blackbox.rules.yaml`, all three in `dns.rules.yaml`, `GatewayFilesystemCritical`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled`, the three container-state rules from diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 93c4cdd7..bdc6113b 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -151,7 +151,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. 2. **Prometheus → Status → Rules.** Every rule loaded, none in error. The - page lists 166: the 160 alert rules this repository counts everywhere + page lists 167: the 161 alert rules this repository counts everywhere else, plus the six recording rules: `homelab_suricata_expected_interface`, `homelab_battery_runtime_seconds`, and the two directions each of diff --git a/docs/security.md b/docs/security.md index 42e66442..24a90ead 100644 --- a/docs/security.md +++ b/docs/security.md @@ -24,7 +24,7 @@ What this network is actually built to survive: | Someone on the lab segment reaching the SOC's web UIs | **Scoped by address, and accepted.** The Wazuh dashboard (`443`) and the Velociraptor GUI (`8889`) on `odin` are published by the stack's `caddy` service (container `soc-agent-msi`), not by their own containers, and its Caddyfile answers Hicks's `10.0.50.0/24` and refuses everyone else ([#1139](https://github.com/Gerrrt/HomeLab/issues/1139)). Until then both answered all of VLAN 30, the six domain guests included, because traffic between VLAN 30 hosts never crosses pfSense — so no firewall rule could have narrowed them. WireGuard peers are not in the list. The residuals: Hicks is a whole /24 with no per-host source yet ([ADR-0031](adr/0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md)), and an address on VLAN 30 can be spoofed, so this narrows who reaches the login pages and does not replace them. And the hop from Caddy to Velociraptor skips certificate verification: the GUI's certificate is signed by the CA inside the gitignored `server.config.yaml` on `odin`, and the hop is container to container on the `soc` bridge. The dashboard hop is verified against the Wazuh generator's root | | A range target with a path out | It has none — `gandalf`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 181 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 182 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop, and neither is applied yet, so `trinity` still stops uncleanly when the pack empties; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below | diff --git a/scripts/snmp-mibs.sh b/scripts/snmp-mibs.sh index 84142a5b..642e24aa 100755 --- a/scripts/snmp-mibs.sh +++ b/scripts/snmp-mibs.sh @@ -70,6 +70,9 @@ UPS_MIB_URL='https://raw.githubusercontent.com/pgmillon/observium/9064fa3ae1ae63 # already pins for UPS-MIB carries mibs/apc/PowerNet-MIB too, so this adds a # path to an existing pin and inherits its argument unchanged. POWERNET_MIB_URL='https://raw.githubusercontent.com/pgmillon/observium/9064fa3ae1ae634709198decf99aca7380693351/mibs/apc/PowerNet-MIB' +# The Q-BRIDGE set for the switch's PVID table (#1387), from the same commit +# again, for the same reason: net-snmp v5.9 does not ship these four. +OBSERVIUM_MIBS_URL='https://raw.githubusercontent.com/pgmillon/observium/9064fa3ae1ae634709198decf99aca7380693351/mibs' HPE_URL='https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1580676047/v229101/upd11.85mib.tar.gz' CURL_OPTS=(-L -sS --retry 3 --retry-delay 3 --fail --max-time 180) @@ -120,6 +123,30 @@ fetch UPS-MIB "${UPS_MIB_URL}" info "PowerNet-MIB (APC)" fetch PowerNet-MIB "${POWERNET_MIB_URL}" +# --------------------------------------------------------------------------- +# Bridge MIBs for the switch's per-port VLAN (#1387). The crs326 module walks +# two columns: Q-BRIDGE-MIB's dot1qPvid and BRIDGE-MIB's dot1dBasePortIfIndex. +# Everything else here is an import chain, and every link is load-bearing — +# each was removed in turn on 2026-11-11 and the generator failed with +# "Missing MIB": +# +# Q-BRIDGE-MIB -> P-BRIDGE-MIB (EnabledStatus), SNMP-FRAMEWORK-MIB +# (SnmpAdminString), BRIDGE-MIB, RMON2-MIB (TimeFilter) +# RMON2-MIB -> RMON-MIB, TOKEN-RING-RMON-MIB +# +# TOKEN-RING-RMON-MIB is here for one textual convention three hops away; its +# own SMIv1 imports (RFC1155-SMI, RFC-1212, RFC1271-MIB) net-snmp resolves +# without the files. net-snmp v5.9 carries the first three below and not the +# other four, which come from the observium commit already pinned above. +# --------------------------------------------------------------------------- +info "Bridge MIBs (switch PVID)" +for m in BRIDGE-MIB RMON-MIB SNMP-FRAMEWORK-MIB; do + fetch "${m}" "${NET_SNMP_URL}/${m}.txt" +done +for m in Q-BRIDGE-MIB P-BRIDGE-MIB RMON2-MIB TOKEN-RING-RMON-MIB; do + fetch "${m}" "${OBSERVIUM_MIBS_URL}/${m}" +done + # --------------------------------------------------------------------------- # pfSense exposes pf through FreeBSD's bsnmpd. BEGEMOT-PF-MIB imports from # BEGEMOT-MIB, which imports from FOKUS-MIB; miss either and the generator diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 53a6f428..40a594e1 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -53,7 +53,7 @@ prometheus/ targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/node.yaml node_exporter scrapes, for the host that runs no Alloy (smaug) targets/blackbox*.yaml probe targets, http, dns and latency — hot-reloaded, no restart - rules/*.rules.yaml 160 alert rules: host, network, ups, containers, blackbox, + rules/*.rules.yaml 161 alert rules: host, network, ups, containers, blackbox, dns, backup, ids, deploy, stack, internet and watchdog tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said diff --git a/stacks/observability/prometheus/rules/network.rules.yaml b/stacks/observability/prometheus/rules/network.rules.yaml index 63122231..7ec151df 100644 --- a/stacks/observability/prometheus/rules/network.rules.yaml +++ b/stacks/observability/prometheus/rules/network.rules.yaml @@ -100,6 +100,31 @@ groups: annotations: summary: "Interface {{ $labels.ifDescr }} on {{ $labels.device }} went down" + - alert: SwitchPortVlanChanged + # dot1qPvid is a port's untagged VLAN (#1387). Any change fires, wanted + # or not, and the alert clears an hour later on its own. Its job is the + # prompt to write the change into network.md's port map, so an + # unrecorded move like #481 is noticed the hour it happens, not at the + # next walk of the switch. Whether the switch matches the port map is + # Lemmiwinks' drift check, which reads this same series; that is not + # this rule's job. + # + # The `and` keeps the current PVID as the value; changes() alone would + # give the number of changes. A port that leaves the bridge takes its + # series with it, and this rule does not see that. + expr: | + dot1qPvid + and on (instance, dot1dBasePort) (changes(dot1qPvid[1h]) > 0) + labels: + component: network + severity: warning + category: config + annotations: + summary: "{{ $labels.ifDescr }} on {{ $labels.device }} changed its PVID; it is now VLAN {{ $value }}" + description: >- + Record the change in docs/network.md's port map, or put the port + back. Clears an hour after the change. + # No counter-wrap rule. One stood here reading rate(ifInOctets[5m]) < 0, # and it could never fire: rate() treats a decrease as a counter reset, # never as a wrap, so it is never negative (#63 again — loaded, healthy, diff --git a/stacks/observability/prometheus/tests/network.test.yaml b/stacks/observability/prometheus/tests/network.test.yaml index dc6fb01d..92d3b424 100644 --- a/stacks/observability/prometheus/tests/network.test.yaml +++ b/stacks/observability/prometheus/tests/network.test.yaml @@ -914,3 +914,41 @@ tests: category: availability exp_annotations: summary: "Interface ether15 on neo went down" + + # SwitchPortVlanChanged: ether9 moves from VLAN 50 to 99 at 30m and fires at once, + # and the alert holds for the hour changes() can see the move, then clears. + # ether17 never changes and stays quiet. The labels are what the crs326 module + # stamps: the bridge port, its ifIndex (port plus one, measured 2026-11-11) + # and the ifDescr found through it. + - interval: 1m + input_series: + - series: 'dot1qPvid{job="snmp",instance="10.7.7.2",device="neo",dot1dBasePort="9",dot1dBasePortIfIndex="10",ifDescr="ether9"}' + values: "50x30 99x100" + - series: 'dot1qPvid{job="snmp",instance="10.7.7.2",device="neo",dot1dBasePort="17",dot1dBasePortIfIndex="18",ifDescr="ether17"}' + values: "40x130" + alert_rule_test: + - eval_time: 29m + alertname: SwitchPortVlanChanged + exp_alerts: [] + - eval_time: 31m + alertname: SwitchPortVlanChanged + exp_alerts: + - exp_labels: + alertname: SwitchPortVlanChanged + job: snmp + instance: 10.7.7.2 + device: neo + dot1dBasePort: 9 + dot1dBasePortIfIndex: 10 + ifDescr: ether9 + component: network + severity: warning + category: config + exp_annotations: + summary: "ether9 on neo changed its PVID; it is now VLAN 99" + description: >- + Record the change in docs/network.md's port map, or put the + port back. Clears an hour after the change. + - eval_time: 95m + alertname: SwitchPortVlanChanged + exp_alerts: [] diff --git a/stacks/observability/snmp-exporter/generator.yaml b/stacks/observability/snmp-exporter/generator.yaml index a506a515..2afd8da5 100644 --- a/stacks/observability/snmp-exporter/generator.yaml +++ b/stacks/observability/snmp-exporter/generator.yaml @@ -301,7 +301,8 @@ modules: values: ["^/"] # MikroTik CRS326-24G-2S+RM on RouterOS 7 (neo, 10.7.7.2), in service since - # 2026-11-05 (#444, ADR-0041) — IF-MIB, five columns plus sysDescr. + # 2026-11-05 (#444, ADR-0041) — IF-MIB, five columns plus sysDescr, and two + # bridge columns for each port's VLAN (#1387, below). # # It replaced a MokerLink whose management CPU wedged under exporter polling, # bricked once under a full walk, and returned a zero-length ifSpecific that @@ -332,17 +333,43 @@ modules: # sysDescr now carries the model and the RouterOS release, where the # MokerLink's said only "Switch". scripts/check_versions.py selects the # pfSense claim by device name, so it is unaffected. + # + # dot1qPvid is each port's untagged VLAN, the port map's claim in one series + # (#1387). It replaces the MokerLink's vlan.cgi page, which the drift check + # scraped over HTTP until the swap (Lemmiwinks#316); #482 asked for this and + # was closed because the MokerLink had no Q-BRIDGE-MIB. SwitchPortVlanChanged + # reads it. Measured 2026-11-11 with scripts/snmp-walk.sh over v3: + # + # dot1qPvid 24 rows, indexed by bridge port 1–24 (ether1–24; + # the SFP+ cages are not bridge ports). Under bridge + # VLAN filtering it is the bridge port's pvid: port 1 + # reads 7, ether17 (smaug) 40, ether20 30, as + # network.md has them. 1 request, 0.55s, one + # overshoot into dot1qPortAcceptableFrameTypes. + # dot1dBasePortIfIndex bridge port N is ifIndex N+1 on every row, the same + # port-plus-one as IF-MIB. One overshoot, an OID. + # dot1qVlanStaticName not implemented: 0 rows. So there are no VLAN names. + # + # The bridge port is not an ifIndex, so the two lookups chain: bridge port to + # ifIndex, then ifIndex to ifDescr, and dot1qPvid carries the same port name + # as every IF-MIB series. scripts/snmp-mibs.sh says why it takes seven MIBs. crs326: walk: - - 1.3.6.1.2.1.2.2.1.1 # ifIndex - - 1.3.6.1.2.1.2.2.1.2 # ifDescr (metric, and the lookup below) - - 1.3.6.1.2.1.2.2.1.8 # ifOperStatus (SwitchInterfaceDown) - - 1.3.6.1.2.1.31.1.1.1.6 # ifHCInOctets (64-bit) - - 1.3.6.1.2.1.31.1.1.1.10 # ifHCOutOctets (64-bit) - - 1.3.6.1.2.1.1.1 # sysDescr (model and RouterOS release) + - 1.3.6.1.2.1.2.2.1.1 # ifIndex + - 1.3.6.1.2.1.2.2.1.2 # ifDescr (metric, and the lookups below) + - 1.3.6.1.2.1.2.2.1.8 # ifOperStatus (SwitchInterfaceDown) + - 1.3.6.1.2.1.31.1.1.1.6 # ifHCInOctets (64-bit) + - 1.3.6.1.2.1.31.1.1.1.10 # ifHCOutOctets (64-bit) + - 1.3.6.1.2.1.1.1 # sysDescr (model and RouterOS release) + - 1.3.6.1.2.1.17.7.1.4.5.1.1 # dot1qPvid (SwitchPortVlanChanged) + - 1.3.6.1.2.1.17.1.4.1.2 # dot1dBasePortIfIndex (the lookup below) lookups: - source_indexes: [ifIndex] lookup: ifDescr + - source_indexes: [dot1dBasePort] + lookup: dot1dBasePortIfIndex + - source_indexes: [dot1dBasePortIfIndex] + lookup: ifDescr # HPE ProLiant DL360 Gen9 iLO (shiva, 10.0.30.10) — SNMPv2-MIB system group # plus the Compaq/HP Insight enterprise tree for hardware health. diff --git a/stacks/observability/snmp-exporter/snmp.yaml b/stacks/observability/snmp-exporter/snmp.yaml index cdeef6d0..a40357d6 100644 --- a/stacks/observability/snmp-exporter/snmp.yaml +++ b/stacks/observability/snmp-exporter/snmp.yaml @@ -464,6 +464,8 @@ modules: 7: sunday crs326: walk: + - 1.3.6.1.2.1.17.1.4.1.2 + - 1.3.6.1.2.1.17.7.1.4.5.1.1 - 1.3.6.1.2.1.2.2.1.1 - 1.3.6.1.2.1.2.2.1.2 - 1.3.6.1.2.1.2.2.1.8 @@ -476,6 +478,48 @@ modules: oid: 1.3.6.1.2.1.1.1 type: DisplayString help: A textual description of the entity - 1.3.6.1.2.1.1.1 + - name: dot1dBasePortIfIndex + oid: 1.3.6.1.2.1.17.1.4.1.2 + type: gauge + help: The value of the instance of the ifIndex object, defined in IF-MIB, for + the interface corresponding to this port. - 1.3.6.1.2.1.17.1.4.1.2 + indexes: + - labelname: dot1dBasePort + type: gauge + - labelname: dot1dBasePortIfIndex + type: gauge + lookups: + - labels: + - dot1dBasePort + labelname: dot1dBasePortIfIndex + oid: 1.3.6.1.2.1.17.1.4.1.2 + type: gauge + - labels: + - dot1dBasePortIfIndex + labelname: ifDescr + oid: 1.3.6.1.2.1.2.2.1.2 + type: DisplayString + - name: dot1qPvid + oid: 1.3.6.1.2.1.17.7.1.4.5.1.1 + type: gauge + help: The PVID, the VLAN-ID assigned to untagged frames or Priority-Tagged frames + received on this port - 1.3.6.1.2.1.17.7.1.4.5.1.1 + indexes: + - labelname: dot1dBasePort + type: gauge + - labelname: dot1dBasePortIfIndex + type: gauge + lookups: + - labels: + - dot1dBasePort + labelname: dot1dBasePortIfIndex + oid: 1.3.6.1.2.1.17.1.4.1.2 + type: gauge + - labels: + - dot1dBasePortIfIndex + labelname: ifDescr + oid: 1.3.6.1.2.1.2.2.1.2 + type: DisplayString - name: ifIndex oid: 1.3.6.1.2.1.2.2.1.1 type: gauge @@ -15114,7 +15158,7 @@ modules: help: Remote Insight Communications Port - 1.3.6.1.4.1.232.9.2.4.1.1.1 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15127,7 +15171,7 @@ modules: help: Remote Insight Communications Type - 1.3.6.1.4.1.232.9.2.4.1.1.2 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15144,7 +15188,7 @@ modules: Insight. - 1.3.6.1.4.1.232.9.2.4.1.1.3 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15154,7 +15198,7 @@ modules: help: Remote Insight Communications Parity - 1.3.6.1.4.1.232.9.2.4.1.1.4 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15169,7 +15213,7 @@ modules: help: Remote Insight Communications Data Bits - 1.3.6.1.4.1.232.9.2.4.1.1.5 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15183,7 +15227,7 @@ modules: help: Remote Insight Communications Stop Bits - 1.3.6.1.4.1.232.9.2.4.1.1.6 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15197,7 +15241,7 @@ modules: help: The string that is sent to reset the modem. - 1.3.6.1.4.1.232.9.2.4.1.1.7 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15207,7 +15251,7 @@ modules: help: The string that is sent to initialize the modem. - 1.3.6.1.4.1.232.9.2.4.1.1.8 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15217,7 +15261,7 @@ modules: help: The string that is prepended to phone numbers before dialing. - 1.3.6.1.4.1.232.9.2.4.1.1.9 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15228,7 +15272,7 @@ modules: - 1.3.6.1.4.1.232.9.2.4.1.1.10 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15238,7 +15282,7 @@ modules: help: Comm Dial-in - 1.3.6.1.4.1.232.9.2.4.1.1.11 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15252,7 +15296,7 @@ modules: help: Remote Insight Dialback Required - 1.3.6.1.4.1.232.9.2.4.1.1.12 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15266,7 +15310,7 @@ modules: help: Non-PPP connections - 1.3.6.1.4.1.232.9.2.4.1.1.13 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15280,7 +15324,7 @@ modules: help: SNMP Trap Delivery - 1.3.6.1.4.1.232.9.2.4.1.1.14 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15294,7 +15338,7 @@ modules: help: Page Delivery - 1.3.6.1.4.1.232.9.2.4.1.1.15 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15309,7 +15353,7 @@ modules: the Remote Insight. - 1.3.6.1.4.1.232.9.2.4.1.1.16 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15319,7 +15363,7 @@ modules: help: Remote Insight Communications Pager Parity - 1.3.6.1.4.1.232.9.2.4.1.1.17 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15334,7 +15378,7 @@ modules: help: Remote Insight Communications Pager Data Bits - 1.3.6.1.4.1.232.9.2.4.1.1.18 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15348,7 +15392,7 @@ modules: help: Remote Insight Communications Pager Stop Bits - 1.3.6.1.4.1.232.9.2.4.1.1.19 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15362,7 +15406,7 @@ modules: help: Remote Insight Network PCMCIA Card Model - 1.3.6.1.4.1.232.9.2.4.1.1.20 indexes: - labelname: cpqSm2CommPort - type: gauge + type: EnumAsInfo enum_values: 1: internal 2: auxiliary @@ -15373,7 +15417,7 @@ modules: Insight/ Integrated Lights-Out - 1.3.6.1.4.1.232.9.2.5.1.1.1 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15389,7 +15433,7 @@ modules: - 1.3.6.1.4.1.232.9.2.5.1.1.2 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15401,7 +15445,7 @@ modules: - 1.3.6.1.4.1.232.9.2.5.1.1.3 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15417,7 +15461,7 @@ modules: Remote Insight/ Integrated Lights-Out - 1.3.6.1.4.1.232.9.2.5.1.1.4 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15429,7 +15473,7 @@ modules: Address - 1.3.6.1.4.1.232.9.2.5.1.1.5 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15441,7 +15485,7 @@ modules: Mask. - 1.3.6.1.4.1.232.9.2.5.1.1.6 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15455,7 +15499,7 @@ modules: This network interface controller is enabled - 1.3.6.1.4.1.232.9.2.5.1.1.7' indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15471,7 +15515,7 @@ modules: Status - 1.3.6.1.4.1.232.9.2.5.1.1.8 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15488,7 +15532,7 @@ modules: - 1.3.6.1.4.1.232.9.2.5.1.1.9 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15502,7 +15546,7 @@ modules: 1.3.6.1.4.1.232.9.2.5.1.1.10' indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15518,7 +15562,7 @@ modules: - 1.3.6.1.4.1.232.9.2.5.1.1.11 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15535,7 +15579,7 @@ modules: Unit - 1.3.6.1.4.1.232.9.2.5.1.1.12 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15547,7 +15591,7 @@ modules: Gateway IP Address - 1.3.6.1.4.1.232.9.2.5.1.1.13 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15558,7 +15602,7 @@ modules: help: Remote Insight/ Integrated Lights-Out Fully Qualified DNS Name - 1.3.6.1.4.1.232.9.2.5.1.1.14 indexes: - labelname: cpqSm2NicLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15570,7 +15614,7 @@ modules: Insight/ Integrated Lights-Out - 1.3.6.1.4.1.232.9.2.5.2.1.1 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15586,7 +15630,7 @@ modules: Transmitted. - 1.3.6.1.4.1.232.9.2.5.2.1.2 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15598,7 +15642,7 @@ modules: Transmit Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.3 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15610,7 +15654,7 @@ modules: Transmit Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.4 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15622,7 +15666,7 @@ modules: Transmit Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.5 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15634,7 +15678,7 @@ modules: Transmit Discarded Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.6 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15646,7 +15690,7 @@ modules: Error Transmit Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.7 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15658,7 +15702,7 @@ modules: of Outstanding Packets in Transmit Queue. - 1.3.6.1.4.1.232.9.2.5.2.1.8 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15670,7 +15714,7 @@ modules: Received. - 1.3.6.1.4.1.232.9.2.5.2.1.9 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15682,7 +15726,7 @@ modules: Receive Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.10 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15694,7 +15738,7 @@ modules: Receive Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.11 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15706,7 +15750,7 @@ modules: Receive Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.12 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15718,7 +15762,7 @@ modules: Receive Discarded Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.13 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15730,7 +15774,7 @@ modules: Receive Error Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.14 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded @@ -15742,7 +15786,7 @@ modules: of Unknown Protocol Packets. - 1.3.6.1.4.1.232.9.2.5.2.1.15 indexes: - labelname: cpqSm2NicStatsLocation - type: gauge + type: EnumAsInfo enum_values: 1: other 2: embedded