Skip to content

feat(switch): poll each CRS326 port's PVID and alert on a change (#1387) - #1396

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/crs326-pvid-1387
Nov 11, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/crs326-pvid-1387

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Nov 11, 2026

Copy link
Copy Markdown
Owner

What changed

The crs326 SNMP module now walks Q-BRIDGE-MIB's dot1qPvid and BRIDGE-MIB's dot1dBasePortIfIndex. Each port's untagged VLAN lands in Prometheus with the same ifDescr="etherN" label as the IF-MIB series. A new warning, SwitchPortVlanChanged, fires for an hour after any port's PVID changes, as the prompt to record the change in the port map.

  • scripts/snmp-mibs.sh fetches the seven MIBs the generator needs for those two columns. Three come from net-snmp v5.9. The other four (Q-BRIDGE, P-BRIDGE, RMON2, TOKEN-RING-RMON) are not in net-snmp, so they come from the observium commit already pinned for UPS-MIB. Each one was removed in turn and the generator then failed with "Missing MIB".
  • snmp.yaml is regenerated. Besides the crs326 additions, it picks up the v0.31.0 generator's enum-label change (feat: map enum lookup columns to string labels prometheus/snmp_exporter#1624), which feat(switch): poll the CRS326 over SNMPv3 and record it as built (#444, #84) #1303's regeneration did not include. Three iLO index labels (cpqSm2CommPort, cpqSm2NicLocation, cpqSm2NicStatsLocation) now carry the enum name instead of the number, across 49 metrics. Nothing in rules, dashboards or docs reads them. The same diff comes out of main's own generator.yaml with the current MIBs.
  • Rule counts go from 160/181 to 161/182 everywhere they are stated. A changelog entry is added.

Why

Lemmiwinks' drift check read the switch's PVIDs from the MokerLink's vlan.cgi page, and that page went with the swap (#444, Lemmiwinks#316). #482 asked for this over SNMP and was closed because the MokerLink had no Q-BRIDGE-MIB. The CRS326 has one. With the series in Prometheus, the port-map claim can come back as a query, with no switch login and no credential on oracle. The alert catches an unrecorded port move like #481.

The issue's two open questions were answered by walking the live switch over v3 on 2026-11-11 with scripts/snmp-walk.sh:

  • dot1qPvid is the bridge port's pvid under bridge VLAN filtering. It returns 24 rows, one per bridge port (ether1–24; the SFP+ cages are not bridge ports), and every value matches network.md: port 1 is 7, ether17/smaug is 40, ether20 is 30.
  • ifIndex = port + 1 carries over: dot1dBasePortIfIndex maps bridge port N to ifIndex N+1 on every row.
  • dot1qVlanStaticName is not implemented (0 rows), so it is not polled.

Refs #1387. Refs #482. Refs #481.

Blast radius

prometheus's snmp-exporter and the switch's scrape. If the new columns broke the walk, neo's scrape would fail and SnmpTargetUnreachable would say so, and SwitchInterfaceDown coverage would go with it. The walk was measured first: 1 GETBULK each, 0.55 s and 0.08 s, and every overshoot varbind (an INTEGER and an OID) decodes. The iLO label change touches only cpqSm2 comm-port and NIC series that nothing reads.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • scripts/snmp-walk.sh --device neo against the live switch, for the three OIDs above (results in Why).

  • make snmp-mibs && make snmp-generate: metrics went from 1792 to 1794, and the 7 placeholders survived. A trial generator run in scratch, with each MIB removed in turn, established the minimal set.

  • promtool test rules network.test.yaml: SUCCESS. The new case fires at 31m on a 50→99 change, stays quiet on a steady port, and clears at 95m.

  • ./scripts/lint.sh: passes. python3 scripts/check_docs.py: docs OK — 161 Prometheus + 21 Loki rules.

  • make validate: all checks passed, 5 skipped (not run): the offsite path refusal (/tmp is tmpfs here) and four unit/timer checks that only run in the deployment checkout. CI runs them.

  • Not yet done: deploy. After merge, make render from the main checkout, a force-recreate of snmp-exporter, then confirm that dot1qPvid{instance="10.7.7.2"} returns 24 series with ifDescr and that SwitchPortVlanChanged is loaded and inactive. Poll the CRS326's VLAN table: dot1qPvid now exists, and the drift check lost its port-map read with the MokerLink #1387 is then closed by hand with that proof. Lemmiwinks#316 can then query dot1qPvid instead of retiring the row.

  • make validate passes

  • Deployed to the lab and confirmed working

  • Docs updated (docs/, service README, or docs/roadmap.md)

🤖 Generated with Claude Code

Walk Q-BRIDGE-MIB dot1qPvid and BRIDGE-MIB dot1dBasePortIfIndex in the
crs326 module, chained to ifDescr so each port's untagged VLAN carries
the same port name as the IF-MIB series. Measured first on neo over v3:
24 bridge ports, bridge port N is ifIndex N+1, and the values match the
port map. dot1qVlanStaticName is not implemented, so it is not polled.

SwitchPortVlanChanged fires for an hour after any port's PVID changes.

snmp-mibs.sh fetches the seven MIBs the two columns need. The
regeneration also picks up the v0.31.0 generator's enum-label change
for three unread iLO cpqSm2 index labels.

Refs #1387. Refs #482. Refs #481.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 03:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the SNMP VLAN monitoring change, though deployment and live Prometheus verification remain pending.

No open issues 🎉

What changed in this PR

Adds SNMP-based per-port VLAN monitoring for the CRS326, enabling the port-map drift check to replace its retired MokerLink HTTP reader. Deployment and live Prometheus verification remain pending.

Changes:

  • Polls port PVIDs and bridge-to-interface mappings, with chained lookups providing ifDescr labels.
  • Adds SwitchPortVlanChanged and tests for changed, steady, and cleared states.
  • Fetches required MIBs, regenerates exporter configuration—including iLO enum-label updates—and updates documentation and rule counts.
File Description
stacks/​observability/​snmp-exporter/​snmp.yaml Adds generated bridge metrics and updates iLO enum index labels.
stacks/​observability/​snmp-exporter/​generator.yaml Adds PVID polling and chained interface-name lookups.
stacks/​observability/​README.md Updates the Prometheus alert count.
stacks/​observability/​prometheus/​tests/​network.test.yaml Tests VLAN-change alert firing, silence, and clearing.
stacks/​observability/​prometheus/​rules/​network.rules.yaml Adds the one-hour VLAN-change warning.
scripts/​snmp-mibs.sh Fetches seven required bridge MIB dependencies.
README.md Updates overall and Prometheus alert counts.
docs/​security.md Updates the total alert count.
docs/​runbooks/​deploy-stack.md Updates expected loaded-rule counts.
docs/​observability.md Documents PVID monitoring, the alert, and revised coverage counts.
docs/​changelog.md Records VLAN monitoring and regenerated iLO label changes.

Review effort: Balanced

@Gerrrt
Gerrrt enabled auto-merge (squash) November 11, 2026 03:48
@Gerrrt
Gerrrt merged commit 9754ba3 into main Nov 11, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/crs326-pvid-1387 branch November 11, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants