Skip to content

Move the iLO and the UPS card to SNMPv3 authPriv (ADR-0036) #85

Closed

Description

@Gerrrt

As decided by ADR-0036, which the body below predates

Both premises as filed are wrong, and were wrong on the day they were written. The body says pfSense, the APC NMC and the iLO support SNMPv3 authPriv and "the MokerLink switch does not, which is the blocker." ADR-0036 measured the opposite on both counts: morpheus is the device that cannot — bsnmpd is the only daemon that serves the pf MIB and pfSense writes no USM user for it — and neo answers v3 on the wire and was never the blocker. v3 is decided per poll, by where the poll travels: the iLO and the UPS card go to authPriv; the firewall stays on v2c over bsnmpd; the MokerLink stays on v2c until #444 replaces it, and #84 rides along.

What is done

The repository half, in #431: the generator's auth blocks exist for each device, the secrets stanzas are shaped, rotate-snmp-community.md §4 is the device-side procedure. Nothing device-side has been run — all four auth blocks in stacks/observability/snmp-exporter/generator.yaml still read version: 2 (auth_apc, auth_ilo, auth_mokerlink, auth_pfsense).

What remains

  • rotate-snmp-community.md §4 against shiva (the iLO) — physical access; the iLO can drop the session on save
  • the same against mjolnir (the UPS card)
  • auth_ilo and auth_apc to version: 3 in the generator, secrets rendered, snmp-verify.sh clean
  • docs/roadmap.md's Security section stops bundling this with the switch

docs/roadmap.md lists this under Decided but not built, which is right.


Body as filed, superseded 2026-09-19

SNMPv2c sends the community in cleartext, so anyone with a port on the
management VLAN can read it off a single packet. Per-device communities limit
what one captured packet grants; they do not make the transport private.

pfSense, the APC NMC and iLO all support SNMPv3 authPriv. The MokerLink switch
does not
, which is the blocker for doing it uniformly — and a fleet where
three devices are encrypted and one is not still leaks the fourth.

scripts/snmp-verify.sh:20 notes the same constraint.

Worth deciding: move the three that can and accept a mixed estate, or treat this
as blocked until the switch is replaced.

Tracked in docs/roadmap.md; filed as an issue so it has a place to be discussed and closed.

Activity

  1. added
    securityA credential, an exposure, or a control that is weaker than recorded
    on Aug 21, 2026
  2. Gerrrt commented on Sep 9, 2026

    @Gerrrt
    OwnerAuthor

    Decided in ADR-0035, PR #431. Neither of the two framings in the issue survived a check against the devices on 2026-09-09:

    • "Three can" is two, and the one that cannot is the firewall, not the switch. morpheus answers from bsnmpd; pfSense's GUI writes no USM user for it, and bsnmpd is the only daemon that serves the pf MIB. The Net-SNMP package (would do SHA/AES v3) is installed on the box, disabled, and has no pf module — using it would trade PfNotRunning, PfStateTableNearLimit and PfMemoryDropsIncreasing for encryption of the credential that reads those tables. Not worth it; morpheus stays v2c.
    • The switch's agent speaks SNMPv3 on the wire. A discovery packet to 10.7.7.2 gets a USM Report ("Unknown user name"), exactly like the other three. Whether the web UI has a page to create a user is unchecked — its login is the only way to look. So ADR-0018's "cannot do SNMPv3 at all" is narrowed to "may have no way to create a user". Worth one look at the next login; recorded in hardware.md either way.
    • The iLO and the AP9641 both do authPriv with SHA and AES, and both can switch v1 off separately (iLO 4 2.82: three v3 users, SNMPv1 Request; NMC3 AOS 2.0.0.6: four profiles, SNMPv1 access as its own enable).
    • The poll that matters is the iLO's. It is delivered into VLAN 30, layer-2 adjacent to the attack VM ADR-0014 puts there on purpose — a guest that ARP-spoofs 10.0.30.10 reads the community every minute. The other three polls ride on Winterfell, where a sniffer already sits on the segment with the firewall's UI and the monitoring host.

    So: per poll, mixed on purpose. shiva first, then mjolnir. authPriv only — snmp-targets.sh --check refuses anything else.

    The repository half is in #431: a device's version and key names now come from its auth block in generator.yaml, snmp-verify.sh/snmp-walk.sh speak v3 through the same snmp.conf route, and USM rejections are reported as rejections rather than timeouts. Tested against the live iLO with a user that does not exist (net-snmp: Unknown user name; the pinned exporter: incoming packet is not authentic, discarding). All four devices are still v2c in that PR — device first, then repository.

    What is left is the device side, in rotate-snmp-community.md §4: create the user on the iLO (physical access, it can drop the session on save), flip auth_ilo, two sops keys in place of one, make render && make reload, verify, switch SNMPv1 Request off, prove it with --old. Then the card. Please paste the snmp-verify output here after each one, including the --old line, and the scrape_duration_seconds before and after — ADR-0035 says the v3 cost is measured, not assumed.

  3. added 2 commits that reference this issue on Sep 10, 2026
  4. changed the title [-]Move to SNMPv3 authPriv where the hardware supports it[/-] [+]Move the iLO and the UPS card to SNMPv3 authPriv (ADR-0036)[/+] on Sep 19, 2026
  5. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    shiva is polled over SNMPv3 authPriv since 2026-09-20 22:28 UTC (#581, runbook §4.1–§4.4). The device went first: the prometheus user (SHA, AES) was created on the iLO with SNMPv1 Request still enabled, then auth_ilo flipped in the repository and the stack rendered and reloaded from the main checkout.

    snmp-verify.sh --device shiva from the branch, before the exporter changed — the PASS went over v3, --dry-run showing shiva … v3 SNMP_AUTHPASS_ILO SNMP_PRIVPASS_ILO:

    Current community
      PASS shiva      10.0.30.10   "Integrated Lights-Out 4 2.82 Feb 06 2023"
    Unauthenticated PDUs (junk community must be refused)
      PASS shiva      10.0.30.10   refuses a junk community over GET and GETBULK, short and long
    Stock communities (must be refused)
      PASS shiva      10.0.30.10   refuses public private
    all SNMP targets verified
    

    After make render && make reload, through the running exporter with the rendered v3 block: HTTP 200, 3866 cpq* series, sysDescr as above, and the exporter's own log clean of incoming packet is not authentic since the reload. up{job="snmp",instance="10.0.30.10"} was 1 on the first sample after the reload.

    Scrape cost, per ADR-0036 ("measured, not assumed") — the v3 discovery round trip is in the noise on a 12-second walk:

    scrape_duration_seconds on shiva
    v2c, 24 h average before the move 12.05 s
    v3, exporter-measured first walk 12.53 s (781 packets, 3872 PDUs)
    v3, first Prometheus sample 14.60 s

    A one-hour average after the move follows once there is one.

    One thing the dry run in #431 could not see, now fixed in #581: the generator writes its stock default into a v3 block that has no community: line, and gitleaks' community rule correctly fails validate on the generated snmp.yaml. A v3 block now carries an explicit empty string, snmp-targets.sh --check requires it, and the three documented copies of the shape show it.

    Still open on this device — §4.5, by hand: SNMPv1 Request off on the iLO, then ./scripts/snmp-verify.sh --old with the community SNMP_COMMUNITY_ILO held, wanting rejected; then a second --device shiva verify. The docs claim the v3 poll and nothing about v1 until that output is here. mjolnir is next, same procedure.

  6. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    §4.5 done on shiva: SNMPv1 Request is off, and the iLO refuses its own former community. snmp-verify.sh --old returned rejected for the device — the check that matters, because it probes over v2c whatever the device speaks now. The move is complete on this device.

    Confirming verify afterwards, since switching v1 off is another save of the iLO's SNMP page:

    Current community
      PASS shiva      10.0.30.10   "Integrated Lights-Out 4 2.82 Feb 06 2023"
    Unauthenticated PDUs (junk community must be refused)
      PASS shiva      10.0.30.10   refuses a junk community over GET and GETBULK, short and long
    Stock communities (must be refused)
      PASS shiva      10.0.30.10   refuses public private
    all SNMP targets verified
    

    The v3 scrape cost, now measured over a full hour rather than from one walk, which is what ADR-0036 asked for:

    scrape_duration_seconds on shiva
    v2c, 24 h average before the move 12.05 s
    v3, 1 h average after it 12.09 s

    60 samples in that hour, no gaps, and min_over_time(up{job="snmp"}[30m]) is 1 for all four devices — the discovery round trip per scrape session costs nothing measurable against a 12-second walk.

    So on this device the whole of §4 is done: the user exists, the exporter polls it over authPriv, v1 is off and proved off. The documents now say so rather than pointing here — SECURITY.md, docs/security.md, docs/hardware.md and the roadmap bullet.

    mjolnir is what is left, and it closes this issue. Same four steps, device first: two passphrases, a user profile on the NMC with SHA and AES and an access-control entry for 10.0.99.20, then auth_apc to the v3 block, then §4.5 on the card's Configuration → Network → SNMPv1 → Access.

  7. added 2 commits that reference this issue on Sep 21, 2026
  8. Gerrrt commented on Sep 21, 2026

    @Gerrrt
    OwnerAuthor

    mjolnir is polled over SNMPv3 authPriv since 2026-09-21 (#592). The profile was created on the NMC on 2026-09-20 beside the iLO's — prometheus, SHA, AES, an access-control entry for 10.0.99.20 and nothing else — and the repository half landed today.

    snmp-verify.sh --device mjolnir from the branch, over v3, before the exporter switched:

    Current community
      PASS mjolnir    10.0.99.10   "APC Web/SNMP Management Card (MB:v4.2.9 PF:v2.0.0.6 PN:apc_
    Unauthenticated PDUs (junk community must be refused)
      PASS mjolnir    10.0.99.10   refuses a junk community over GET and GETBULK, short and long
    Stock communities (must be refused)
      PASS mjolnir    10.0.99.10   refuses public private
    all SNMP targets verified
    

    After make render && make reload, through the running exporter with the rendered v3 block: HTTP 200, 58 series, 5 packets, 0.03 s, and the values that matter still reading — upsAdvTestDiagnosticSchedule 8 (biweekly since last test), upsTestResultsSummary 1 (donePass), upsEstimatedMinutesRemaining 47. No incoming packet is not authentic in the log. up{job="snmp"} is 1 for all four devices.

    The v3 cost here is nothing: 0.03 s against the 0.03 s v2c 24-hour average. The card's walk is 58 varbinds, so the one discovery round trip is the whole difference and it does not show at this resolution.

    What moving it found. shut-down-on-the-ups.md was written while the card was still on v2c and named the v3 keys SNMP_AUTHPASS_MJOLNIR / SNMP_PRIVPASS_MJOLNIR. Those names have never existed — every tool derives them from the auth label, auth_apc, not from the device — so anyone building the NUT server would have searched SOPS for a key that is not there. Corrected, along with the conditional that has now resolved. And §4.5 gained the check ADR-0049 asked it for: anything else holding the device's community moves before SNMPv1 goes off, because a consumer left on v2c is what keeps v2c enabled and it fails silently, the exporter staying green throughout. That consumer does not exist yet here — the NUT package is installed and unconfigured — so there was nothing to move.

    What is left, and it is the last thing on this issue: the card's Configuration → Network → SNMPv1 → Access → disabled, then ./scripts/snmp-verify.sh --old with the community SNMP_COMMUNITY_APC held, wanting rejected, then a second --device mjolnir.

  9. added a commit that references this issue on Sep 21, 2026
  10. Gerrrt commented on Sep 21, 2026

    @Gerrrt
    OwnerAuthor

    Closed. Both devices ADR-0036 names are on SNMPv3 authPriv, and neither answers a community.

    mjolnir's SNMPv1 access went off in the card's web UI, and snmp-verify.sh --old reported its former community rejected — the check that proves it, because that probe is v2c whatever the device speaks now. The confirming verify afterwards, since switching v1 off is another save of that page:

    Current community
      PASS mjolnir    10.0.99.10   "APC Web/SNMP Management Card (MB:v4.2.9 PF:v2.0.0.6 PN:apc_
    Unauthenticated PDUs (junk community must be refused)
      PASS mjolnir    10.0.99.10   refuses a junk community over GET and GETBULK, short and long
    Stock communities (must be refused)
      PASS mjolnir    10.0.99.10   refuses public private
    all SNMP targets verified
    

    The cost on the card, measured rather than assumed. The walk is 58 varbinds, so one engine-discovery round trip is the whole difference:

    scrape_duration_seconds on mjolnir
    v2c, 24 h before 0.031 s
    v3, the 10 min after the switchover 0.038 s

    Two transients, named rather than averaged away: the first v3 scrape took 0.188 s, and one scrape took 10.0 s at the moment SNMPv1 was switched off — the NMC restarting its network interface on save, exactly as the runbook's Before you start warns. The next scrape was 0.041 s, up never dropped, and the UPS series all still read (58 of them; upsTestResultsSummary 1, upsAdvTestDiagnosticSchedule 8, upsEstimatedChargeRemaining 100). On the iLO the same measurement was 12.09 s against 12.05 s.

    The estate is mixed on purpose, and that was the whole point of the decision. The firewall stays on v2c because bsnmpd is the only daemon serving the pf MIB; the switch stays until #444 replaces it, and ADR-0041 commissions the CRS326 with an authPriv user by this same runbook. Both of those polls ride on Winterfell.

    One thing this issue caused that is worth recording. While finishing it, a command meant to inspect a single auth block printed the whole rendered snmp.yaml, whose redaction covered the password fields and not the community ones — putting both live v2c communities into a session transcript. A value-level sweep found them in a third file too, from an unrelated project and older than today. All redacted in place at mode 600, nothing committed or off the host, and the four v3 passphrases checked individually and found nowhere. Not rotated, by decision: the MokerLink's value retires with the hardware within days, and the firewall's poll never leaves Winterfell. It is a row in SECURITY.md now, beside the 2026-08-20 one it resembles.

    Closed by #597; the two moves were #581 and #592, on the tooling #431 built.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    securityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions