Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/runbooks/verify-the-alert-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ what makes the next one mean anything.
alertname=AlertPathCutover severity=critical category=availability \
--annotation=summary='#136 cutover: in-house and ntfy.sh'
# on Wi-Fi: it arrives on the in-house urgent topic, on both phones,
# readable rather than JSON
# as a title and one line (templates/homelab.yml), not JSON
# on mobile data, Wi-Fi off: it arrives on the ntfy.sh urgent topic
amtool alert add --alertmanager.url=http://localhost:9093 \
alertname=AlertPathCutover severity=critical category=availability \
Expand Down
19 changes: 11 additions & 8 deletions secrets/observability.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,22 +58,25 @@ SNMP_PRIVPASS_ILO: change-me-ilo-priv
# Since #136 these three are the sensitive tier's own ntfy, one topic each.
# The topic names are NTFY_TOPIC_ALERTS / _URGENT / _SECURITY in
# secrets/sensitive.sops.yaml, where ntfy grants access to them; copy each
# name here, after the host, and end every URL with ?template=alertmanager so
# the phone shows a title and a summary instead of JSON. Nothing about the
# name here, after the host, and end every URL with ?template=homelab, the
# repository's own rendering in stacks/sensitive/ntfy/templates/, so the phone
# shows a title and one line instead of JSON. Nothing about the
# token goes in the URL β€” it is ALERTMANAGER_NTFY_TOKEN below, sent as a
# header, because Caddy's access log records the URL.
ALERTMANAGER_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-alerts-topic?template=alertmanager
ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-urgent-topic?template=alertmanager
ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-security-topic?template=alertmanager
ALERTMANAGER_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-alerts-topic?template=homelab
ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-urgent-topic?template=homelab
ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-security-topic?template=homelab

# The same urgent and security channels, AGAIN, on ntfy.sh β€” for a phone that
# is not on the home network and so cannot reach the in-house server (#136).
# On ntfy.sh the topic name is the credential (SECURITY.md), so these are two
# long random names, and they are the topics these two channels used before
# the in-house server existed: moving the old values here is the whole
# migration. No `?template=` β€” the ntfy.sh copies stay as they were.
ALERTMANAGER_URGENT_EXTERNAL_URL: https://ntfy.sh/change-me-to-a-long-random-topic
ALERTMANAGER_SECURITY_EXTERNAL_URL: https://ntfy.sh/change-me-to-another-long-random-topic
# migration. Add `?template=alertmanager`, ntfy's built-in rendering: ntfy.sh
# cannot load the repository's own template, and without either one the
# phone shows the raw JSON payload.
ALERTMANAGER_URGENT_EXTERNAL_URL: https://ntfy.sh/change-me-to-a-long-random-topic?template=alertmanager
ALERTMANAGER_SECURITY_EXTERNAL_URL: https://ntfy.sh/change-me-to-another-long-random-topic?template=alertmanager

# The bearer token the three in-house receivers present, and the SAME value as
# NTFY_ALERTMANAGER_TOKEN in secrets/sensitive.sops.yaml β€” two copies because
Expand Down
6 changes: 4 additions & 2 deletions stacks/observability/alertmanager/alertmanager.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -175,8 +175,10 @@ receivers:
# - The certificate is verified against the TIER'S root, mounted from
# certificates/tier-ca.pem. Per receiver, not process-wide: nothing else
# Alertmanager talks to is asked to trust it (ADR-0037, docs/security.md).
# - Each URL ends `?template=alertmanager`, ntfy's built-in rendering of
# this payload, so a phone shows a title and a summary instead of JSON.
# - Each URL ends `?template=homelab`, the repository's rendering of this
# payload in stacks/sensitive/ntfy/templates/homelab.yml, so a phone
# shows a title and one line instead of JSON. The ntfy.sh copies use
# `?template=alertmanager`, the built-in, since ntfy.sh cannot load ours.
#
# urgent and security carry a SECOND webhook each, to their ntfy.sh topic,
# for the phone that is not on the home network and so cannot reach the
Expand Down
7 changes: 7 additions & 0 deletions stacks/sensitive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,13 @@ here.
already delivered. [ADR-0023] classes ntfy as unclassed for the same reason.
`backup-volumes.sh` skips the volume by name, which also keeps ntfy up
through a backup, the moment a failed backup would want to page.
- **Alerts arrive as a title and a line.** `ntfy/templates/homelab.yml`
renders each Alertmanager payload with a severity marker and the summary as
the title, the description's first sentence and the host as the message,
and a priority that follows severity. Critical is 5, which on Android is the
loud channel and on iOS is time-sensitive. The full text stays in
Alertmanager and Grafana. The ntfy.sh copies use ntfy's built-in
`alertmanager` template, because ntfy.sh cannot load ours.
- **No second factor.** ntfy has passwords and tokens. The only human account
is `phone`, which can read three topics of alert text and nothing else.
[ADR-0022] leaves ntfy out of its table for that reason: it authenticates no
Expand Down
4 changes: 4 additions & 0 deletions stacks/sensitive/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -663,6 +663,10 @@ services:
alertmanager:${NTFY_ALERTMANAGER_TOKEN:?set in secrets/sensitive.sops.yaml}:alertmanager
volumes:
- ./ntfy/server.yml:/etc/ntfy/server.yml:ro
# ntfy's default template-dir. homelab.yml there is what turns an
# Alertmanager payload into a title and a line; a directory mount, so
# check_mounted_config.py's single-file inode concern does not apply.
- ./ntfy/templates:/etc/ntfy/templates:ro
- ntfy-data:/var/lib/ntfy
expose:
- "8080"
Expand Down
62 changes: 62 additions & 0 deletions stacks/sensitive/ntfy/templates/homelab.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
# How an Alertmanager notification looks on a phone (#136). Alertmanager's
# in-house receivers post to `?template=homelab`, and ntfy renders this file
# against the webhook's JSON before it stores or pushes anything.
#
# Every rule in the estate carries a one-line `summary` and a `description`
# that is a paragraph, written for someone at a keyboard. A phone wants the
# first and a hint of the second:
#
# title a severity marker and the summary. For a group of several, the
# count and the alert name.
# message the description's first sentence (at most 200 characters), and
# the host. For a group, one line per alert's summary.
# priority critical 5, warning 3, and resolved or anything else 2, so a
# page is loud, a warning is ordinary and the daily Watchdog,
# whose severity is `none`, is quiet.
#
# Checked against every alerting rule's description when it was written. Four
# first sentences run past 200 characters and are trimmed with an ellipsis,
# and the rules with no description show severity and category instead,
# because an empty message makes ntfy print "triggered". The full text is in
# Alertmanager and Grafana. `.yml`, not `.yaml`: ntfy loads nothing else.
title: |
{{- $sev := .commonLabels.severity | default "" -}}
{{- if eq .status "resolved" }}βœ… Resolved: {{ else if eq $sev "critical" }}πŸ”΄ {{ end -}}
{{- if and (ne .status "resolved") (ne $sev "critical") -}}
{{- if eq $sev "warning" }}🟠 {{ else }}πŸ”΅ {{ end -}}
{{- end -}}
{{- $n := len .alerts -}}
{{- if eq $n 1 -}}
{{- $a := index .alerts 0 -}}
{{- $a.annotations.summary | default $a.labels.alertname -}}
{{- else -}}
{{- $n }} Γ— {{ .commonLabels.alertname | default "alerts" -}}
{{- end -}}
message: |
{{- $n := len .alerts -}}
{{- if eq $n 1 -}}
{{- $a := index .alerts 0 -}}
{{- $d := $a.annotations.description | default "" | trim -}}
{{- if $d }}{{ $d = first (splitList ". " $d) | trimSuffix "." }}{{ end -}}
{{- if gt (len $d) 200 -}}
{{- $d = printf "%s…" (trunc 199 $d | trim) -}}
{{- else if $d -}}
{{- $d = printf "%s." $d -}}
{{- end -}}
{{- if not $d -}}
{{- $sevl := $a.labels.severity | default "no severity" -}}
{{- $d = printf "%s Β· %s" $sevl ($a.labels.category | default "uncategorised") -}}
{{- end -}}
{{- $d -}}
{{- $where := $a.labels.host | default ($a.labels.instance | default "") -}}
{{- if $where }}{{ "\n" }}πŸ“ {{ $where }}{{ end -}}
{{- else -}}
{{- range $i, $a := .alerts -}}
{{- if $i }}{{ "\n" }}{{ end -}}
β€’ {{ $a.annotations.summary | default $a.labels.alertname }}
{{- end -}}
{{- end -}}
priority: |
{{- $sev := .commonLabels.severity | default "" -}}
{{- if eq .status "resolved" }}2{{ else if eq $sev "critical" }}5{{ else if eq $sev "warning" }}3{{ else }}2{{ end -}}
Loading