Skip to content

Run Wazuh on Saruman — the heaviest component in ADR-0007 #266

Closed

Description

@Gerrrt

Split out of #101, which named
it directly: "Wazuh in particular is not a small thing to run — it is the
heaviest component in the ADR by a wide margin."

ADR-0007 puts Wazuh on Saruman as part of the defended estate: the thing that
turns the Windows domain from a set of VMs into something that reports what
happened to it.

Blocked behind the domain — an agentless Wazuh has nothing to say.

Why it gets its own issue rather than a line in the umbrella

Wazuh is three services (indexer, server, dashboard), and the indexer is
OpenSearch. Against ADR-0007's stated constraint — "128 GB and 48 threads
against a single mirrored pair of 7.2K disks. The fleet is sized against
spindles, not RAM"
— the indexer is the single component in #101 most likely
to be what actually runs out. It is also the one whose default sizing guidance
assumes hardware this host does not have.

Worth deciding before starting

  • Retention and shard sizing, from the spindles rather than from the
    defaults.
  • Whether it lands in stacks/lab/ or beside it. ADR-0004 says one compose
    stack per host, and the lab guest is one host — which argues for one stack.
    Against: the lab observability stack is four small services and Wazuh is an
    OpenSearch cluster, and make down taking out both is a worse blast radius
    than the ADR was written to consider. Decide it explicitly; do not let it be
    decided by whoever writes the compose file first.
  • Its relationship to the lab's Loki. Wazuh has its own store and its own
    dashboard. Two log stores in one lab, each with a UI, is either sensible
    division or duplicated upkeep, and which one depends on whether anything ever
    queries across them.
  • Whether the guest can carry it at all, or whether Wazuh wants its own.

Not in scope

Sending anything to 10.0.99.20. Lab telemetry stays in the lab (ADR-0007);
the hypervisor's own agent is the only exception and it predates this (#88).


Corrected 2026-09-19

Every item under "Worth deciding before starting" is decided and delivered as code: ADR-0030 (its own guest, its own stack, sized to what Saruman has), stacks/soc in #434, build-the-soc-guest.md §0–§12 with odin at 10.0.30.60, 8 GiB, 96 GB, and the four index settings in §9. What is left is a deploy, not a decision.

The gate is unchanged and still true: #414, because an agentless Wazuh has nothing to say — and #414 now sits behind #527, the disk measurement. Closes on the day the six agents report in, not the day the guest comes up.


Corrected 2026-09-20

"#414 now sits behind #527" — no longer; #527 closed 2026-09-20 with the disks measured. The gate is #414 alone. Nothing else here changed.

Activity

  1. Gerrrt commented on Sep 8, 2026

    @Gerrrt
    OwnerAuthor

    Decided by ADR-0030, which answers this and #267 together because they are one question: same clients, same disks, same lifecycle.

    Where it runs: stacks/soc/ on odin, a second guest at 10.0.30.60 — with Velociraptor beside it.

    This issue framed the choice as stacks/lab/ versus beside it, and warned against letting it be settled by whoever writes the compose file first. The arithmetic settles it instead. alexander is 4 vCPU / 8 GiB / 64 GiB; Wazuh's own stated minimums are 4 GiB indexer plus 2 GiB manager — six before the dashboard, and before the four services already running there. stacks/lab sets no mem_limit: by decision, so nothing bounds the collision: the failure is the guest's OOM killer choosing between the SIEM and the thing that would have recorded the SIEM dying. And make down STACK=lab would become "stop detection, and stop the ability to see that detection stopped", in one command.

    A second guest is what ADR-0004 asks for, not a departure from it — "a second host means a second directory under stacks/". The tension this issue felt is not with that ADR; it is with the assumption that Wazuh has to land on the guest that already exists.

    Two things checked rather than assumed, and recorded so they are not re-litigated: two stacks on one host would have worked (stacks.sh keys on directories, and check_host_stack_table lets one host row name two stacks) — it is rejected on the guest's memory, not on the tooling. And the directory is not called wazuh, because a directory named for one of two components goes stale the day the second lands, which is ADR-0004's own argument turned on this decision.

    The premise here needs correcting

    the indexer is the single component in #101 most likely to be what actually runs out

    Likely true, but not for disk. Six agents at Wazuh's own per-agent figures make roughly 21 GB of alerts a quarter, which a 1 TB pair does not notice. What runs out is heap-per-shard: Wazuh writes one primary shard per daily index, OpenSearch's guidance is no more than 25 shards per GiB of heap, and a 2 GiB heap buys ~50 shards with ~15 taken by the monitoring, statistics, vulnerability and inventory families. Retention is 30 days because that is what the shard budget buys, not because of capacity — stated as a bound, with the re-derivation queries in the ADR.

    Four settings that are not defaults, each with its trade: number_of_replicas: 0 (the default leaves a replica unassigned forever, so the cluster is permanently yellow — and a health signal that is always yellow is not a health signal), number_of_shards: 1, refresh_interval: 30s, and an ISM delete policy at 30d, because Wazuh ships none and without one retention is "forever". vm.max_map_count is not namespaced, so it is a /etc/sysctl.d file on the guest rather than anything compose can set.

    Its relationship to the lab's Loki

    Two stores, bounded, and nothing queries across them. Loki holds the lab's own infrastructure logs; the indexer holds the estate's security events. A cross-store query is "what was the guest's Docker daemon doing when the DC alerted", which is a question you ask twice a year by opening two tabs.

    Wazuh's alerts.json is deliberately not tailed into Loki — it duplicates every alert into a second store with a second retention and untyped fields, and "which one is authoritative" arrives immediately.

    The crossing that is built runs the other way: indexer health into the lab's Prometheus, via Alloy's first-party prometheus.exporter.elasticsearch. No new image, one .alloy file, and it is what says the SIEM stopped ingesting — the #62/#63 shape this repository keeps paying for. Rules and promtool tests land in stacks/lab/prometheus/, where the series actually arrive.

    The Grafana OpenSearch datasource was considered as a way to drop the dashboard service and refused: it is a plugin, so GF_INSTALL_PLUGINS fetches an unpinned artefact at every container start — worse than a digest written where Dependabot cannot bump it. And the dashboard is not a viewer; it is agent enrolment, group management, the ruleset editor and the MITRE mapping.

    One constraint from #265 that lands on this issue

    ADR-0029 runs the two Windows endpoints per session, not continuously — the four servers are the continuous half. So endpoint detections are only meaningful during a session, and any rule written about workstation behaviour has to be written knowing that. The servers give a continuous baseline; the endpoints do not.

    Still open

    Decisions only. No stacks/soc/ directory, no age key, no .sops.yaml rule, no dependabot entry, no runbook — all of which follow the decision rather than preceding it. This stays blocked behind #265, which is now decided and has a runbook but is not built. An agentless Wazuh still has nothing to say.

  2. Gerrrt commented on Sep 9, 2026

    @Gerrrt
    OwnerAuthor

    stacks/soc/ is authored and CI-validated ahead of odin, in #434 — the way stacks/lab landed before alexander. This issue stays open: the stack cannot be deployed until #414 builds the domain and the guest exists, and an agentless Wazuh still has nothing to say. The build is build-the-soc-guest.md, and it closes this on the day the six agents report in, not the day the guest comes up.

    Two of ADR-0030's open checks are answered by the PR rather than left for build day:

    • Velociraptor's registry publishes versioned tags, so the container path stands and 0.77.2 is pinned by tag and digest; the paragraph the ADR said to supersede otherwise does not need to be.
    • Wazuh's service names must carry a dot (wazuh.indexer, upstream's names): the certs generator refuses a bare hostname as an invalid DNS name — measured by running it — and filebeat verifies the indexer's certificate in full mode, so the SAN and the dialled name have to match.

    What still waits for the day: the four index settings are applied by hand after the first start (two JSON files, one runbook step), the lab's two ports: blocks are uncommented in the same change that brings odin up, and the agents go out by GPO in a second evening — Velociraptor's MSI by software installation, Wazuh's by startup script because its MSI takes the manager address and enrolment password as properties.

    🤖 Generated with Claude Code

  3. added a commit that references this issue on Sep 10, 2026
  4. added
    socstacks/soc on odin (VLAN 30)
    on Sep 19, 2026
  5. added 2 commits that reference this issue on Sep 22, 2026
  6. Gerrrt commented on Sep 26, 2026

    @Gerrrt
    OwnerAuthor

    2026-09-26: The #414 gate has cleared for this issue's purposes. The domain was built on 2026-09-24/25, with six guests joined and scraped. #414's remaining sections (§6's users, §10, §11) don't gate agent enrolment. What's left is building odin from build-the-soc-guest.md §1–§10, then the §11 GPO rollout on a second evening. Nothing to buy. This still closes when the six agents report in.

  7. Gerrrt commented on Sep 27, 2026

    @Gerrrt
    OwnerAuthor

    odin is built and the Wazuh stack runs (2026-09-27). build-the-soc-guest.md §1–§10 are done on VMID 160 (10.0.30.60, Ubuntu 26.04.1, 8 GB / 4 core, 32 GB OS + 96 GB /srv/soc-data on large_data):

    • wazuh-indexer, wazuh-manager, wazuh-dashboard all healthy; internal-CA mTLS in place (12 certs from the generator).
    • The four non-default index settings applied — {"acknowledged":true} for both the wazuh-alerts-homelab template (1 shard, 0 replicas, 30s refresh) and the wazuh-alerts-30d ISM retention policy.
    • Vulnerability detection's feed (staging + the ~8.5 GB content DB) is on the data disk, not the OS disk — the guest is stable at its designed 8 GB (peak import RAM ~4 GB).
    • Dashboard reachable at https://10.0.30.60/.

    Fixes this shook out, all merged: the render file-mount bug (#680), first-bring-up perms + the vuln-feed placement (#681), and the docs write-down (#683).

    What's left to close this: §11 — deploy the Wazuh agent to ADR-0029's six domain machines by GPO (startup script, WAZUH_MANAGER=10.0.30.60). This issue closes when those agents report in and each server's wazuh-alerts-* count climbs — the "second evening", not the day the guest came up.

  8. Gerrrt commented on Sep 27, 2026

    @Gerrrt
    OwnerAuthor

    Closing — Wazuh is live with all six domain machines reporting (2026-09-27, §11).

    Deployed by GPO (SOC - Wazuh Agent, a computer startup script linked to the Domain Controllers, Servers and Workstations OUs), agent MSI 4.14.7 on SYSVOL, WAZUH_MANAGER=10.0.30.60, registration password from SOPS:

    • agent_control -l shows 6 agents Active — bahamut, leviathan, titan, ramuh, carbuncle, siren — plus the manager (000).
    • The SIEM is ingesting: wazuh-alerts-4.x-2026.09.27 at 3,375+ docs and climbing.
    • Indexer cluster green (after zeroing the ISM config index's replica on the single node — docs(soc): zero the ISM config index replica so the single node stays green #685), and the lab Prometheus SOC rules are all quiet.

    The stack (indexer/manager/dashboard) has been up and healthy since the guest was built earlier today. Enrolment is by GPO, so a reverted or rebuilt machine re-enrols at the next policy refresh (ADR-0030).

  9. Gerrrt commented on Oct 8, 2026

    @Gerrrt
    OwnerAuthor

    Lab-weakness detection: where it stands after #449 + #1077

    Cross-linking from #449 (the deliberate weaknesses) so #266 has a clear, honest line on detection coverage.

    Covered: the weakness being planted is now detected

    #1077 (merged via #1081) added lab rules in stacks/soc/wazuh/manager/rules/local_rules.xml. The #449 enable→disable loop on 2026-10-08 confirmed each of the six raises a dedicated, named alert on odin (before/after table in #1077's closing comment and #449):

    Weakness Rule Lvl MITRE Fires on
    asreproast 100100 10 T1558.004 4738/4742 UAC %%2096 (Don't-Require-Preauth)
    ucd 100101 12 T1558 4738/4742 UAC %%2093 (Trusted-For-Delegation)
    cd 100102 10 T1558 4738/4742 allowedToDelegateTo set
    kerberoast 100103 10 T1558.003 5136 SPN added to a user object
    rbcd 100104 12 T1134 5136 msDS-AllowedToActOnBehalfOfOtherIdentity added
    dcsync (grant) 100105 12 T1003.006 / T1222 5136 nTSecurityDescriptor change on the domain head

    Prereqs that made this possible are in #1077: Directory Service Changes auditing on the DCs, the member-server audit-policy fix, and a __PSScriptPolicyTest_* suppression (100110) for Ansible's own footprint.

    The gap: execution is not yet validated

    Every rule above fires on the configuration change — "someone planted a Kerberoast target", not "someone kerberoasted". That is real and useful (it catches the setup), but it is not technique detection.

    • 100106 (level 14, T1003.006) is the only rule aimed at an actual attack event — 4662 replication by a non-DC account, i.e. DCSync itself. It has never fired, because no attack has run; it is exercised only in a throwaway manager.
    • The characteristic attack telemetry — 4769 RC4 (kerberoast), 4768 no-preauth (AS-REP), non-DA 4662 (DCSync), S4U2Self/Proxy chains (cd/rbcd), unconstrained-delegation TGT theft (ucd) — has never been generated in the lab.

    What closing that gap needs

    A live technique→detection pass from an analyst host on VLAN30 (Kali 902), which does not exist yet — blocked on ifrit #790. Until that runs, #266 can claim planting detection for all six, but not execution detection. When the attacker host lands, re-running the loop with live tooling (impacket/Rubeus) against these same rules is the test that promotes this from posture to technique coverage.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a changesocstacks/soc on odin (VLAN 30)

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions