Skip to content

Decide what stands between ZFS and the Exos pair: the MegaRAID SAS3008 nobody recorded, after the swap #571

Closed

Description

@Gerrrt

What the swap found, and what it defers

Read at smaug's console on 2026-09-19 and 2026-09-20 (#558): both Exos X20 drives sit behind a Broadcom/LSI MegaRAID SAS3008 in the PCIe slot — 1000:005f, subsystem 1000:9340, the ThinkServer RAID 520i — in JBOD pass-through on a cacheless iMR, driver 07.727.03.00-rc1, JBOD sequence map : enabled. The chipset's AHCI carries the boot SSD alone. No document had recorded the card; hardware.md said AHCI was what kept a controller out from between ZFS and the disks, and #565 corrected it.

The card's firmware handled the fault, and its error path — 60 s command timeouts, task aborts, a target reset, an Online Controller Reset at 21:03:51 — is what held the exporter and what ZFS saw. The disks are JBOD, so smartctl reaches them by their own model and serial and ZFS owns the mirror; that is the better of the two MegaRAID arrangements and it is still not an IT-mode HBA.

#558 says the decision — leave the card, flash the 3008 to IT firmware, or cable the bays to the chipset's free ports and remove the card — is "for this issue after the swap". But #558 closes on the swap and the resilver, and a decision parked in a closing issue is the shape #485 was filed against. This is the decision's home.

What has to be read first

The decision, and its three answers

Answer For Against
Leave it, JBOD Nothing to buy or flash; measured working through a real fault Firmware between ZFS and the disks; its error path is slow and opaque; -d megaraid may be needed on a future firmware
Flash to IT mode The HBA ZFS wants; mpt3sas instead of megaraid_sas A flash on the only controller, with the pool on it; recovery from a bad flash is a second card
Chipset AHCI, card out Simplest path; what the documents assumed all along Only if the bays reach the chipset; loses the card's SAS ports for nothing this box needs

Either way an ADR or an amendment to ADR-0040, because the pool layout is the part of that ADR outside CI's reach and this is the layer beneath it.

Done when

The controller arrangement is decided and recorded, hardware.md describes the path from bay to ZFS as it is, and replace-the-nas-disk.md step 4's "confirm the new device in lsblk" says which arrangement it is written for.

Refs #558, #413, ADR-0040. Not before the swap.

Activity

  1. added this to the NAS milestone on Sep 20, 2026
  2. Gerrrt commented on Sep 22, 2026

    @Gerrrt
    OwnerAuthor

    Firmware read 2026-09-22, at the power-on after ZVTBSDL3 was pulled (#558 step 4), in the UEFI setup under Advanced → AVAGO MegaRAID Configuration → Controller Management: AVAGO MegaRAID SAS 9340-8i, PCI ID 1000:005F:1000:9340, package 24.16.0-0104, firmware 4.660.01-8219, NVDATA 3.1605.01-0008, 2 connectors, Optimal, no BBU, 0 virtual drives, and the remaining Exos on drive port 0 as JBOD. Recorded in hardware.md on #620. The controller decision still waits until after the swap.

  3. added 2 commits that reference this issue on Sep 23, 2026
  4. reopened this on Sep 26, 2026
  5. Gerrrt commented on Sep 26, 2026

    @Gerrrt
    OwnerAuthor

    2026-09-26: reopened. #652 closed this by accident. Its body says "Closes nothing yet… (§6 now closes #571 with #558)", and GitHub matched "closes #571" inside that sentence.

    The decision is made: ADR-0052 cables the pool to the chipset's AHCI ports and takes the card out. The done-when also needs hardware.md to describe the path from bay to ZFS as it is, and that only happens at #558's swap. replace-the-nas-disk.md §6's checklist resolves this issue together with #558.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    mediastacks/media on smaug (VLAN 40)

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions