N5 MAX: the final verdict

1 · Executive summary

Each headline finding below carries its evidence: stage, layout, test id and the run timestamp of the sealed case.

1. Recommended default stays RAIDZ1 5-wide, and stage 7 now backs it.

   Degraded with one member out, RAIDZ1 still read 93.51 % of healthy sequential throughput and resilvered in 1,740 s at 1.1656 GB/s. btrfs raid5 fell to 5.87 % and took 3,720 s at 0.5498 GB/s.

   S7 · zfs-raidz1 / btrfs-raid5 · STO-09 STO-01_seq_read_1M_j4 · runs 2026-09-24-19.58 / 2026-09-24-05.07

2. Fastest measured is now a tie: btrfs raid5 and mdadm raid5 (cold seq read).

   btrfs raid5 9.034 GB/s at 14.43 TiB, mdadm+ext4 raid5 9.035 GB/s at 14.41 TiB. Both are on the capacity × cold-read Pareto frontier from the lab’s Pareto tool. RAIDZ1 5-wide reads 7.83 GB/s (1.154× behind btrfs raid5).

   S5 · btrfs-raid5 / mdadm-ext4-raid5 · STO-01_seq_read_1M_j4 · runs 2026-09-14-13.57 / 2026-09-25-18.53

3. The “mdadm is 8× slower” anomaly is resolved: it was a link fault, not mdadm.

   SN1 had trained at Gen1 2.5 GT/s x1 in every original mdadm cell (#72), which pinned seq read at 1.14 GB/s. The reruns at Gen4 read 9.035 GB/s.

   S5 · mdadm-ext4-raid5 · STO-01_seq_read_1M_j4 · run 2026-09-25-18.53 (superseded 2026-09-15-00.41)

4. mdadm parity writes fast in sequence but collapses on 16K random writes.

   mdadm raid5 has the highest cold sequential write (5.04 GB/s vs btrfs raid5 4.066), but only 2,489 IOPS on 16K random writes vs btrfs raid5 22,981. That is partial-stripe read-modify-write. mdadm raid6 is the same: 2,313 IOPS.

   S5 · mdadm-ext4-raid5 · STO-01_seq_write_1M_j4, STO-02_rand_write_16k_j4 · run 2026-09-25-18.53

5. The ZFS-vs-btrfs write gap in the FS controls is a drive-state effect, not a filesystem verdict.

   The ZFS stage-1 cells fully preconditioned their members before measuring, and the btrfs fs-control cells did not (per the lab’s backend notes). There btrfs raid5 wrote 4.066 GB/s vs RAIDZ1 0.718. Under STO-09 S1 (same protocol, file-on-FS, both after the same 50 % incompressible fill) RAIDZ1 wrote 4.499 GB/s and btrfs raid5 2.78 GB/s.

   S7 S1 · zfs-raidz1 / btrfs-raid5 · STO-01_seq_write_1M_j4 · runs 2026-09-24-19.58 / 2026-09-24-05.07

6. The four x1 slots tax sequential work about 3×, but barely touch random and sync work.

   One NM790 at x4 vs x1: seq read 5.888 vs 1.8 GB/s (3.27×), 4K random read 41,318 vs 35,116 IOPS (1.18×), sync 16K 1,398 vs 1,283 (1.09×). One x1 drive reaches 91 % of the 1.969 GB/s lane.

   S1 · single-x1 / single-x4 · STO-01, STO-02, STO-04 · runs 2026-09-13-13.00 / 2026-09-12-19.42

7. ZFS RAID10 is the sync-write layout; RAIDZ1 pays for parity on sync.

   Cold sync 16K: RAID10 1,675 IOPS vs RAIDZ1 415. primarycache=metadata does not help sync (RAIDZ1 256), but it lifts 4K random reads by 50.2 % on RAIDZ1 and 58.7 % on RAID10.

   S1/S2 · raid10 / raidz1 · STO-04_sync_write_16k_j4, STO-02_rand_read_4k_j4 · runs 2026-09-13-15.49 / 2026-09-13-17.14

8. Once a model is resident, the pool does not matter for inference.

   Gemma 4 26B tg128 is 50.15 tok/s on btrfs and 50.08 on ZFS. Qwen3.8-27B (gufo-model-bench) decodes 12.44 tok/s from btrfs cold and 12.34 from ext4 warm. Pools differ only in cold load (5.035 s vs 6.758 s for Gemma).

   S3 · btrfs raid5 / ZFS RAIDZ1 · AI-01 card + load rail · runs 2026-09-22-06.02 / 2026-09-23-04.57

9. We sit below the Minisforum vendor floor on all 16 comparisons.

   Ours ÷ floor ranges from 0.7128 to 0.9715. gpt-oss-120b HIP is 0.8562 (pp512) and 0.8976 (tg128). Decode on Gemma 4 and Qwen3.5-122B is the furthest behind (0.7128, 0.7252 HIP). The vendor states no conditions beyond ngl 99 and FA on.

   S3 · btrfs raid5 · AI-01 vendor group · run 2026-09-23-15.03

10. Vulkan decodes faster, HIP prefills faster, and Vulkan loads much slower.

    Across the four floor models, Vulkan tg128 is 1.062–1.107× HIP and HIP pp512 is 1.038–1.082× Vulkan. Vulkan cold load takes 1.34–5.22× as long (gpt-oss-120b 49.375 s vs 11.256 s).

    S3 · btrfs raid5 · AI-01 vendor group + load rail · HIP/Vulkan cells

11. Streaming models larger than memory is a load-time story, and ZFS loses it.

    Kimi K2 (streamed) cold load took 38.334 s on btrfs vs 181.147 s on ZFS (primarycache=all). Decode is unusable either way: 1.09 / 1.55 tok/s.

    S3 · btrfs raid5 / ZFS RAIDZ1 · AI-01 kimi-k2 streamed · runs 2026-09-29-03.21 / 2026-09-29-13.35

12. Thermals are a non-issue for an hour of sustained work.

    In the 1-hour RAIDZ1 soak, mixed 70/30 held 0.886 GB/s and was steady, with drives peaking at 59.85 °C and the SoC at 52.0 °C. The sequential-write soak (1.178 GB/s) was not steady.

    S4 · raidz1 · SOAK-02_mixed_70r30w_16k_j4 · run 2026-09-12-14.31

2 · Setup & method

Selector accounting at build time: 252 run directories, 964 published subjects. Case verdicts: VOID 250, PASS 1428, FAIL 127, UNMEASURED 37. Coverage of the campaign matrix:

3 · Storage findings

3.1 Layouts: the tradeoff compass

Plotting capacity against cold sequential read shows the whole decision on one chart. The parity layouts on all five drives cluster at the top. With SN5 at x4 they read 9.0 GB/s (btrfs raid5/6, mdadm raid5/6) and 8.911 GB/s (RAIDZ2). RAIDZ1 sits a step lower at 7.83 GB/s with the second-highest usable capacity of any candidate (14.11 TiB, against 14.43 TiB for btrfs raid5). The mirrors pay half the raw capacity: ZFS RAID10 holds 6.71 TiB at 7.024 GB/s.

3.2 Every layout, six workloads

Three patterns hold up. (1) Every parity layout reaches 9.0 GB/s sequential reads once SN5’s x4 lane is a member. The x1 members are the limit, not the filesystem. (2) Sync writes favour mirrors on ZFS (RAID10 1,675 vs RAIDZ1 415 IOPS). On files they favour btrfs over mdadm raid10 (2,153 vs 323). (3) mdadm parity is a sequential-write specialist: highest seq write (5.04 GB/s), worst 16K random write (2,489 IOPS).

RAIDZ1 cold mixed 70/30 (757 IOPS) is PROVISIONAL: its own prior good value was 14,582 IOPS, a 19× spread (lab storage report). Do not quote it.

3.3 Cold vs metadata

primarycache=metadata keeps the indirect blocks in ARC, so random reads stop paying for metadata fetches. 4K random read rises 50.2 % on RAIDZ1 and 58.7 % on RAID10. Sync writes do not improve, and on RAIDZ1 they fall (415 → 256 IOPS). Sequential reads barely move. Production pools would normally run primarycache=all, which is warmer still, so the cold numbers here are a floor.

3.4 The x1 lane tax

One x1 drive reads 1.8 GB/s, 91 % of its 1.969 GB/s lane, and ZFS RAID10 over four x1 drives reads 3.90× that. So the striping works, and the lanes are the ceiling. Small-block random and sync IO are bound by the drive’s controller and flash, not the link. The four x1 slots are therefore a sequential-bandwidth tax, not a responsiveness tax.

3.5 ZFS vs btrfs vs mdadm

Healthy sequential. On the fs-control battery, btrfs raid5 out-reads RAIDZ1 (9.034 vs 7.83 GB/s at j4; 9.033 vs 6.748 at j1) and out-writes it by 5.66× (4.066 vs 0.718). That write gap does not survive a like-for-like test. STO-09 S1 runs the same file-on-FS protocol on both after the same 50 % incompressible fill. There RAIDZ1 wrote 4.499 GB/s and btrfs raid5 2.78. Reads kept their order: RAIDZ1 7.434 vs btrfs 9.045 GB/s.

The likeliest explanation is NM790 write state. The ZFS stage-1 cells fully wrote their zvols and benchmark file before measuring. The btrfs fs-control cells got no equivalent preconditioning (mkfs.btrfs takes seconds), as the lab notes in its backend code. The soak and stage-7 control show this drive slows sharply once its SLC cache is spent (§3.7, §6). The fs-control seq-write gap therefore compares different drive states, not filesystems.

Random IO cannot be ranked across stacks from this campaign: ZFS used zvols (volblocksize-bound), btrfs and mdadm used files. In STO-09 S1, ZFS 4K random reads on a recordsize=1M dataset ran at 0.033 GB/s against btrfs 2.226 GB/s. That is mostly whole-record read amplification of 1M records. It is a tuning artefact, and the reason a VM or database dataset needs a small recordsize or a zvol.

3.6 Write amplification

Every layout amplifies writes by what its geometry predicts: mirrors 2.006× (ZFS), 2.002× (btrfs), 1.999× (mdadm); 5-wide single parity 1.245× / 1.255× / 1.25×. Double parity sits slightly above the 5/3 expectation on btrfs (1.764×) and mdadm (1.716×), and close to it on RAIDZ2 (1.679×). For endurance on a no-PLP, DRAM-less drive, RAIDZ1 writes 38 % fewer flash bytes per logical byte than a ZFS mirror.

3.7 Degraded, rebuild and the write return (stage 7)

Reads with a drive out. ZFS reconstructs from parity at nearly full speed: RAIDZ1 93.51 %, loaded 97.11 %, RAIDZ2 59.55 %. btrfs raid5 and raid6 collapse to about 0.53 GB/s (5.87 % / 5.86 %), whichever member failed, including the x4 SN5 (5.86 %). btrfs raid10 keeps 71.19 %.

Rebuild. ZFS resilvers only allocated blocks, at 1.1656–1.3828 GB/s (29–28 min at 50 % fill). btrfs replace ran 0.5498 GB/s on raid5 (62 min) and 0.3207 GB/s on raid6 (106 min). The replacement received its allocated-only share as predicted (observed/predicted 1.002 on zfs-raidz1, lab storage report).

The write return. All arms got their reads back (S4 93.17–112.7 % of S1). Every ZFS arm and btrfs raid6 failed the harness’s S4 write test (more than 10 % below S1): RAIDZ1 39.32 %, RAIDZ2 46.95 %, btrfs raid6 32.67 %.

The preregistered no-failure control ran the same sequence with no failure and no rebuild. It returned 66.58 % and 67.71 % of S1, with S4/S2 = 0.9986 / 1.0167. So roughly two-thirds of the S4 level is what these drives deliver after the write volume of S1–S2 alone. The extra loss in the failure arms (RAIDZ1 down to 39.32 %) is failure- or rebuild-specific by the preregistered rule.

The lab report places the knee at about 2.3–2.4 TB written per survivor. Idle and TRIM recovery rungs did not restore the rate. This is a drive write-state and endurance characteristic; it says nothing about data integrity either way.

3.8 Soak and thermals (stage 4)

One-hour soaks on RAIDZ1 and RAID0, metadata regime. RAIDZ1 mixed 70/30 held 0.886 GB/s (54,107 IOPS, steady). Drives peaked at 59.85 °C and the SoC at 52.0 °C. RAID0 mixed held 1.616 GB/s (steady, drives 60.85 °C). The sequential-write soaks were not steady: RAIDZ1 1.178 GB/s, RAID0 1.473 GB/s. That is the same post-cache write sag as in stage 7, not thermal throttling: drive temperatures stayed under 60 °C.

3.9 USB ingest (stage 6)

The campaign’s stage-6 STO-13 cell passed, but its result was lost: it was written to a tmpfs scratchpad that a host reboot wiped. The post-campaign variant STO-13b (a USB-attached NVMe ↔ RAIDZ1 5-wide, 200.74 GiB corpus, different OS image) recorded 905.6 MB/s reading from USB into the pool and 376.2 MB/s writing out (rsync including the sync drain). The pool was not the limit in either direction. The enclosure fell back from UAS to usb-storage, so treat these as a floor for this enclosure, not for the port. Integrity: 0 failures.

4 · AI findings

Stage 3 ran llama.cpp (HIP/ROCm and Vulkan/RADV) on 11 models, from a 0.6B smoke model to Kimi K2 streamed from disk. Most cells load from cold btrfs raid5; paired cells load from ZFS RAIDZ1 (primarycache=all, ARC cold at start). All 21 cells passed. The lab’s status table still lists 21 per-test rows as UNMEASURED (#67/#68). Full table in the appendix.

4.1 Throughput per model

Resident models (HIP, pp512 / tg128 tok/s): gpt-oss-20b 1,488.02 / 75.49; Gemma 4 26B-A4B 1,130.87 / 50.15; gpt-oss-120b 564.01 / 53.12; Qwen3.5-122B-A10B 308.21 / 22.97; dense Qwen3.8-27B Q4 332.94 / 12.59; DeepSeek V4 Flash IQ2_XXS 27.35 / 15.26. Streamed models are not interactive: DeepSeek V3.1 Q2 1.78 and Kimi K2 1.09 tok/s decode.

4.2 Cold load, resident, streamed

Resident. Layout is irrelevant to pp and tg. Gemma 4 on btrfs vs ZFS: 1,130.87 vs 1,136.82 pp512, 50.15 vs 50.08 tg128. Qwen3.8-27B decodes 12.44 tok/s off cold btrfs and 12.34 off a warm ext4 USB boot. Cold load is where pools differ, and only modestly for resident models: 5.035 s btrfs vs 6.758 s ZFS for Gemma. Backend matters more than layout: Vulkan cold loads take 1.34–5.22× as long as HIP on the same pool. Streamed models (larger than practical GPU memory, read via mmap with ~21× read amplification) load 4.73× slower from ZFS (Kimi K2), although decode stays at 1–2 tok/s either way.

The DeepSeek V3.1 streamed cell’s load-rate field (49.927 GB/s) is mmap-deferred bytes over wall time, not a pool rate. Its pool rail read 3.761 GB/s.

4.3 Prefill and decode with context

Prefill holds within a few percent to 8k for the MoE models and drops to 457.82 tok/s at 32k for gpt-oss-120b (81 % of pp512). The dense Llama-2-7B falls hardest (432.08 tok/s at 32k, 33 %).

Decode at 55k keeps 86.8 % on Qwen3.5-122B but only 59.0 % on gpt-oss-120b (31.24 tok/s). The fastest decoders lose the largest share, because the per-token KV read is a bigger fraction of a fast token.

Stage-3 curves stop at 32k (prefill) and 55k (decode). The gufo-model-bench rows carry Qwen3.8-27B to 131,072 tokens: pp2048 329.8 → 130.25 tok/s.

KV-cache quantisation was not run. Every cell used f16 KV. There are no q8_0 rows (#61).

Speculative decoding (USB live boot, ext4, post-campaign, gufo DFlash2): Qwen3.8-27B tg128 rises from 12.34 to 22.68 tok/s on prose (1.84×). That is the largest single decode lever measured on this box for a dense model.

4.4 Which layout helps AI

For resident models, none: put models wherever capacity is convenient. Cold load favours btrfs slightly for resident models (5.035 vs 6.758 s) and strongly for streamed ones. If you plan to stream models larger than memory, keep them on a non-ARC filesystem or a dataset with primarycache=metadata. The second option is a recommendation from mechanism; it was not tested here. For a RAG box the deciding properties are the storage ones: degraded reads, rebuild time, and capacity for embeddings and documents. All three point to RAIDZ1.

5 · Competitive analysis

External figures are quoted as “X reports Y”, with condition differences noted. None of them is our result.

Reading. On llama-bench we land 3.5–4.3 % below the vendor’s prefill numbers for three of four models (HIP; gpt-oss-120b 14.4 %) and 9.4–28.7 % below its decode numbers. The vendor’s conditions are unstated, so this is a floor comparison and not a defect claim. Against reviewers using consumer front-ends (Ollama, LM Studio), our llama.cpp numbers are higher, as expected from the leaner engine. Against same-engine community references (Gufo), we match decode and trail prefill by about 8 %. Against newer ROCm toolboxes (kyuz0), we trail prefill by about 12 %. The gap tracks the software stack, not the box.

As a NAS. Every candidate pool reads well above what one 10GbE link carries (1.25 GB/s), so a single client will be network-bound on reads. In the post-cache write state, though, RAIDZ1 sequential write (0.718 GB/s cold, stage 1) is below 10GbE line rate. Sustained large ingest can therefore be drive-bound on this NM790 population. Network throughput itself was not measured in this campaign.

6 · Cross-test insights

• Drive write state outranks the filesystem. The same RAIDZ1 layout wrote 4.499 GB/s in STO-09 S1 (after a 50 % fill), 0.718 GB/s after full preconditioning (stage 1), and 1.178 GB/s (non-steady) in the one-hour soak. The no-failure control lost a third of S1 without any failure. Any write comparison on NM790s must state the drive state.
• SN5 at x4 is what makes the 5-wide layouts fast. Every 5-member parity layout reads 9.0 GB/s. The superseded 4-wide RAIDZ1 on four x1 members read 6.467 GB/s (RUN SUPERSEDED, comparison only).
• Link training is a silent failure mode. SN1 trained at Gen1 in six cells (#72) and again on 2026-09-29 (that AI run is excluded). A per-run link-speed gate is cheap insurance; see gaps.
• Warm reads hide layout. All six warm FS controls read 35.685–36.74 GB/s, the DRAM ceiling. Small-file copy (STO-06) converges at 0.418–0.724 GB/s across every cold layout. Neither separates layouts.
• AI and storage barely interact once loaded. Inference reads the model once. After that, the only storage-sensitive AI behaviour is cold load and streaming. Pick the pool on storage merits.

7 · Recommendations

Per workload

Per OS / stack

8 · Buying guidance

• Buy it if you want one quiet box that serves files at 10GbE and runs 20–120B MoE models at interactive speed: gpt-oss-120b at 53.12 tok/s, Qwen3.5-122B at 22.97 tok/s.
• Do not expect NVMe-array sequential numbers beyond about 9 GB/s, or more than about 2 GB/s from any one of the four x1 slots. Small-block IO is unaffected.
• Drives: the NM790 is DRAM-less with no PLP, and its sustained write falls sharply after the SLC cache (§3.7). For write-heavy or sync-heavy use, prefer drives with DRAM and PLP, especially in the x4 slot.
• Dense models are bandwidth-bound: ours decodes Qwen3.8-27B Q4 at 12.59 tok/s; ServeTheHome reports ~9 tok/s for Qwen3.6 27B (FP8) and Level1Techs reports 4.97 tok/s for Llama 3.1 70B on this silicon. MoE models are where the box shines.
• Expect software to move results more than storage does: ROCm version, backend and engine shift pp by 10–25 %.

9 · Limitations & open questions

• Write-hole test (#59) not run; SLOG (#57) not run; KV-cache quantisation (#61) not run; prefill past 32k in stage 3 (#60) covered only by the Gufo rows to 131k; scheduling probe (#62) not run.
• Stage-6 STO-13 result lost (tmpfs). Only the post-campaign STO-13b exists, on a different OS image.
• SATA bays unmeasured. Network throughput unmeasured.
• Random IO across stacks is access-method-confounded (zvol vs file). No file-based ZFS random cell exists.
• RAIDZ1 cold mixed 70/30 is PROVISIONAL (19× spread). Stage 1 cell 5 (RAIDZ2 4-wide) is incomplete.
• Stage 7: the attended rebuild arm with per-member io_ticks, which the control decision asked for, has not run. Only two control replicates exist.
• Vendor floor conditions unknown. Vulkan vs HIP gaps (#46/#48) are characterised, not explained.
• The btrfs raid10 low cold read (3.616 GB/s) is unexplained.
• NM790 endurance (TBW consumed by the campaign) is not sealed.

Each of these is tracked as a follow-up in the lab’s private issue tracker.

10 · What changed since the 2026-09-18 review

A · Evidence tables

Run ids are timestamps of sealed result directories (appendix tables omit the 2026 year). Red = non-steady; grey = no steadiness verdict; ⚠ = PROVISIONAL. Full sealed precision is kept in the lab’s private metrics ledger.