nvme(4) Apple quirks

Source: sys/dev/nvme/nvme_pci.c (quirk table), nvme_private.h (flags), applied across nvme_ctrlr.c, nvme_qpair.c, nvme_sim.c.

Two controllers

Controller	PCI ID	Machines	Quirks
S3X (pre-T2)	106b:2003	MacBookPro13/14 era	NO_ASYNC_EVENT, S3X_NS1_ONLY, PCIE_FLR_ON_FATAL, IGNORE_RAB
ANS2 (T2)	106b:2005	Every T2 Mac	IDENTIFY_CNS_BROKEN, SHARED_CID_SPACE, NO_ASYNC_EVENT, SINGLE_VECTOR, 128_BYTE_SQES, IGNORE_RAB

All quirks are latched at PCI probe, before attach starts, so every stage of bring-up already knows what it is dealing with.

The T2 set

128-byte submission queue entries. ANS2 firmware demands CC.IOSQES = 7: 128-byte I/O SQEs instead of the spec’s 64. The command layout does not change; only the ring stride does. The driver carries a per-queue sqe_shift (0 for admin, 1 for T2 I/O queues), doubles the ring allocation, and addresses slots through one macro. The upper 64 bytes of each slot are zeroed padding.

One shared CID table. The spec scopes command identifiers per queue; ANS2 keeps a single table for admin and I/O together, so the same CID in flight on both queues collides inside the firmware. The fix partitions the space: the admin queue owns CIDs 0-15, I/O queues get a cid_base of 16 and their depth is clamped so everything fits in a 128-entry table. Wire CIDs are offset on submit and range-checked on completion demux.

Single MSI vector. No MSI-X. Interrupt setup short-circuits to one MSI (or INTx), a shared handler that masks INTMS, polls admin plus the single I/O queue, and unmasks, so no per-CPU queues on a T2.

Identify CNS above 1 is rejected. ANS2 faults on exotic Identify variants. The driver’s own bring-up only uses CNS 0 and 1; the guard is in the CAM passthrough path, shielding the firmware from nvmecontrol probes.

No async events. AERs never complete on Apple firmware, which would permanently pin trackers, extra poisonous when all queues share one CID table. AER configuration returns immediately with zero AERs.

The S3X set

Namespaces above 1 are hidden. The controller exposes Apple-internal namespaces that were never meant for the OS. A pair of central helpers clamp the count and filter visibility everywhere: namespace construction, child notification, changed-namespace handling, CAM’s max_lun.

FLR on fatal status. A wedged S3X asserts CSTS.CFS, and a plain CC.EN toggle then panics or times out. When reset finds CFS set, the driver saves PCI state, issues a PCIe Function Level Reset with a pending-transaction wait, restores state, and only then runs the normal disable/enable.

RAB is ignored. Upstream programs the arbitration burst from Identify’s Recommended Arbitration Burst; programming the S3X with its own reported RAB wedged sustained I/O. Quirked controllers keep the default, and an explicit tunable still overrides with a console warning.

Where each quirk bites

The quirk that died

QUIRK_APPLE_S3X_SERIALIZE forced one outstanding I/O and an 8 KB transfer clamp on the S3X. It was removed once the bugs it was papering over got fixed for real: the 8 KB clamp was exposing raw character device I/O to an oversized-mapping stall fixed separately, and the single tracker turned any stranded request into a total I/O stall. With the quirk gone, an S3X runs 128 trackers at its MDTS-derived transfer size: about seven times the serialized throughput, with no retries, failures or timeouts.

The lesson generalizes: the honest fix was making the generic driver respect what the controller reports (the MAXCMD tracker clamp) rather than hardcoding a per-device throttle. Its flag bit is now a hole in the flag space.

Seeing it

nvmecontrol devlist
sysctl dev.nvme | grep -i quirk

Validated on MacBookPro16,2 (cipher) and Mac mini 8,1 (bastion) for the T2 set; the S3X work came out of the pre-T2 MacBook Pros.