apple_bce(4) / bce_vhci

Source: sys/dev/apple_bce/apple_bce.c, apple_bce_mailbox.c, apple_bce_queue.c, apple_bce_vhci.c plus headers, sys/modules/apple_bce/.

What it is

On a T2 Mac the keyboard, trackpad bridge, Touch Bar and ambient light sensor do not sit on a USB controller the host can see. They sit behind the T2, and the T2 offers exactly one way in: the Buffer Copy Engine, PCI function 1 of device 0x106b:0x1801 (function 0 is the NVMe controller). The BCE is a generic DMA pipe: named submission and completion queues carrying buffer-copy commands to and from the coprocessor.

apple_bce drives the pipe. bce_vhci, its child, uses it to present a virtual USB 2.0 host controller to the FreeBSD USB stack:

pci → apple_bce → bce_vhci → usbus → uhub → keyboard, sensors, ...

Three T2 rituals happen before any of that works: function 0 must be made bus master or the T2 rejects DMA on function 1; the host must write a boottime timestamp to BAR4 every 150 ms or firmware assumes it died; and DMA is limited to 37 bits.

The mailbox

Everything starts with a 64-bit mailbox in BAR4: type in the top 6 bits, value in the rest. It sets the protocol version, registers the first queues, and handles sleep and wake. One message in flight, ever.

The cold-boot fix lives here. During early boot sema_timedwait has no working timer, so a cold sender polls the status word and calls the interrupt handler by hand with DELAY(100) between rounds. The other half: the ISR posts the semaphore only when not cold, because a cold sender never consumes the token, and a leftover token would satisfy the next warm send instantly with a stale result.

Queues

Completion queue entries carry a pending flag as the valid bit, the source queue id, and a status. Submission entries are an address and a length. Doorbells live at BAR2 + 0x44000 + qid*4. There is no zero-copy: every VHCI transfer queue owns one 4 KB bounce buffer, and payloads shuttle through it with usbd_copy_in/usbd_copy_out.

The DMA interrupt walks the completion queues, routes each entry to its submission queue’s completion mailbox by qid, rings the consumer doorbell, then fires each pending queue’s callback. On top of one such pair sits a synchronous command queue, which is how queues themselves get registered with firmware under names like VHC1HostCommands and VHC1-<dev>-<ep>.

The virtual host controller

bce_vhci embeds a usb_bus, emulates the root hub in software, and translates every USB transfer into queue traffic. Five host-to-firmware message queues, five firmware-to-host event queues sharing one completion queue. A message is 16 bytes: command, status, two params. Reply flag 0x8000, cancel flag 0x4000.

Firmware events that need to sleep (endpoint pause handling flushes queues) are deferred from the ISR through a small ring to a taskqueue. That task must never send a command of its own: the reply would arrive on the very event queue the task is responsible for draining. Replies to firmware go out on the system message queue instead.

After attach, the driver kicks the USB explore thread by hand. The keyboard was connected before the kernel booted, so no port-change event will ever announce it; without the kick, nothing behind the T2 enumerates.

The control transfer state machine

Endpoint zero is where the protocol gets personal. Firmware drives OUT transfers with TRANSFER_REQUEST events, but never sends one for the IN data phase: the host must initiate that itself, which means the host then receives a TRANSFER_REQUEST it already acted on.

Ordering is race-tolerant in both directions. If the status event arrives before the data DMA lands, the status is parked and processed when the DMA completes; error statuses are never parked, so a lost DMA completion cannot hang the endpoint forever. If firmware’s TRANSFER_REQUEST arrives before the USB stack has even started the transfer, the event is saved and replayed from pipe_start.

Bulk and interrupt endpoints

One active transfer per endpoint, one parked behind it. Parked, not stalled: returning STALLED would send the stack into stall-recovery loops for what is just back-pressure. The invariant everywhere is reserve the message slot first, then the submission slot, so an SQ entry can never be orphaned without its announcement.

Two port fixes worth naming:

• Connect-status changes are computed as edge transitions from the previous cached port status. Firmware likes to report a spurious connect-change after reset, and uhub reads connect-change-after-reset as “the device vanished,” which detaches your keyboard mid-reset.
• After a root-hub port reset, any connect-change the port task set while the bus lock was dropped gets cleared for the same reason.

Endpoint pause and stall

Firmware can demand an endpoint pause. The handler flushes both submission queues, because firmware flushes its side and a pre-submitted IN buffer would desynchronize the resume. On resume, in-flight work is re-submitted, but only if the endpoint was actually firmware-paused, because firmware also sends an ACTIVE event right after endpoint creation, when the initial request is already on the wire and a second one would confuse it.

Lifecycle

Suspend tears the whole VHCI down and tells the mailbox SLEEP_NO_STATE; resume rebuilds it from scratch. Detach destroys the child first, stops the timestamp, tears down interrupts, then unwinds queues in reverse registration order. Every creation path in the driver has a full goto-ladder unwind, and command timeouts always attempt a firmware-side cancel before giving up.

Seeing it on hardware

kldload apple_bce
usbconfig                 # a new usbus with the internal keyboard
sysctl dev.bce_vhci

On bastion (Mac mini 8,1) this is the difference between a machine with a keyboard and a machine without one.