ConnectX-5 (mlx5) — the product NIC candidate

Status: retired — the X520 smoothed-bucket plan removes the need for this card

The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan (../x520/README.md) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is parked in a git stash ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved.

Known:

The two capabilities that decide the NIC — all-packet RX stamps and host-mastered module I2C with writes — both work. /dev/fwctl is read-only for MCIA (writes silently no-op under its restricted uctx); the mstflint PCICONF path (VSC → ICMD → register-access TLV, reimplemented in Go) does writes. Details below.
The MCIA module field is card-global (module 0 = port 0, module 1 = port 1, per the PCI function digit) and must be set per port; unset it defaults to 0 and both ports read port 0 — a silent fake-dual-end bug the identical Wiitek vendor name hides and the serial numbers expose.
The RollBall mailbox is slow here. Each register read waits for the module µC to flip the mailbox DONE bit, and that completion is quantized: ~50 ms (fast) or ~200 ms or ~350 ms — base + 0/1/2 steps of ~150 ms, never in between. A 7-register poll of both modules runs ~4–5 s, so SNR first appears ~5 s in and refreshes slowly. On the X520 the same µC answered in <25 ms, so this is imposed by the CX-5 side, not the module. The value trace is clean (uniform reads, µC holds 0x02 then flips to 0x04), so it is not a transport stall, page clobber, or stale read.
Slightly under line rate: ~0.03 Gb/s below the E810 on the same traffic. Host tuning (coalescing/rings) was never re-derived for mlx5, so this is probably untuned, not a ceiling.

Not known:

The source of the ~150 ms quantum. Something periodic on the CX-5 side (firmware module monitoring? a driver poll? — no mlx5 thermal zone and hwmon module temp is on-demand, so a firmware task is the suspect) gates when the µC's result becomes committable; the mailbox completion phase-locks to it. Not confirmed, and not yet known whether it can be slowed/disabled (may be thermal-safety machinery).
Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
The MCIA l lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.

Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set:

CQE-stamps every RX packet (HWTSTAMP_FILTER_ALL) — ethtool -T reports rx filter all on both ports; the committed bucketing and probe.go run unchanged.
One hardware oscillator across both ports, but two PHC devices — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
Native ETHER_FLOW ntuple steering — no driver patch, no encap needed.
Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).

PHC topology — measured

Each port is its own PCI function and exposes its own PHC (mlx5_ptp ×2), fed by one shared oscillator: both clocks drift against CLOCK_REALTIME at an identical rate (~59 ppm uncorrected), and their mutual offset moves only ~10 ppb (590 ns over 60 s — timecounter-update jitter, not oscillator difference). The offset itself is quasi-static but nonzero (~33 µs observed; set by each function's timecounter init). Consequences:

Per-port rate bucketing is unaffected.
Cross-port stamp comparison (one-way latency) needs the differential measured at startup — one PTP_SYS_OFFSET (phc_ctl cmp) per clock and a subtraction, good to sub-µs.
This firmware (16.35.4506) has no real-time-clock NV option to collapse the two functions onto one hardware time.

MCIA register format (verified in kernel headers)

The public PRM documents only the data path plus the ACCESS_REGISTER envelope — MCIA's definition lives in NDA space; struct mlx5_ifc_mcia_reg_bits (include/linux/mlx5/mlx5_ifc.h) is the working reference:

i2c_device_address is a full 8-bit field — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
8-bit status per access — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
Every read is offset-addressed: 16-bit device_address + 16-bit size, 12 data dwords (48 B max per transaction). There is no raw-read encoding — a read with no preceding offset cannot even be requested.
l lock bit — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.

BCM SMI through MCIA — proven end-to-end on hardware

Every MCIA open question resolves in favor of the transport, live against the FS BCM in the CX-5's own cage (mstflint's mstreg; mstflint comes from the AUR, not the Arch repos):

Firmware accepts arbitrary I2C device addresses. i2c_device_address=0x56 (the BCM bridge) transacts with status 0 in both directions; kernel paths only ever use 0x50/0x51 but the firmware doesn't care.
MCIA writes emit exactly one offset byte, so device_address = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
The bridge tolerates MCIA's write-offset-then-read framing live (the offset byte injected before the read phase is ignored, as the X520 bit-bang emulation predicted — ~/work/phydiag-work/bcm_mcia_emu.py); PHY ID 0x3590 reads back correctly.
Handler mailbox ops work: DATA1–5 writes, CMD write, STATUS poll (PASS observed), all through the same framing.
RollBall is plain offset addressing at 0x51 — fits MCIA by construction.

The recipe (one SMI transaction = one MCIA SET, plus one MCIA GET for reads):

SMI read of devad.reg: MCIA SET with i2c_device_address=0x56, device_address=(0x20|devad), size=2, dword[0]=reg<<16 → MCIA GET with device_address=0, size=2 → value in dword[0] bits 31:16. Respect the >1 ms bridge delay between the two.
SMI write of devad.reg=val: MCIA SET with device_address=devad, size=4, dword[0]=reg<<16|val.
Live example: mstreg -d 01:00.1 --reg_name MCIA --set "dword[0]=0x00020000" --indexes "module=1,i2c_device_address=0x56,page_number=0,device_address=0x21,size=2" --yes then the GET → 0x3590.

How cabletest reaches MCIA in production (`phy_mcia.go`)

There are two host paths to MCIA, and the split is decisive:

/dev/fwctl (the kernel's sanctioned firmware back-channel) is read-only for MCIA. ACCESS_REG reads work at scope DEBUG_READ_ONLY; writes are silently dropped — the RPC returns success at every layer (errno=0, command status 0, MCIA status 0) yet the register never changes, at every scope including DEBUG_WRITE_FULL. The firmware honours the ACCESS_REG opcode under fwctl's restricted user context (uid 2, caps 0x04000000) but no-ops the actual I2C write. Proven in ~/work/phydiag-work/fwctl_mcia/diag. Since a RollBall register read requires writes (set page/devad/command), fwctl cannot run RollBall at all.
The mstflint PCICONF path (VSC vendor capability → ICMD → FLASH_REG_ACCESS) can write, so cabletest uses it. Per-dword gateway ops through the PCI config-space vendor capability reach ICMD space; a register-access command wrapped in operation+reg TLVs carries the MCIA transaction. Reimplemented in Go (phy_mcia.go), no mstflint dependency at runtime.

Two things that are not optional and one that is not needed:

The MCIA module field is card-global and must be set per port (module 0 = port 0, module 1 = port 1, i.e. the PCI function digit), regardless of which function issues the command — function 1's ICMD with module=0 reads port 0's module. It defaults to 0, so leaving it unset makes both ports read port 0: identical to a working dual-end read except the second end is a phantom of the first. It hid for a while because the vendor name is identical on both Wiiteks; the serial numbers (…X256U vs …X039U) exposed it. This was also the real cause of the SNR "coming and going": two functions hammering one module's I2C serialised into multi-second reads that blew the freshness window. Routed to their own modules, reads are prompt and the margin holds steady.
A process-wide lock holds each MCIA transaction whole. The two ports drive independent per-function gateways but one shared card firmware; the lock keeps a command and its result from interleaving with the other port's. Not a retry, not a workaround — a plain mutex.
The gateway cap9 semaphore is taken in one shot (this run is its only user; the driver reaches the firmware by its own BAR interface, not this config-space back-channel) and dropped at open if a SIGKILLed prior run left it held. The ICMD-level semaphore mstflint also takes is not needed — the mutex plus the ICMD busy-bit check cover it, and taking it raced the driver. The MCIA l lock bit did nothing useful in testing.

There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.

The FS cannot link on the CX-5 — host-interface deadlock

The FS's XFI never comes up against this firmware; diagnostics to the FS work regardless (all of the above ran with the link down). The mechanism, each side observed directly:

The firmware never enables the port's serdes TX before seeing module signal. The port FSM sits in ETH_AN_FSM_ENABLE/Polling and SLTP reads all-zero TX parameters (the linked Wiitek port shows real ones), in force-10G mission mode and every other reachable mode.
The FS never enables its XFI TX before seeing valid host signal. It asserts RX_LOS (the pin, mirrored in MCION, and A2h byte 110 bit 1) and its TX is electrically dark (RX eye grade 0 on the NIC) — while its copper side trains and stays up normally (7.33 shows master resolution, both receivers OK). PRBS31 and electrical-idle test patterns at 10.3125G do not clear the LOS: the gate wants valid 10GBASE-R, which only an active mission-mode port emits.
Each side waits for the other; the link never starts. The Wiitek escapes because it transmits unconditionally (its port reaches Physical LinkUp immediately, then relays remote-fault from the FS's dead host side over the trained copper — so its netdev also stays down until the FS end resolves).
ixgbe masks the FS behavior entirely: SFI idles are on the wire from driver load, so the FS side always has host signal first.

Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31/IDLE test-mode TX, PH and EX loopback, PMMP eeprom_override (accepted but reads back zero — not implemented for SFP on 16.35.4506), PMAOS module toggle (ICMD error 0x3), FS LED-controller rewrites (1.0xA83B/0xA83C do not source the LOS pin), FS PMA soft reset, FS KR mode enable + AN restarts (../../modules/fs/README.md).

The deadlock is module-typed, not port-typed — confirmed by cage swap: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).

Paths forward, in escalation order:

Fibergaga + Wiitek as the test pair. Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — ../../modules/fibergaga/README.md) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
EEPROM surgery on the FS (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.

Arrival notes (resolved)

Ports arrived in Ethernet mode (enp1s0f*np* netdevs from first boot); no mlxconfig LINK_TYPE change needed.
mlx5_ib autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs.
SFP28 autoneg does not settle with these modules; ethtool -s <dev> speed 10000 autoneg off is the working configuration on the module ports.
Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).

Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`)

Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:

RX flow steering needs rx-ntuple-filter enabled first (ETHTOOL_SFEATURES); mlx5 ships it off and reports filter capacity 0 and rejects inserts until it is on. It then advertises no fixed filter table (capacity 0) and rejects RX_CLS_LOC_ANY over the legacy SIOCETHTOOL ioctl, but accepts any explicit slot — so ethertype rules are placed at explicit locations allocated upward from 0 (ice/ixgbe allocate downward into their advertised table; the split is keyed on the advertised capacity).
Error counters: the NIC-side rx error set is the PHY RMON categories rx_{crc_errors,in_range_len_errors,out_of_range_len,undersize_pkts,oversize_pkts,fragments,jabbers}_phy, mutually exclusive per frame. This is the NIC's view of the module↔NIC SFI link; the copper's own pre-FEC errors come from the module over RollBall, not here. Do not fold tx_dropped into cable-health — mlx5 counts millions of tx drops while the link is down at bringup.
Bringup forces a retrain (AN restart) that drops the link for several seconds, so the run waits for carrier before baselining counters and starting traffic; otherwise the settling flap and tx-into-dark-link drops are charged to the fresh run.
PHC: one oscillator, two PHC devices, ~33 µs quasi-static cross-port offset (see the PHC-topology section) — calibrate once at startup for cross-port latency.

15 KiB Raw Blame History Unescape Escape