cabletest/docs/nics/connectx-5

ConnectX-5 (mlx5) — retired

Status: retired — the X520 smoothed-bucket plan removes the need for this card

The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan (../x520/README.md) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR with the headline as worst pair across both ends, BCM+length dropped) ran on hardware — line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules — and its code is dropped; this file is the complete record (including the gateway constants below), with mstflint as the reimplementation reference. Everything below is what was proven and what was never resolved.

Known:

• The two capabilities that decide the NIC — all-packet RX stamps and host-mastered module I2C with writes — both work. /dev/fwctl is read-only for MCIA (writes silently no-op under its restricted uctx); the mstflint PCICONF path (VSC → ICMD → register-access TLV, reimplemented in Go) does writes. Details below.
• The MCIA module field is card-global (module 0 = port 0, module 1 = port 1, per the PCI function digit) and must be set per port; unset it defaults to 0 and both ports read port 0 — a silent fake-dual-end bug the identical Wiitek vendor name hides and the serial numbers expose.
• The RollBall mailbox is slow here. Each register read waits for the module µC to flip the mailbox DONE bit, and that completion is quantized: ~50 ms (fast) or ~200 ms or ~350 ms — base + 0/1/2 steps of ~150 ms, never in between. A 7-register poll of both modules runs ~4–5 s, so SNR first appears ~5 s in and refreshes slowly. On the X520 the same µC answered in <25 ms, so this is imposed by the CX-5 side, not the module. The value trace is clean (uniform reads, µC holds 0x02 then flips to 0x04), so it is not a transport stall, page clobber, or stale read.
• Slightly under line rate: ~0.03 Gb/s below the E810 on the same traffic. Host tuning (coalescing/rings) was never re-derived for mlx5, so this is probably untuned, not a ceiling.

Not known:

• The source of the ~150 ms quantum. Something periodic on the CX-5 side (firmware module monitoring? a driver poll? — no mlx5 thermal zone and hwmon module temp is on-demand, so a firmware task is the suspect) gates when the µC's result becomes committable; the mailbox completion phase-locks to it. Not confirmed, and not yet known whether it can be slowed/disabled (may be thermal-safety machinery).
• Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
• The MCIA l lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.

Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 was the one driver meeting the old requirement set (all-packet stamps + module-I2C writes):

• CQE-stamps every RX packet (HWTSTAMP_FILTER_ALL) — ethtool -T reports rx filter all on both ports.
• One hardware oscillator across both ports, but two PHC devices — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
• Native ETHER_FLOW ntuple steering — no driver patch, no encap needed.
• Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).

PHC topology — measured

Each port is its own PCI function and exposes its own PHC (mlx5_ptp ×2), fed by one shared oscillator: both clocks drift against CLOCK_REALTIME at an identical rate (~59 ppm uncorrected), and their mutual offset moves only ~10 ppb (590 ns over 60 s — timecounter-update jitter, not oscillator difference). The offset itself is quasi-static but nonzero (~33 µs observed; set by each function's timecounter init). Consequences:

• Per-port rate bucketing is unaffected.
• Cross-port stamp comparison (one-way latency) needs the differential measured at startup — one PTP_SYS_OFFSET (phc_ctl cmp) per clock and a subtraction, good to sub-µs.
• This firmware (16.35.4506) has no real-time-clock NV option to collapse the two functions onto one hardware time.

MCIA register format (verified in kernel headers)

The public PRM documents only the data path plus the ACCESS_REGISTER envelope — MCIA's definition lives in NDA space; struct mlx5_ifc_mcia_reg_bits (include/linux/mlx5/mlx5_ifc.h) is the working reference:

• i2c_device_address is a full 8-bit field — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
• 8-bit status per access — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
• Every read is offset-addressed: 16-bit device_address + 16-bit size, 12 data dwords (48 B max per transaction). There is no raw-read encoding — a read with no preceding offset cannot even be requested.
• l lock bit — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.

BCM SMI through MCIA — proven end-to-end on hardware

Every MCIA open question resolves in favor of the transport, live against the FS BCM in the CX-5's own cage (mstflint's mstreg; mstflint comes from the AUR, not the Arch repos):

• Firmware accepts arbitrary I2C device addresses. i2c_device_address=0x56 (the BCM bridge) transacts with status 0 in both directions; kernel paths only ever use 0x50/0x51 but the firmware doesn't care.
• MCIA writes emit exactly one offset byte, so device_address = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
• The bridge tolerates MCIA's write-offset-then-read framing live (the offset byte injected before the read phase is ignored, as the X520 bit-bang emulation predicted — ~/work/phydiag-work/bcm_mcia_emu.py); PHY ID 0x3590 reads back correctly.
• Handler mailbox ops work: DATA1–5 writes, CMD write, STATUS poll (PASS observed), all through the same framing.
• RollBall is plain offset addressing at 0x51 — fits MCIA by construction.

The recipe (one SMI transaction = one MCIA SET, plus one MCIA GET for reads):

• SMI read of devad.reg: MCIA SET with i2c_device_address=0x56, device_address=(0x20|devad), size=2, dword[0]=reg<<16 → MCIA GET with device_address=0, size=2 → value in dword[0] bits 31:16. Respect the >1 ms bridge delay between the two.
• SMI write of devad.reg=val: MCIA SET with device_address=devad, size=4, dword[0]=reg<<16|val.
• Live example: mstreg -d 01:00.1 --reg_name MCIA --set "dword[0]=0x00020000" --indexes "module=1,i2c_device_address=0x56,page_number=0,device_address=0x21,size=2" --yes then the GET → 0x3590.

How cabletest reaches MCIA in production (phy_mcia.go)

There are two host paths to MCIA, and the split is decisive:

• /dev/fwctl (the kernel's sanctioned firmware back-channel) is read-only for MCIA. ACCESS_REG reads work at scope DEBUG_READ_ONLY; writes are silently dropped — the RPC returns success at every layer (errno=0, command status 0, MCIA status 0) yet the register never changes, at every scope including DEBUG_WRITE_FULL. The firmware honours the ACCESS_REG opcode under fwctl's restricted user context (uid 2, caps 0x04000000) but no-ops the actual I2C write. Proven in ~/work/phydiag-work/fwctl_mcia/diag. Since a RollBall register read requires writes (set page/devad/command), fwctl cannot run RollBall at all.
• The mstflint PCICONF path (VSC vendor capability → ICMD → FLASH_REG_ACCESS) can write, so cabletest uses it. Per-dword gateway ops through the PCI config-space vendor capability reach ICMD space; a register-access command wrapped in operation+reg TLVs carries the MCIA transaction. Reimplemented in Go (phy_mcia.go), no mstflint dependency at runtime.

Two things that are not optional and one that is not needed:

• The MCIA module field is card-global and must be set per port (module 0 = port 0, module 1 = port 1, i.e. the PCI function digit), regardless of which function issues the command — function 1's ICMD with module=0 reads port 0's module. It defaults to 0, so leaving it unset makes both ports read port 0: identical to a working dual-end read except the second end is a phantom of the first. It hid for a while because the vendor name is identical on both Wiiteks; the serial numbers (…X256U vs …X039U) exposed it. This was also the real cause of the SNR "coming and going": two functions hammering one module's I2C serialised into multi-second reads that blew the freshness window. Routed to their own modules, reads are prompt and the margin holds steady.
• A process-wide lock holds each MCIA transaction whole. The two ports drive independent per-function gateways but one shared card firmware; the lock keeps a command and its result from interleaving with the other port's. Not a retry, not a workaround — a plain mutex.
• The gateway cap9 semaphore is taken in one shot (this run is its only user; the driver reaches the firmware by its own BAR interface, not this config-space back-channel) and dropped at open if a SIGKILLed prior run left it held. The ICMD-level semaphore mstflint also takes is not needed — the mutex plus the ICMD busy-bit check cover it, and taking it raced the driver. The MCIA l lock bit did nothing useful in testing.

There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.

PCICONF gateway constants (verified live; rebuildable without re-mining mstflint)

• Vendor-specific capability ID 0x09; offsets from the cap base: ctrl 0x4 (low 16 bits select the address space; a readback with bits 31:29 zero means the space is unsupported), counter 0x8, semaphore 0xC, addr 0x10, data 0x14.
• The semaphore is a ticket lock: free at 0; taken by writing the counter register's current value and reading it back unchanged. Address spaces: CR 0x2, ICMD 0x3.
• One dword op: for a write, data register then addr with bit 31 set; for a read, addr with bit 31 clear, then read data. The flag bit toggles inside the device's own access (clear = write done, set = read done) — always settled before the PCI round-trip that reads it back, so an unsettled flag is a fault, not a wait.
• ICMD space: ctrl at 0x0 (opcode bits 31:16, status bits 15:8, go bit 0 — set to fire, cleared by firmware on completion, observed < 5 ms), mailbox size at 0x1000, mailbox at 0x100000. CR-space 0xB5E04 bit 31 = firmware static config not done; check before any command.
• Command FLASH_REG_ACCESS 0x9001: operation TLV (type 1, 4 dwords — register ID at bytes 4–5, method byte 6 with GET=1/SET=2, class byte 7 = 1 for register access; completion status returns in byte 2 & 0x7F) + reg TLV (type 3, length in dwords) + register payload, all big-endian dwords in the mailbox.
• MCIA register ID 0x9014, 16-byte header + 48 data bytes: module at byte 1, per-access status at byte 3, 8-bit I2C device address at byte 4, device offset at byte 7, size at bytes 10–11, data from byte 16.

The FS cannot link on the CX-5 — host-interface deadlock

The FS's XFI never comes up against this firmware; diagnostics to the FS work regardless (all of the above ran with the link down). The mechanism, each side observed directly:

• The firmware never enables the port's serdes TX before seeing module signal. The port FSM sits in ETH_AN_FSM_ENABLE/Polling and SLTP reads all-zero TX parameters (the linked Wiitek port shows real ones), in force-10G mission mode and every other reachable mode.
• The FS never enables its XFI TX before seeing valid host signal. It asserts RX_LOS (the pin, mirrored in MCION, and A2h byte 110 bit 1) and its TX is electrically dark (RX eye grade 0 on the NIC) — while its copper side trains and stays up normally (7.33 shows master resolution, both receivers OK). PRBS31 and electrical-idle test patterns at 10.3125G do not clear the LOS: the gate wants valid 10GBASE-R, which only an active mission-mode port emits.
• Each side waits for the other; the link never starts. The Wiitek escapes because it transmits unconditionally (its port reaches Physical LinkUp immediately, then relays remote-fault from the FS's dead host side over the trained copper — so its netdev also stays down until the FS end resolves).
• ixgbe masks the FS behavior entirely: SFI idles are on the wire from driver load, so the FS side always has host signal first.

Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31/IDLE test-mode TX, PH and EX loopback, PMMP eeprom_override (accepted but reads back zero — not implemented for SFP on 16.35.4506), PMAOS module toggle (ICMD error 0x3), FS LED-controller rewrites (1.0xA83B/0xA83C do not source the LOS pin), FS PMA soft reset, FS KR mode enable + AN restarts (../../modules/fs/README.md).

The deadlock is module-typed, not port-typed — confirmed by cage swap: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).

Paths forward if this card ever returns, in escalation order:

1. Fibergaga + Wiitek as the test pair. Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — ../../modules/fibergaga/README.md) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
2. EEPROM surgery on the FS (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.

Arrival notes

• Ports arrived in Ethernet mode (enp1s0f*np* netdevs from first boot); no mlxconfig LINK_TYPE change needed.
• mlx5_ib autoloads; no channel-change interference observed so far (the irdma lesson, ../e810/README.md) — re-verify when cabletest's channel setup first runs.
• SFP28 autoneg does not settle with these modules; ethtool -s <dev> speed 10000 autoneg off is the working configuration on the module ports.
• Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).

Host-setup deltas from the ice/ixgbe path

Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:

• RX flow steering needs rx-ntuple-filter enabled first (ETHTOOL_SFEATURES); mlx5 ships it off and reports filter capacity 0 and rejects inserts until it is on. It then advertises no fixed filter table (capacity 0) and rejects RX_CLS_LOC_ANY over the legacy SIOCETHTOOL ioctl, but accepts any explicit slot — so ethertype rules are placed at explicit locations allocated upward from 0 (ice/ixgbe allocate downward into their advertised table; the split is keyed on the advertised capacity).
• Error counters: the NIC-side rx error set is the PHY RMON categories rx_{crc_errors,in_range_len_errors,out_of_range_len,undersize_pkts,oversize_pkts,fragments,jabbers}_phy, mutually exclusive per frame. This is the NIC's view of the module↔NIC SFI link; the copper's own pre-FEC errors come from the module over RollBall, not here. Do not fold tx_dropped into cable-health — mlx5 counts millions of tx drops while the link is down at bringup.
• Bringup forces a retrain (AN restart) that drops the link for several seconds, so the run waits for carrier before baselining counters and starting traffic; otherwise the settling flap and tx-into-dark-link drops are charged to the fresh run.
• PHC: one oscillator, two PHC devices, ~33 µs quasi-static cross-port offset (see the PHC-topology section) — calibrate once at startup for cross-port latency.