108 lines
15 KiB
Markdown
108 lines
15 KiB
Markdown
# ConnectX-5 (mlx5) — the product NIC candidate
|
||
|
||
## Status: retired — the X520 smoothed-bucket plan removes the need for this card
|
||
|
||
The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is **parked in a git stash** ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved.
|
||
|
||
**Known:**
|
||
|
||
- The two capabilities that decide the NIC — all-packet RX stamps and host-mastered module I2C *with writes* — both work. `/dev/fwctl` is read-only for MCIA (writes silently no-op under its restricted uctx); the mstflint PCICONF path (VSC → ICMD → register-access TLV, reimplemented in Go) does writes. Details below.
|
||
- The MCIA `module` field is **card-global** (module 0 = port 0, module 1 = port 1, per the PCI function digit) and must be set per port; unset it defaults to 0 and *both* ports read port 0 — a silent fake-dual-end bug the identical Wiitek vendor name hides and the serial numbers expose.
|
||
- **The RollBall mailbox is slow here.** Each register read waits for the module µC to flip the mailbox DONE bit, and that completion is **quantized**: ~50 ms (fast) or ~200 ms or ~350 ms — base + 0/1/2 steps of ~150 ms, never in between. A 7-register poll of both modules runs ~4–5 s, so SNR first appears ~5 s in and refreshes slowly. On the X520 the *same* µC answered in <25 ms, so this is imposed by the CX-5 side, not the module. The value trace is clean (uniform reads, µC holds `0x02` then flips to `0x04`), so it is not a transport stall, page clobber, or stale read.
|
||
- Slightly under line rate: ~0.03 Gb/s below the E810 on the same traffic. Host tuning (coalescing/rings) was never re-derived for mlx5, so this is probably untuned, not a ceiling.
|
||
|
||
**Not known:**
|
||
|
||
- **The source of the ~150 ms quantum.** Something periodic on the CX-5 side (firmware module monitoring? a driver poll? — no mlx5 thermal zone and hwmon module temp is on-demand, so a firmware task is the suspect) gates when the µC's result becomes committable; the mailbox completion phase-locks to it. Not confirmed, and not yet known whether it can be slowed/disabled (may be thermal-safety machinery).
|
||
- Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
|
||
- The MCIA `l` lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.
|
||
|
||
Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set:
|
||
|
||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports; the committed bucketing and `probe.go` run unchanged.
|
||
- **One hardware oscillator across both ports, but two PHC devices** — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
|
||
- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
|
||
- Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).
|
||
|
||
## PHC topology — measured
|
||
|
||
Each port is its own PCI function and exposes its own PHC (`mlx5_ptp` ×2), fed by **one shared oscillator**: both clocks drift against CLOCK_REALTIME at an identical rate (~59 ppm uncorrected), and their mutual offset moves only ~10 ppb (590 ns over 60 s — timecounter-update jitter, not oscillator difference). The offset itself is **quasi-static but nonzero** (~33 µs observed; set by each function's timecounter init). Consequences:
|
||
|
||
- Per-port rate bucketing is unaffected.
|
||
- Cross-port stamp comparison (one-way latency) needs the differential measured at startup — one `PTP_SYS_OFFSET` (`phc_ctl cmp`) per clock and a subtraction, good to sub-µs.
|
||
- This firmware (16.35.4506) has no real-time-clock NV option to collapse the two functions onto one hardware time.
|
||
|
||
## MCIA register format (verified in kernel headers)
|
||
|
||
The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope — MCIA's definition lives in NDA space; `struct mlx5_ifc_mcia_reg_bits` (`include/linux/mlx5/mlx5_ifc.h`) is the working reference:
|
||
|
||
- **`i2c_device_address` is a full 8-bit field** — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
|
||
- **8-bit `status` per access** — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
|
||
- **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested.
|
||
- **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.
|
||
|
||
## BCM SMI through MCIA — proven end-to-end on hardware
|
||
|
||
Every MCIA open question resolves in favor of the transport, live against the FS BCM in the CX-5's own cage (mstflint's `mstreg`; mstflint comes from the AUR, not the Arch repos):
|
||
|
||
- **Firmware accepts arbitrary I2C device addresses.** `i2c_device_address=0x56` (the BCM bridge) transacts with status 0 in both directions; kernel paths only ever use 0x50/0x51 but the firmware doesn't care.
|
||
- **MCIA writes emit exactly one offset byte**, so `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
|
||
- **The bridge tolerates MCIA's write-offset-then-read framing live** (the offset byte injected before the read phase is ignored, as the X520 bit-bang emulation predicted — `~/work/phydiag-work/bcm_mcia_emu.py`); PHY ID 0x3590 reads back correctly.
|
||
- **Handler mailbox ops work**: DATA1–5 writes, CMD write, STATUS poll (PASS observed), all through the same framing.
|
||
- RollBall is plain offset addressing at 0x51 — fits MCIA by construction.
|
||
|
||
The recipe (one SMI transaction = one MCIA SET, plus one MCIA GET for reads):
|
||
|
||
- SMI read of `devad.reg`: MCIA SET with `i2c_device_address=0x56, device_address=(0x20|devad), size=2, dword[0]=reg<<16` → MCIA GET with `device_address=0, size=2` → value in dword[0] bits 31:16. Respect the >1 ms bridge delay between the two.
|
||
- SMI write of `devad.reg=val`: MCIA SET with `device_address=devad, size=4, dword[0]=reg<<16|val`.
|
||
- Live example: `mstreg -d 01:00.1 --reg_name MCIA --set "dword[0]=0x00020000" --indexes "module=1,i2c_device_address=0x56,page_number=0,device_address=0x21,size=2" --yes` then the GET → 0x3590.
|
||
|
||
## How cabletest reaches MCIA in production (`phy_mcia.go`)
|
||
|
||
There are two host paths to MCIA, and the split is decisive:
|
||
|
||
- **`/dev/fwctl` (the kernel's sanctioned firmware back-channel) is read-only for MCIA.** ACCESS_REG reads work at scope `DEBUG_READ_ONLY`; **writes are silently dropped** — the RPC returns success at every layer (`errno=0`, command status 0, MCIA status 0) yet the register never changes, at every scope including `DEBUG_WRITE_FULL`. The firmware honours the ACCESS_REG opcode under fwctl's restricted user context (uid 2, caps `0x04000000`) but no-ops the actual I2C write. Proven in `~/work/phydiag-work/fwctl_mcia/diag`. Since a RollBall register **read** requires **writes** (set page/devad/command), fwctl cannot run RollBall at all.
|
||
- **The mstflint PCICONF path (VSC vendor capability → ICMD → FLASH_REG_ACCESS) can write**, so cabletest uses it. Per-dword gateway ops through the PCI config-space vendor capability reach ICMD space; a register-access command wrapped in operation+reg TLVs carries the MCIA transaction. Reimplemented in Go (`phy_mcia.go`), no mstflint dependency at runtime.
|
||
|
||
Two things that are not optional and one that is not needed:
|
||
|
||
- **The MCIA `module` field is card-global and must be set per port** (module 0 = port 0, module 1 = port 1, i.e. the PCI function digit), *regardless of which function issues the command* — function 1's ICMD with `module=0` reads port 0's module. It defaults to 0, so leaving it unset makes **both** ports read port 0: identical to a working dual-end read except the second end is a phantom of the first. It hid for a while because the vendor name is identical on both Wiiteks; the serial numbers (`…X256U` vs `…X039U`) exposed it. This was also the real cause of the SNR "coming and going": two functions hammering **one** module's I2C serialised into multi-second reads that blew the freshness window. Routed to their own modules, reads are prompt and the margin holds steady.
|
||
- **A process-wide lock holds each MCIA transaction whole.** The two ports drive independent per-function gateways but one shared card firmware; the lock keeps a command and its result from interleaving with the other port's. Not a retry, not a workaround — a plain mutex.
|
||
- The gateway cap9 semaphore is taken in one shot (this run is its only user; the driver reaches the firmware by its own BAR interface, not this config-space back-channel) and dropped at open if a `SIGKILL`ed prior run left it held. The ICMD-level semaphore mstflint also takes is **not** needed — the mutex plus the ICMD busy-bit check cover it, and taking it raced the driver. The MCIA `l` lock bit did nothing useful in testing.
|
||
|
||
There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.
|
||
|
||
## The FS cannot link on the CX-5 — host-interface deadlock
|
||
|
||
The FS's XFI never comes up against this firmware; **diagnostics to the FS work regardless** (all of the above ran with the link down). The mechanism, each side observed directly:
|
||
|
||
- **The firmware never enables the port's serdes TX before seeing module signal.** The port FSM sits in `ETH_AN_FSM_ENABLE`/Polling and SLTP reads all-zero TX parameters (the linked Wiitek port shows real ones), in force-10G mission mode and every other reachable mode.
|
||
- **The FS never enables its XFI TX before seeing valid host signal.** It asserts RX_LOS (the pin, mirrored in MCION, and A2h byte 110 bit 1) and its TX is electrically dark (RX eye grade 0 on the NIC) — while its copper side trains and stays up normally (7.33 shows master resolution, both receivers OK). PRBS31 and electrical-idle test patterns at 10.3125G do not clear the LOS: the gate wants valid 10GBASE-R, which only an active mission-mode port emits.
|
||
- Each side waits for the other; the link never starts. The Wiitek escapes because it transmits unconditionally (its port reaches Physical LinkUp immediately, then relays remote-fault from the FS's dead host side over the trained copper — so its netdev also stays down until the FS end resolves).
|
||
- ixgbe masks the FS behavior entirely: SFI idles are on the wire from driver load, so the FS side always has host signal first.
|
||
|
||
Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31/IDLE test-mode TX, PH and EX loopback, PMMP `eeprom_override` (accepted but reads back zero — not implemented for SFP on 16.35.4506), PMAOS module toggle (ICMD error 0x3), FS LED-controller rewrites (1.0xA83B/0xA83C do not source the LOS pin), FS PMA soft reset, FS KR mode enable + AN restarts ([../../modules/fs/README.md](../../modules/fs/README.md)).
|
||
|
||
**The deadlock is module-typed, not port-typed — confirmed by cage swap**: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).
|
||
|
||
Paths forward, in escalation order:
|
||
|
||
1. **Fibergaga + Wiitek as the test pair.** Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — [../../modules/fibergaga/README.md](../../modules/fibergaga/README.md)) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
|
||
2. **EEPROM surgery on the FS** (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.
|
||
|
||
## Arrival notes (resolved)
|
||
|
||
- Ports arrived in Ethernet mode (`enp1s0f*np*` netdevs from first boot); no `mlxconfig` LINK_TYPE change needed.
|
||
- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs.
|
||
- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
|
||
- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
|
||
|
||
## Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`)
|
||
|
||
Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
|
||
|
||
- **RX flow steering needs `rx-ntuple-filter` enabled first** (`ETHTOOL_SFEATURES`); mlx5 ships it off and reports filter capacity 0 and rejects inserts until it is on. It then advertises **no fixed filter table** (capacity 0) and rejects `RX_CLS_LOC_ANY` over the legacy `SIOCETHTOOL` ioctl, but accepts any **explicit** slot — so ethertype rules are placed at explicit locations allocated upward from 0 (ice/ixgbe allocate downward into their advertised table; the split is keyed on the advertised capacity).
|
||
- **Error counters**: the NIC-side rx error set is the PHY RMON categories `rx_{crc_errors,in_range_len_errors,out_of_range_len,undersize_pkts,oversize_pkts,fragments,jabbers}_phy`, mutually exclusive per frame. This is the NIC's view of the module↔NIC SFI link; the copper's own pre-FEC errors come from the module over RollBall, not here. Do **not** fold `tx_dropped` into cable-health — mlx5 counts millions of tx drops while the link is down at bringup.
|
||
- **Bringup forces a retrain** (AN restart) that drops the link for several seconds, so the run waits for carrier before baselining counters and starting traffic; otherwise the settling flap and tx-into-dark-link drops are charged to the fresh run.
|
||
- **PHC**: one oscillator, two PHC devices, ~33 µs quasi-static cross-port offset (see the PHC-topology section) — calibrate once at startup for cross-port latency.
|