X520 returns as product NIC: smoothed-bucket rate plan (read-time buckets, backward excess-fill, rate as headline not verdict) dissolves the per-frame RX timestamp requirement; CX-5 retired with full record (MCIA via VSC/ICMD proven, fwctl write-block, card-global module index, ~150ms mailbox quantum unexplained, FS link deadlock); E810 below-AQ bit-bang idea recorded and retired; FS+Wiitek pair and length return as goals

docs/nics/connectx-5/README.md

@@ -1,11 +1,36 @@
 # ConnectX-5 (mlx5) — the product NIC candidate
 
-Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
+## Status: retired — the X520 smoothed-bucket plan removes the need for this card
 
-- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — the committed bucketing and `probe.go` run unchanged.
-- **One shared clock across both ports** — `probe.go`'s shared-PHC assumption holds.
+The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is **parked in a git stash** ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved.
+
+**Known:**
+
+- The two capabilities that decide the NIC — all-packet RX stamps and host-mastered module I2C *with writes* — both work. `/dev/fwctl` is read-only for MCIA (writes silently no-op under its restricted uctx); the mstflint PCICONF path (VSC → ICMD → register-access TLV, reimplemented in Go) does writes. Details below.
+- The MCIA `module` field is **card-global** (module 0 = port 0, module 1 = port 1, per the PCI function digit) and must be set per port; unset it defaults to 0 and *both* ports read port 0 — a silent fake-dual-end bug the identical Wiitek vendor name hides and the serial numbers expose.
+- **The RollBall mailbox is slow here.** Each register read waits for the module µC to flip the mailbox DONE bit, and that completion is **quantized**: ~50 ms (fast) or ~200 ms or ~350 ms — base + 0/1/2 steps of ~150 ms, never in between. A 7-register poll of both modules runs ~4–5 s, so SNR first appears ~5 s in and refreshes slowly. On the X520 the *same* µC answered in <25 ms, so this is imposed by the CX-5 side, not the module. The value trace is clean (uniform reads, µC holds `0x02` then flips to `0x04`), so it is not a transport stall, page clobber, or stale read.
+- Slightly under line rate: ~0.03 Gb/s below the E810 on the same traffic. Host tuning (coalescing/rings) was never re-derived for mlx5, so this is probably untuned, not a ceiling.
+
+**Not known:**
+
+- **The source of the ~150 ms quantum.** Something periodic on the CX-5 side (firmware module monitoring? a driver poll? — no mlx5 thermal zone and hwmon module temp is on-demand, so a firmware task is the suspect) gates when the µC's result becomes committable; the mailbox completion phase-locks to it. Not confirmed, and not yet known whether it can be slowed/disabled (may be thermal-safety machinery).
+- Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
+- The MCIA `l` lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.
+
+Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set:
+
+- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports; the committed bucketing and `probe.go` run unchanged.
+- **One hardware oscillator across both ports, but two PHC devices** — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
 - **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
-- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
+- Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).
+
+## PHC topology — measured
+
+Each port is its own PCI function and exposes its own PHC (`mlx5_ptp` ×2), fed by **one shared oscillator**: both clocks drift against CLOCK_REALTIME at an identical rate (~59 ppm uncorrected), and their mutual offset moves only ~10 ppb (590 ns over 60 s — timecounter-update jitter, not oscillator difference). The offset itself is **quasi-static but nonzero** (~33 µs observed; set by each function's timecounter init). Consequences:
+
+- Per-port rate bucketing is unaffected.
+- Cross-port stamp comparison (one-way latency) needs the differential measured at startup — one `PTP_SYS_OFFSET` (`phc_ctl cmp`) per clock and a subtraction, good to sub-µs.
+- This firmware (16.35.4506) has no real-time-clock NV option to collapse the two functions onto one hardware time.
 
 ## MCIA register format (verified in kernel headers)
 
@@ -16,27 +41,67 @@ The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope
 - **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested.
 - **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.
 
-## BCM SMI through MCIA framing — module side proven
+## BCM SMI through MCIA — proven end-to-end on hardware
 
-MCIA has no raw-read encoding (every read emits an offset write first), so SMI reads work only if the BCM bridge tolerates that framing. **It does — proven on hardware** (X520 bit-bang emulation of MCIA's write-offset-then-read, `~/work/phydiag-work/bcm_mcia_emu.py` against the FS BCM):
+Every MCIA open question resolves in favor of the transport, live against the FS BCM in the CX-5's own cage (mstflint's `mstreg`; mstflint comes from the AUR, not the Arch repos):
 
-- A 1-byte offset write (0x00 and 0xFF tried) injected between SMI command and read phase leaves the pending read data intact — immediately and after 3 ms; the expected register value (PHY ID 0x3590/0x5081) reads back correctly every time.
-- A cold offset-write+read returns the last-latched SMI data — the offset byte is ignored, not parsed as a new command frame.
-- Normal SMI operation recovers cleanly afterward; no lasting disturbance.
-- Caveat: emulated with STOP-separated transactions; a repeated-START combined transfer (if that's what firmware emits) is untested.
+- **Firmware accepts arbitrary I2C device addresses.** `i2c_device_address=0x56` (the BCM bridge) transacts with status 0 in both directions; kernel paths only ever use 0x50/0x51 but the firmware doesn't care.
+- **MCIA writes emit exactly one offset byte**, so `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
+- **The bridge tolerates MCIA's write-offset-then-read framing live** (the offset byte injected before the read phase is ignored, as the X520 bit-bang emulation predicted — `~/work/phydiag-work/bcm_mcia_emu.py`); PHY ID 0x3590 reads back correctly.
+- **Handler mailbox ops work**: DATA1–5 writes, CMD write, STATUS poll (PASS observed), all through the same framing.
+- RollBall is plain offset addressing at 0x51 — fits MCIA by construction.
 
-SMI *writes* fit MCIA's model directly if firmware emits a 1-byte offset: an MCIA write emits `[offset][data…]` as one I2C write, so choosing `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
+The recipe (one SMI transaction = one MCIA SET, plus one MCIA GET for reads):
 
-## Open questions (decide on arrival)
+- SMI read of `devad.reg`: MCIA SET with `i2c_device_address=0x56, device_address=(0x20|devad), size=2, dword[0]=reg<<16` → MCIA GET with `device_address=0, size=2` → value in dword[0] bits 31:16. Respect the >1 ms bridge delay between the two.
+- SMI write of `devad.reg=val`: MCIA SET with `device_address=devad, size=4, dword[0]=reg<<16|val`.
+- Live example: `mstreg -d 01:00.1 --reg_name MCIA --set "dword[0]=0x00020000" --indexes "module=1,i2c_device_address=0x56,page_number=0,device_address=0x21,size=2" --yes` then the GET → 0x3590.
 
-- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (`MLX5_I2C_ADDR_LOW/HIGH`). The register format allows it and the per-access status byte makes rejection observable.
-- Whether MCIA writes emit exactly **one offset byte** (needed for the SMI-frame synthesis above), and whether reads use STOP-separated or repeated-START framing.
-- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
-- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
+## How cabletest reaches MCIA in production (`phy_mcia.go`)
 
-## Arrival notes
+There are two host paths to MCIA, and the split is decisive:
 
-- Ports may ship in InfiniBand mode: `mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2`, reboot.
-- Check whether auto-loading `mlx5_ib` interferes with channel changes (the irdma lesson, hardware.md).
-- Force 10G if SFP28 autoneg is stubborn: `ethtool -s <dev> speed 10000 autoneg off`.
-- Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.
+- **`/dev/fwctl` (the kernel's sanctioned firmware back-channel) is read-only for MCIA.** ACCESS_REG reads work at scope `DEBUG_READ_ONLY`; **writes are silently dropped** — the RPC returns success at every layer (`errno=0`, command status 0, MCIA status 0) yet the register never changes, at every scope including `DEBUG_WRITE_FULL`. The firmware honours the ACCESS_REG opcode under fwctl's restricted user context (uid 2, caps `0x04000000`) but no-ops the actual I2C write. Proven in `~/work/phydiag-work/fwctl_mcia/diag`. Since a RollBall register **read** requires **writes** (set page/devad/command), fwctl cannot run RollBall at all.
+- **The mstflint PCICONF path (VSC vendor capability → ICMD → FLASH_REG_ACCESS) can write**, so cabletest uses it. Per-dword gateway ops through the PCI config-space vendor capability reach ICMD space; a register-access command wrapped in operation+reg TLVs carries the MCIA transaction. Reimplemented in Go (`phy_mcia.go`), no mstflint dependency at runtime.
+
+Two things that are not optional and one that is not needed:
+
+- **The MCIA `module` field is card-global and must be set per port** (module 0 = port 0, module 1 = port 1, i.e. the PCI function digit), *regardless of which function issues the command* — function 1's ICMD with `module=0` reads port 0's module. It defaults to 0, so leaving it unset makes **both** ports read port 0: identical to a working dual-end read except the second end is a phantom of the first. It hid for a while because the vendor name is identical on both Wiiteks; the serial numbers (`…X256U` vs `…X039U`) exposed it. This was also the real cause of the SNR "coming and going": two functions hammering **one** module's I2C serialised into multi-second reads that blew the freshness window. Routed to their own modules, reads are prompt and the margin holds steady.
+- **A process-wide lock holds each MCIA transaction whole.** The two ports drive independent per-function gateways but one shared card firmware; the lock keeps a command and its result from interleaving with the other port's. Not a retry, not a workaround — a plain mutex.
+- The gateway cap9 semaphore is taken in one shot (this run is its only user; the driver reaches the firmware by its own BAR interface, not this config-space back-channel) and dropped at open if a `SIGKILL`ed prior run left it held. The ICMD-level semaphore mstflint also takes is **not** needed — the mutex plus the ICMD busy-bit check cover it, and taking it raced the driver. The MCIA `l` lock bit did nothing useful in testing.
+
+There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.
+
+## The FS cannot link on the CX-5 — host-interface deadlock
+
+The FS's XFI never comes up against this firmware; **diagnostics to the FS work regardless** (all of the above ran with the link down). The mechanism, each side observed directly:
+
+- **The firmware never enables the port's serdes TX before seeing module signal.** The port FSM sits in `ETH_AN_FSM_ENABLE`/Polling and SLTP reads all-zero TX parameters (the linked Wiitek port shows real ones), in force-10G mission mode and every other reachable mode.
+- **The FS never enables its XFI TX before seeing valid host signal.** It asserts RX_LOS (the pin, mirrored in MCION, and A2h byte 110 bit 1) and its TX is electrically dark (RX eye grade 0 on the NIC) — while its copper side trains and stays up normally (7.33 shows master resolution, both receivers OK). PRBS31 and electrical-idle test patterns at 10.3125G do not clear the LOS: the gate wants valid 10GBASE-R, which only an active mission-mode port emits.
+- Each side waits for the other; the link never starts. The Wiitek escapes because it transmits unconditionally (its port reaches Physical LinkUp immediately, then relays remote-fault from the FS's dead host side over the trained copper — so its netdev also stays down until the FS end resolves).
+- ixgbe masks the FS behavior entirely: SFI idles are on the wire from driver load, so the FS side always has host signal first.
+
+Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31/IDLE test-mode TX, PH and EX loopback, PMMP `eeprom_override` (accepted but reads back zero — not implemented for SFP on 16.35.4506), PMAOS module toggle (ICMD error 0x3), FS LED-controller rewrites (1.0xA83B/0xA83C do not source the LOS pin), FS PMA soft reset, FS KR mode enable + AN restarts ([../../modules/fs/README.md](../../modules/fs/README.md)).
+
+**The deadlock is module-typed, not port-typed — confirmed by cage swap**: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).
+
+Paths forward, in escalation order:
+
+1. **Fibergaga + Wiitek as the test pair.** Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — [../../modules/fibergaga/README.md](../../modules/fibergaga/README.md)) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
+2. **EEPROM surgery on the FS** (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.
+
+## Arrival notes (resolved)
+
+- Ports arrived in Ethernet mode (`enp1s0f*np*` netdevs from first boot); no `mlxconfig` LINK_TYPE change needed.
+- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs.
+- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
+- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
+
+## Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`)
+
+Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
+
+- **RX flow steering needs `rx-ntuple-filter` enabled first** (`ETHTOOL_SFEATURES`); mlx5 ships it off and reports filter capacity 0 and rejects inserts until it is on. It then advertises **no fixed filter table** (capacity 0) and rejects `RX_CLS_LOC_ANY` over the legacy `SIOCETHTOOL` ioctl, but accepts any **explicit** slot — so ethertype rules are placed at explicit locations allocated upward from 0 (ice/ixgbe allocate downward into their advertised table; the split is keyed on the advertised capacity).
+- **Error counters**: the NIC-side rx error set is the PHY RMON categories `rx_{crc_errors,in_range_len_errors,out_of_range_len,undersize_pkts,oversize_pkts,fragments,jabbers}_phy`, mutually exclusive per frame. This is the NIC's view of the module↔NIC SFI link; the copper's own pre-FEC errors come from the module over RollBall, not here. Do **not** fold `tx_dropped` into cable-health — mlx5 counts millions of tx drops while the link is down at bringup.
+- **Bringup forces a retrain** (AN restart) that drops the link for several seconds, so the run waits for carrier before baselining counters and starting traffic; otherwise the settling flap and tx-into-dark-link drops are charged to the fresh run.
+- **PHC**: one oscillator, two PHC devices, ~33 µs quasi-static cross-port offset (see the PHC-topology section) — calibrate once at startup for cross-port latency.