X520 returns as product NIC: smoothed-bucket rate plan (read-time buckets, backward excess-fill, rate as headline not verdict) dissolves the per-frame RX timestamp requirement; CX-5 retired with full record (MCIA via VSC/ICMD proven, fwctl write-block, card-global module index, ~150ms mailbox quantum unexplained, FS link deadlock); E810 below-AQ bit-bang idea recorded and retired; FS+Wiitek pair and length return as goals

docs/modules/fs/README.md

@@ -16,6 +16,16 @@ PHY at I2C 8-bit **0xAC write / 0xAD read**.
 - Single-byte reads see the bridge as inert (returns 0) — the DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead. Treat 0x0000 reads as retry-with-longer-delay: 0 is also the bridge's not-ready signature.
 - Recorded for a future multi-byte firmware transport (mlx5 MCIA shape): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`.
 
+## Host interface (XFI) bring-up gating
+
+**The module never enables its XFI TX toward the host until it receives valid host signal** — observed live on the ConnectX-5, where it produces a mutual-wait deadlock ([../../nics/connectx-5/README.md](../../nics/connectx-5/README.md)):
+
+- While ungated it asserts RX_LOS (the SFP pin and A2h byte 110 bit 1 both) and its host-side TX is electrically dark (NIC RX eye grade 0). The copper side is independent: it trains and holds link normally the whole time, so RX_LOS here means "no host signal", not "no copper link".
+- The gate is PCS-qualified, not analog: PRBS31 and electrical-idle patterns at 10.3125G into the module do not clear it — only valid 10GBASE-R does. Hosts that emit SFI idles from driver load (ixgbe) never see any of this.
+- With its host side dead and copper trained, the module signals fault over the copper; the far-end module relays it to its own host as remote-fault ordered sets (the far end's tell that this end's host interface is down).
+- Not a path to the gate: PMA soft reset (1.0.15, self-clearing — copper retrains, host TX stays dark), KR mode (SET 0x800F PASSes, no host TX with or without AN restart; GET 0x800E is a DATA1-answer readback, unusable), the LED controller (1.0xA83B = 0x8400, 1.0xA83C = 0x00E1; rewriting either does not move the RX_LOS pin — the pin is not LED-sourced). All of it survived un-bricked, values restored.
+- A0h byte 36 = 0x1C (10GBASE-T Short Reach) is the honest class declaration; power is not the issue (A0h byte 64 = 0x00 declares no high-power-level mechanism; A2h 92–94 are filler bytes).
+
 ## MDIO command handler
 
 Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-commands.md). Traps:

docs/nics/README.md

@@ -1,10 +1,14 @@
 # NICs
 
-The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path gets the hardware timestamps it requires. Per-device directories hold the details.
+The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path can report an honest rate. Per-device directories hold the details.
 
 ## The transport rule
 
-A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols.
+A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols (E810: writes blocked outright; CX-5: writes work but the firmware's own module traffic quantizes every mailbox read to ~150 ms steps).
+
+## The timestamp requirement is dissolved
+
+All-packet hardware RX timestamps existed for one consumer — bucketing received frames by arrival time so the displayed rate stayed honest under host read-jitter. The smoothed-bucket plan ([x520](x520/README.md)) computes the rate from read-time buckets with a backward excess-fill instead, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. With that, the capability that disqualified the X520 — and the only thing the firmware-managed NICs offered over it — is no longer required (pending the audit that the rate buckets were the stamps' sole consumer). Raw-L2 steering was never a hard requirement either (encap acceptable).
 
 ## Comparison
 
@@ -12,14 +16,12 @@ Which card is where: [../state.md](../state.md).
 
 | NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict |
 |---|---|---|---|
-| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (single serial latch; 1588-event frames only, EtherType configurable) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC |
-| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set |
-| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform |
-| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes |
-
-Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable). Full analyses live in each device directory.
+| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven**, fast (<25 ms RollBall) | No (single serial latch; 1588-event frames only) — no longer required | **The product NIC** under the smoothed-bucket rate plan: only card with unmediated host I2C, and the FS links here (mixed FS+Wiitek pair proven) |
+| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA — writes via VSC/ICMD only (fwctl read-only), every mailbox read quantized to ~150 ms steps by firmware module traffic | Yes (CQE-stamps everything) | Retired: its stamps are no longer needed and its I2C mediation is the cost; the FS never links on it either (host-interface deadlock, device page) |
+| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; noise pair |
+| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes; the below-the-AQ bit-bang idea (device page) is retired with the timestamp requirement |
 
 ## Architecture
 
 - **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
-- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules only ([connectx-5](connectx-5/README.md)).
+- **The single card is the X520.** With the timestamp requirement dissolved, the selection weights return to where the X520 originally won: unmediated host I2C is the one capability no firmware-managed card provides (E810 blocks writes; mlx5 mediates and quantizes), and it is non-negotiable for the module protocols. Gen2 ×8 (32 Gb/s raw) clears 2×10G full duplex; ETQF steering and loss attribution are proven on hardware. The general lesson, learned twice: NICs that stamp every packet are all firmware-managed, and firmware in the module-I2C path is what breaks or slows the diagnostics — the two headline capabilities appear to be mutually exclusive off the shelf.

docs/nics/connectx-5/README.md

@@ -1,11 +1,36 @@
 # ConnectX-5 (mlx5) — the product NIC candidate
 
-Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
+## Status: retired — the X520 smoothed-bucket plan removes the need for this card
 
-- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — the committed bucketing and `probe.go` run unchanged.
-- **One shared clock across both ports** — `probe.go`'s shared-PHC assumption holds.
+The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is **parked in a git stash** ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved.
+
+**Known:**
+
+- The two capabilities that decide the NIC — all-packet RX stamps and host-mastered module I2C *with writes* — both work. `/dev/fwctl` is read-only for MCIA (writes silently no-op under its restricted uctx); the mstflint PCICONF path (VSC → ICMD → register-access TLV, reimplemented in Go) does writes. Details below.
+- The MCIA `module` field is **card-global** (module 0 = port 0, module 1 = port 1, per the PCI function digit) and must be set per port; unset it defaults to 0 and *both* ports read port 0 — a silent fake-dual-end bug the identical Wiitek vendor name hides and the serial numbers expose.
+- **The RollBall mailbox is slow here.** Each register read waits for the module µC to flip the mailbox DONE bit, and that completion is **quantized**: ~50 ms (fast) or ~200 ms or ~350 ms — base + 0/1/2 steps of ~150 ms, never in between. A 7-register poll of both modules runs ~4–5 s, so SNR first appears ~5 s in and refreshes slowly. On the X520 the *same* µC answered in <25 ms, so this is imposed by the CX-5 side, not the module. The value trace is clean (uniform reads, µC holds `0x02` then flips to `0x04`), so it is not a transport stall, page clobber, or stale read.
+- Slightly under line rate: ~0.03 Gb/s below the E810 on the same traffic. Host tuning (coalescing/rings) was never re-derived for mlx5, so this is probably untuned, not a ceiling.
+
+**Not known:**
+
+- **The source of the ~150 ms quantum.** Something periodic on the CX-5 side (firmware module monitoring? a driver poll? — no mlx5 thermal zone and hwmon module temp is on-demand, so a firmware task is the suspect) gates when the µC's result becomes committable; the mailbox completion phase-locks to it. Not confirmed, and not yet known whether it can be slowed/disabled (may be thermal-safety machinery).
+- Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
+- The MCIA `l` lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.
+
+Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set:
+
+- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports; the committed bucketing and `probe.go` run unchanged.
+- **One hardware oscillator across both ports, but two PHC devices** — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
 - **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
-- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
+- Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).
+
+## PHC topology — measured
+
+Each port is its own PCI function and exposes its own PHC (`mlx5_ptp` ×2), fed by **one shared oscillator**: both clocks drift against CLOCK_REALTIME at an identical rate (~59 ppm uncorrected), and their mutual offset moves only ~10 ppb (590 ns over 60 s — timecounter-update jitter, not oscillator difference). The offset itself is **quasi-static but nonzero** (~33 µs observed; set by each function's timecounter init). Consequences:
+
+- Per-port rate bucketing is unaffected.
+- Cross-port stamp comparison (one-way latency) needs the differential measured at startup — one `PTP_SYS_OFFSET` (`phc_ctl cmp`) per clock and a subtraction, good to sub-µs.
+- This firmware (16.35.4506) has no real-time-clock NV option to collapse the two functions onto one hardware time.
 
 ## MCIA register format (verified in kernel headers)
 
@@ -16,27 +41,67 @@ The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope
 - **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested.
 - **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.
 
-## BCM SMI through MCIA framing — module side proven
+## BCM SMI through MCIA — proven end-to-end on hardware
 
-MCIA has no raw-read encoding (every read emits an offset write first), so SMI reads work only if the BCM bridge tolerates that framing. **It does — proven on hardware** (X520 bit-bang emulation of MCIA's write-offset-then-read, `~/work/phydiag-work/bcm_mcia_emu.py` against the FS BCM):
+Every MCIA open question resolves in favor of the transport, live against the FS BCM in the CX-5's own cage (mstflint's `mstreg`; mstflint comes from the AUR, not the Arch repos):
 
-- A 1-byte offset write (0x00 and 0xFF tried) injected between SMI command and read phase leaves the pending read data intact — immediately and after 3 ms; the expected register value (PHY ID 0x3590/0x5081) reads back correctly every time.
-- A cold offset-write+read returns the last-latched SMI data — the offset byte is ignored, not parsed as a new command frame.
-- Normal SMI operation recovers cleanly afterward; no lasting disturbance.
-- Caveat: emulated with STOP-separated transactions; a repeated-START combined transfer (if that's what firmware emits) is untested.
+- **Firmware accepts arbitrary I2C device addresses.** `i2c_device_address=0x56` (the BCM bridge) transacts with status 0 in both directions; kernel paths only ever use 0x50/0x51 but the firmware doesn't care.
+- **MCIA writes emit exactly one offset byte**, so `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
+- **The bridge tolerates MCIA's write-offset-then-read framing live** (the offset byte injected before the read phase is ignored, as the X520 bit-bang emulation predicted — `~/work/phydiag-work/bcm_mcia_emu.py`); PHY ID 0x3590 reads back correctly.
+- **Handler mailbox ops work**: DATA1–5 writes, CMD write, STATUS poll (PASS observed), all through the same framing.
+- RollBall is plain offset addressing at 0x51 — fits MCIA by construction.
 
-SMI *writes* fit MCIA's model directly if firmware emits a 1-byte offset: an MCIA write emits `[offset][data…]` as one I2C write, so choosing `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
+The recipe (one SMI transaction = one MCIA SET, plus one MCIA GET for reads):
 
-## Open questions (decide on arrival)
+- SMI read of `devad.reg`: MCIA SET with `i2c_device_address=0x56, device_address=(0x20|devad), size=2, dword[0]=reg<<16` → MCIA GET with `device_address=0, size=2` → value in dword[0] bits 31:16. Respect the >1 ms bridge delay between the two.
+- SMI write of `devad.reg=val`: MCIA SET with `device_address=devad, size=4, dword[0]=reg<<16|val`.
+- Live example: `mstreg -d 01:00.1 --reg_name MCIA --set "dword[0]=0x00020000" --indexes "module=1,i2c_device_address=0x56,page_number=0,device_address=0x21,size=2" --yes` then the GET → 0x3590.
 
-- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (`MLX5_I2C_ADDR_LOW/HIGH`). The register format allows it and the per-access status byte makes rejection observable.
-- Whether MCIA writes emit exactly **one offset byte** (needed for the SMI-frame synthesis above), and whether reads use STOP-separated or repeated-START framing.
-- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
-- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
+## How cabletest reaches MCIA in production (`phy_mcia.go`)
 
-## Arrival notes
+There are two host paths to MCIA, and the split is decisive:
 
-- Ports may ship in InfiniBand mode: `mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2`, reboot.
-- Check whether auto-loading `mlx5_ib` interferes with channel changes (the irdma lesson, hardware.md).
-- Force 10G if SFP28 autoneg is stubborn: `ethtool -s <dev> speed 10000 autoneg off`.
-- Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.
+- **`/dev/fwctl` (the kernel's sanctioned firmware back-channel) is read-only for MCIA.** ACCESS_REG reads work at scope `DEBUG_READ_ONLY`; **writes are silently dropped** — the RPC returns success at every layer (`errno=0`, command status 0, MCIA status 0) yet the register never changes, at every scope including `DEBUG_WRITE_FULL`. The firmware honours the ACCESS_REG opcode under fwctl's restricted user context (uid 2, caps `0x04000000`) but no-ops the actual I2C write. Proven in `~/work/phydiag-work/fwctl_mcia/diag`. Since a RollBall register **read** requires **writes** (set page/devad/command), fwctl cannot run RollBall at all.
+- **The mstflint PCICONF path (VSC vendor capability → ICMD → FLASH_REG_ACCESS) can write**, so cabletest uses it. Per-dword gateway ops through the PCI config-space vendor capability reach ICMD space; a register-access command wrapped in operation+reg TLVs carries the MCIA transaction. Reimplemented in Go (`phy_mcia.go`), no mstflint dependency at runtime.
+
+Two things that are not optional and one that is not needed:
+
+- **The MCIA `module` field is card-global and must be set per port** (module 0 = port 0, module 1 = port 1, i.e. the PCI function digit), *regardless of which function issues the command* — function 1's ICMD with `module=0` reads port 0's module. It defaults to 0, so leaving it unset makes **both** ports read port 0: identical to a working dual-end read except the second end is a phantom of the first. It hid for a while because the vendor name is identical on both Wiiteks; the serial numbers (`…X256U` vs `…X039U`) exposed it. This was also the real cause of the SNR "coming and going": two functions hammering **one** module's I2C serialised into multi-second reads that blew the freshness window. Routed to their own modules, reads are prompt and the margin holds steady.
+- **A process-wide lock holds each MCIA transaction whole.** The two ports drive independent per-function gateways but one shared card firmware; the lock keeps a command and its result from interleaving with the other port's. Not a retry, not a workaround — a plain mutex.
+- The gateway cap9 semaphore is taken in one shot (this run is its only user; the driver reaches the firmware by its own BAR interface, not this config-space back-channel) and dropped at open if a `SIGKILL`ed prior run left it held. The ICMD-level semaphore mstflint also takes is **not** needed — the mutex plus the ICMD busy-bit check cover it, and taking it raced the driver. The MCIA `l` lock bit did nothing useful in testing.
+
+There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.
+
+## The FS cannot link on the CX-5 — host-interface deadlock
+
+The FS's XFI never comes up against this firmware; **diagnostics to the FS work regardless** (all of the above ran with the link down). The mechanism, each side observed directly:
+
+- **The firmware never enables the port's serdes TX before seeing module signal.** The port FSM sits in `ETH_AN_FSM_ENABLE`/Polling and SLTP reads all-zero TX parameters (the linked Wiitek port shows real ones), in force-10G mission mode and every other reachable mode.
+- **The FS never enables its XFI TX before seeing valid host signal.** It asserts RX_LOS (the pin, mirrored in MCION, and A2h byte 110 bit 1) and its TX is electrically dark (RX eye grade 0 on the NIC) — while its copper side trains and stays up normally (7.33 shows master resolution, both receivers OK). PRBS31 and electrical-idle test patterns at 10.3125G do not clear the LOS: the gate wants valid 10GBASE-R, which only an active mission-mode port emits.
+- Each side waits for the other; the link never starts. The Wiitek escapes because it transmits unconditionally (its port reaches Physical LinkUp immediately, then relays remote-fault from the FS's dead host side over the trained copper — so its netdev also stays down until the FS end resolves).
+- ixgbe masks the FS behavior entirely: SFI idles are on the wire from driver load, so the FS side always has host signal first.
+
+Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31/IDLE test-mode TX, PH and EX loopback, PMMP `eeprom_override` (accepted but reads back zero — not implemented for SFP on 16.35.4506), PMAOS module toggle (ICMD error 0x3), FS LED-controller rewrites (1.0xA83B/0xA83C do not source the LOS pin), FS PMA soft reset, FS KR mode enable + AN restarts ([../../modules/fs/README.md](../../modules/fs/README.md)).
+
+**The deadlock is module-typed, not port-typed — confirmed by cage swap**: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).
+
+Paths forward, in escalation order:
+
+1. **Fibergaga + Wiitek as the test pair.** Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — [../../modules/fibergaga/README.md](../../modules/fibergaga/README.md)) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
+2. **EEPROM surgery on the FS** (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.
+
+## Arrival notes (resolved)
+
+- Ports arrived in Ethernet mode (`enp1s0f*np*` netdevs from first boot); no `mlxconfig` LINK_TYPE change needed.
+- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs.
+- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
+- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
+
+## Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`)
+
+Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
+
+- **RX flow steering needs `rx-ntuple-filter` enabled first** (`ETHTOOL_SFEATURES`); mlx5 ships it off and reports filter capacity 0 and rejects inserts until it is on. It then advertises **no fixed filter table** (capacity 0) and rejects `RX_CLS_LOC_ANY` over the legacy `SIOCETHTOOL` ioctl, but accepts any **explicit** slot — so ethertype rules are placed at explicit locations allocated upward from 0 (ice/ixgbe allocate downward into their advertised table; the split is keyed on the advertised capacity).
+- **Error counters**: the NIC-side rx error set is the PHY RMON categories `rx_{crc_errors,in_range_len_errors,out_of_range_len,undersize_pkts,oversize_pkts,fragments,jabbers}_phy`, mutually exclusive per frame. This is the NIC's view of the module↔NIC SFI link; the copper's own pre-FEC errors come from the module over RollBall, not here. Do **not** fold `tx_dropped` into cable-health — mlx5 counts millions of tx drops while the link is down at bringup.
+- **Bringup forces a retrain** (AN restart) that drops the link for several seconds, so the run waits for carrier before baselining counters and starting traffic; otherwise the settling flap and tx-into-dark-link drops are charged to the fresh run.
+- **PHC**: one oscillator, two PHC devices, ~33 µs quasi-static cross-port offset (see the PHC-topology section) — calibrate once at startup for cross-port latency.

docs/nics/e810/README.md

@@ -7,3 +7,7 @@ The NIC the committed measurement path was built against: its datapath delivers
 - topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. **Reads work perfectly.**
 - Every write returns aq 1 (EPERM) — tested identically against EEPROM 0x50 and BCM 0x56, so the block is write-vs-read, not address-scoped. Intel policy, survives NVM update; NVM 5.01 hardened the 3.10 silent-drop into an explicit EPERM.
 - Dead for anything needing writes: RollBall unlock needs a write; BCM SMI reads *start* with a write.
+
+## The bit-bang-below-the-AQ idea — retired with the timestamp requirement
+
+The write block is at the **Admin Queue** (firmware) level — the `0x06E2/E3` topo-I2C commands are refused with EPERM. That is not the same as the module's SDA/SCL being unreachable: the 82599 reaches the module I2C **without firmware** by toggling SDA/SCL as GPIO bits in `I2CCTL` (the `~/work/ixgbe-sff/` patch bit-bangs arbitrary START…STOP from those primitives), and whether the E810 exposes an equivalent register-level / SDP-GPIO path below the AQ policy was never tested. The idea existed to marry all-packet stamps to raw I2C in one card; the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves the stamps requirement, so the X520 covers everything and this investigation is not needed. It stays recorded as the E810's one open door if per-frame arrival times are ever wanted again — the investigation would start at the datasheet's SDP/GPIO and low-level I2C controller registers, asking whether the cage's SDA/SCL are host-addressable.

docs/nics/x520/README.md

@@ -1,11 +1,30 @@
 # Intel X520-DA2 (82599, ixgbe)
 
-The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial 1588-event latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault. The ConnectX-5 is the product NIC.
+The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate plan below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The plan, its prerequisite audit, and what it retires are in the section below.
 
 - PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
 - Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
 - **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
 
+## The product plan: smoothed-bucket rate, no hardware stamps
+
+The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. The plan replaces arrival-time bucketing with read-time bucketing plus a deconvolution pass, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.
+
+- **Rate is a headline, never a verdict.** Lost/late/corrupt come from per-frame sequence numbers and CRC checks, independent of the rate; no cable-health decision keys off the rate number. This is the license for everything below.
+- **Buckets are keyed by read time; excess above line rate moves backward.** Read jitter is a purely backward smear — a frame is read at or after its arrival, never before — so a bucket's excess above line rate is always frames that arrived earlier and were read late, and moving that excess backward to fill earlier deficits is the physically correct deconvolution, not an estimate. The dominant jitter shape (host stalls, then drains the backlog) is deficit-then-burst, which is exactly what the backward pass repairs. Line rate is known, so over/under is well-defined.
+- **A forward pass is optional edge polish.** A stall/burst pair entirely inside the window balances under the backward pass alone; only pairs cut by the window boundary leave orphan excess or deficit. Pairing those up forward is cosmetic, is the pass most likely to paper over a genuine dip at the window edge, and is droppable if it ever lies.
+- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge (measure the actual stall distribution before choosing). The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs.
+- **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
+- The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.
+
+**Prerequisite audit before building**: confirm the rate buckets are the *only* consumer of hardware RX stamps in the committed path. The buckets are currently keyed by the MAC's clock (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`), and `probe.go` leaned on a shared-PHC assumption; anything else that needs true arrival times (one-way latency, cross-port stamp comparison) does not survive the change. "Late" must remain a sequence-number notion, not a timestamp one.
+
+What this enables and retires:
+
+- **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal.
+- **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears.
+- **Build on HEAD** — the committed X520/BCM/ECD implementation, not the CX-5 stash. The work: re-key the receive buckets from MAC-stamp time to read time, add the backward-fill pass, and remove the `rx_filter=ALL` hard host check (`ts.go`) whose requirement this dissolves. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
+
 ## `sff_i2c` diagnostics transport (patched driver, validated)
 
 Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
@@ -21,7 +40,7 @@ For the timestamp/counter bench pokes the driver also carries `reg_ops` `read <r
 
 ## Timestamping facts
 
-**No all-packet exact RX timestamping, and no path to it — this disqualifies the X520 as product NIC.** A cable tester needs each frame's exact arrival time; the 82599 cannot give it for the 0x88b5 test traffic.
+**No all-packet exact RX timestamping, and no path to it — measured, settled, and no longer disqualifying.** The 82599 cannot stamp the 0x88b5 test traffic per-frame; the smoothed-bucket plan above removes the need. The facts stand for the record:
 
 - RX latches **1588-recognized frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08 = Type 100b) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood — and the datasheet marks 100b *reserved* on the 82599 (it's an X550 define); `TSIP_UT_EN` (0x00800000) will not even hold (X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
 - RX latch eligibility is configurable, not hardwired to 0x88F7 (datasheet §7.9.3.2, §8.2.3.26.1, ETQF bit 30): a frame qualifies when its EtherType sits in an ETQF slot with the 1588 bit set and TSYNCRXCTL.Type admits its message class — valid encodings 000b (L2 sync/delay_req + msgID 2,3), 001b (V1/L4), 010b (V2 L2+L4), 101b (**every V2 event frame**: payload byte 0 low nibble with bit 3 clear). Declaring 0x88b5 as 1588 and shaping byte 0 would latch our own raw frames — datasheet-derived, untested on hardware, and irrelevant to the verdict: the serial ceiling is unchanged.

docs/state.md

@@ -1,24 +1,38 @@
 # Current state
 
-## Committed tree
+## Where the code is right now, and the plan
 
-AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — nics/README.md for what that demands of the NIC; **temporarily bypassed** in `ts.go` so BCM work can run on the X520, which cannot stamp — the check reports yellow and the panel rates read zero there; restore to fatal for the product NIC); read-time-stamped NIC-counter rates; test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix (the modules' jumbo path is exercised, not assumed); module diagnostics (`phy.go`, over the patched-ixgbe `sff_i2c` debugfs; module type dispatched from the EEPROM vendor PN before any protocol touch, with a BCM client — compound-op framing, one loop goroutine per module owning every transport touch, all work time-division-multiplexed with the firmware's own mailbox client: each request admitted only inside a 3.4 s edge-locked window after an observed internal temp poll, re-acquiring phase whenever it was lost, which ends the stale-read poisoning of the corrected channel — and a RollBall client for the Wiitek — unlock/page/mailbox in Go, whitelist-guarded so any register outside the proven-safe set panics before touching hardware; corrected charges and link-down reads print their raw registers so events self-attribute — modules/fs/ and modules/wiitek/ for the trap lists and validation): bringup identifies both modules and forces EEE off, jumbo on and the master/slave roles on every BCM end every boot (A master / B slave when both ends are BCM; with a non-BCM partner the BCM is forced slave and the partner auto-resolves master — the proven mixed combination) — no trustworthy readback exists and no cable is guaranteed to probe through — and every AN restart forces the enable bit (`7.0 |= 0x1200`: the ECD can leave 7.0.12 cleared, which mutes AN entirely and holds the link down through restarts and resets), with an explicit AN restart on the ECD end after every diag; the ECD — per-pair verdicts, lengths and pair maps are the length/wiring path — runs through one async path at startup and on every reset, never blocking the UI, with counters re-baselining only after the diag's own link blip so it is never charged to the run; a 1 Hz poller feeds per-pair SNR margin and the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count) to the panel and console — SNR comes only from ends that measure IEEE margins (the Wiitek's 1.133–136, the reason it is in the pair); the BCM's handler SNR is not polled (the derived number was never actionable and each GET costs handler traffic), so an all-BCM pair shows no SNR; green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run; framebuffer UI; harness.
+**The committed tree (HEAD) is the X520/BCM/ECD implementation** — AF_PACKET datapath, the mixed FS+Wiitek diagnostics, the ECD length path — and **the plan is to build on exactly that**: the smoothed-bucket rate design (nics/x520/, "The product plan") dissolves the per-frame hardware-timestamp requirement that exiled the X520, so it returns as the product NIC with the FS+Wiitek pair (length + SNR) intact. The work, once the timestamp-consumer audit clears: re-key the receive buckets from MAC-stamp time to read time, add the backward excess-fill pass, and drop the `rx_filter=ALL` hard check in `ts.go`. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
+
+## The stashed Wiitek/CX-5 implementation (parts bin)
+
+All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.
+
+AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively — the `ts.go` bypass is now a no-op since the check passes); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
+
+Module diagnostics (`phy.go` + `phy_mcia.go`): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe `sff_i2c` debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because `/dev/fwctl` is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).
+
+Bringup is read-only for the Wiitek pair (both auto-resolve master/slave and advertise no EEE — verified, nothing forced), so it never drops the carrier and the run starts immediately with no wait for link. A ~1 Hz poller feeds **per-pair SNR margin from both ends** (each Wiitek's IEEE 1.133–136 addressed by its own card-global MCIA module index — the headline: worst pair across both ends) plus the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count); green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run. Framebuffer UI; harness.
 
 ## Stashes
 
-- `stash@{0}` — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
-- `stash@{1}` — phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
+Indices shift as stashes are pushed/popped — match by message, not number.
+
+- **"CX-5 Wiitek: MCIA transport…"** (currently `stash@{0}`) — the whole Wiitek-pair rewrite: `phy_mcia.go` (VSC/ICMD MCIA transport), transport-interface `phy.go`, dropped BCM/ECD/length, ntuple + mlx5 counters, per-port card-global MCIA module index. Carries live debug instrumentation and `rbCmdPoll` lowered to 2 ms from the mailbox-latency investigation — **not shippable as-is**, clean it before committing. Also carries the pre-existing uncommitted `ui.go`/`ui_test.go` edits that were entangled in the same files.
+- **"AF_XDP both directions…"** (currently `stash@{1}`) — AF_XDP conversion (hand-rolled XSK + XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
+- **"phydiag campaign…"** (currently `stash@{2}`) — python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
 
 ## The box
 
-- Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
+- Single usable PCIe slot (Gen4 x8): holds the ConnectX-5 (MCX512A-ACU, dual SFP28), trained Gen3 ×8 (8 GT/s, 63 Gb/s). The X520 is out of the box.
 - X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
 - Many CPU cores; goroutine-heavy designs welcome.
+- mstflint (AUR — not in the Arch repos) provides `mstlink`/`mstreg`/`mstconfig` for the CX-5.
 
 | Interface | Device | Role (rules: hardware.md) |
 |---|---|---|
-| `enp1s0f0` | X520 port 0 (ixgbe) | Test pair — Wiitek module (SNR end) |
-| `enp1s0f1` | X520 port 1 (ixgbe) | Test pair — FS module (length end) |
+| `enp1s0f0np0` | ConnectX-5 port 0 (mlx5) | Test pair — Wiitek module (SN …X256U). **Linked at 10G** to its partner over the long (~45 m) cable |
+| `enp1s0f1np1` | ConnectX-5 port 1 (mlx5) | Test pair — Wiitek module (SN …X039U). **Linked at 10G** |
 | `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots). **The FS cannot live here**: X710 firmware disables Rx/Tx on the honest FS EEPROM ("unsupported SFP module type") and i40e has no `allow_unsupported_sfp` — the noise pair needs lying modules. Seated: Fibergaga (f0) + Wiitek SN …X170U (f1), both qualified, noise cycle live |
 | `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark |
 
@@ -26,10 +40,10 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet
 
 | Item | Status | Notes |
 |---|---|---|
-| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Mixed test pair: port 0 Wiitek (SN WAMZ012606X039U), port 1 FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) |
-| ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (nics/connectx-5/). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) |
-| Replacement Wiiteks | One in the X520 test pair (port 0), rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
-| FS SFP-10G-T-100 ×2 | One in the X520 test pair (port 1), one on the shelf (rejected by the X710 — ixgbe-only) | BCM84891L, documented, robust. The mixed FS+Wiitek pair is the configuration under evaluation: length from the FS ECD, IEEE per-pair SNR margins from the Wiitek — the best of both worlds a 2× FS pair cannot give (the BCM never populates IEEE SNR) |
+| ConnectX-5 | **Installed** in the single PCIe slot — retired as product NIC, to swap out for the X520 | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
+| X520-DA2 | Out of the box — **the product NIC under the smoothed-bucket rate plan** (nics/x520/); goes back into the slot with FS+Wiitek | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven — the HEAD implementation targets exactly this |
+| Replacement Wiiteks | Two in the CX-5 test pair (…X256U / …X039U), one in the X710 noise pair (…X170U), rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
+| FS SFP-10G-T-100 ×2 | Both on the shelf — one returns to the test pair with the X520 | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration under the X520 plan, and is what HEAD implements |
 | Fibergaga SFP-10G-T-30M | In the X710 noise pair (f0) | Aquantia, RollBall, the documented oracle |
 | 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
 | E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
@@ -62,7 +76,7 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
 - **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
 - **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.
 
-## Open items
-
+- **The NIC decision is made: the X520, via the smoothed-bucket rate plan** (nics/x520/, "The product plan"). Next steps, in order: (1) the audit — confirm the rate buckets are the only consumer of hardware RX stamps (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`; `probe.go`'s shared-PHC lean; "late" must be sequence-number-based); (2) measure the host read-stall distribution to size the smoothing window; (3) re-key buckets to read time, add the backward excess-fill, drop the `rx_filter=ALL` hard check (`ts.go`). Hardware: put the X520 back in the slot, CX-5 out, FS+Wiitek seated as at HEAD. The CX-5 mailbox-quantum mystery and the E810 bit-bang idea stay recorded in their device pages but are retired.
+- **Length returns as a goal** — the X520 plan restores the FS (ECD length) alongside the Wiitek (SNR), the configuration HEAD already implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
 - **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
 - **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).