X520 returns as product NIC: smoothed-bucket rate plan (read-time buckets, backward excess-fill, rate as headline not verdict) dissolves the per-frame RX timestamp requirement; CX-5 retired with full record (MCIA via VSC/ICMD proven, fwctl write-block, card-global module index, ~150ms mailbox quantum unexplained, FS link deadlock); E810 below-AQ bit-bang idea recorded and retired; FS+Wiitek pair and length return as goals

docs/state.md

@@ -1,24 +1,38 @@
 # Current state
 
-## Committed tree
+## Where the code is right now, and the plan
 
-AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — nics/README.md for what that demands of the NIC; **temporarily bypassed** in `ts.go` so BCM work can run on the X520, which cannot stamp — the check reports yellow and the panel rates read zero there; restore to fatal for the product NIC); read-time-stamped NIC-counter rates; test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix (the modules' jumbo path is exercised, not assumed); module diagnostics (`phy.go`, over the patched-ixgbe `sff_i2c` debugfs; module type dispatched from the EEPROM vendor PN before any protocol touch, with a BCM client — compound-op framing, one loop goroutine per module owning every transport touch, all work time-division-multiplexed with the firmware's own mailbox client: each request admitted only inside a 3.4 s edge-locked window after an observed internal temp poll, re-acquiring phase whenever it was lost, which ends the stale-read poisoning of the corrected channel — and a RollBall client for the Wiitek — unlock/page/mailbox in Go, whitelist-guarded so any register outside the proven-safe set panics before touching hardware; corrected charges and link-down reads print their raw registers so events self-attribute — modules/fs/ and modules/wiitek/ for the trap lists and validation): bringup identifies both modules and forces EEE off, jumbo on and the master/slave roles on every BCM end every boot (A master / B slave when both ends are BCM; with a non-BCM partner the BCM is forced slave and the partner auto-resolves master — the proven mixed combination) — no trustworthy readback exists and no cable is guaranteed to probe through — and every AN restart forces the enable bit (`7.0 |= 0x1200`: the ECD can leave 7.0.12 cleared, which mutes AN entirely and holds the link down through restarts and resets), with an explicit AN restart on the ECD end after every diag; the ECD — per-pair verdicts, lengths and pair maps are the length/wiring path — runs through one async path at startup and on every reset, never blocking the UI, with counters re-baselining only after the diag's own link blip so it is never charged to the run; a 1 Hz poller feeds per-pair SNR margin and the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count) to the panel and console — SNR comes only from ends that measure IEEE margins (the Wiitek's 1.133–136, the reason it is in the pair); the BCM's handler SNR is not polled (the derived number was never actionable and each GET costs handler traffic), so an all-BCM pair shows no SNR; green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run; framebuffer UI; harness.
+**The committed tree (HEAD) is the X520/BCM/ECD implementation** — AF_PACKET datapath, the mixed FS+Wiitek diagnostics, the ECD length path — and **the plan is to build on exactly that**: the smoothed-bucket rate design (nics/x520/, "The product plan") dissolves the per-frame hardware-timestamp requirement that exiled the X520, so it returns as the product NIC with the FS+Wiitek pair (length + SNR) intact. The work, once the timestamp-consumer audit clears: re-key the receive buckets from MAC-stamp time to read time, add the backward excess-fill pass, and drop the `rx_filter=ALL` hard check in `ts.go`. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
+
+## The stashed Wiitek/CX-5 implementation (parts bin)
+
+All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.
+
+AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively — the `ts.go` bypass is now a no-op since the check passes); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
+
+Module diagnostics (`phy.go` + `phy_mcia.go`): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe `sff_i2c` debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because `/dev/fwctl` is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).
+
+Bringup is read-only for the Wiitek pair (both auto-resolve master/slave and advertise no EEE — verified, nothing forced), so it never drops the carrier and the run starts immediately with no wait for link. A ~1 Hz poller feeds **per-pair SNR margin from both ends** (each Wiitek's IEEE 1.133–136 addressed by its own card-global MCIA module index — the headline: worst pair across both ends) plus the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count); green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run. Framebuffer UI; harness.
 
 ## Stashes
 
-- `stash@{0}` — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
-- `stash@{1}` — phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
+Indices shift as stashes are pushed/popped — match by message, not number.
+
+- **"CX-5 Wiitek: MCIA transport…"** (currently `stash@{0}`) — the whole Wiitek-pair rewrite: `phy_mcia.go` (VSC/ICMD MCIA transport), transport-interface `phy.go`, dropped BCM/ECD/length, ntuple + mlx5 counters, per-port card-global MCIA module index. Carries live debug instrumentation and `rbCmdPoll` lowered to 2 ms from the mailbox-latency investigation — **not shippable as-is**, clean it before committing. Also carries the pre-existing uncommitted `ui.go`/`ui_test.go` edits that were entangled in the same files.
+- **"AF_XDP both directions…"** (currently `stash@{1}`) — AF_XDP conversion (hand-rolled XSK + XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
+- **"phydiag campaign…"** (currently `stash@{2}`) — python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
 
 ## The box
 
-- Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
+- Single usable PCIe slot (Gen4 x8): holds the ConnectX-5 (MCX512A-ACU, dual SFP28), trained Gen3 ×8 (8 GT/s, 63 Gb/s). The X520 is out of the box.
 - X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
 - Many CPU cores; goroutine-heavy designs welcome.
+- mstflint (AUR — not in the Arch repos) provides `mstlink`/`mstreg`/`mstconfig` for the CX-5.
 
 | Interface | Device | Role (rules: hardware.md) |
 |---|---|---|
-| `enp1s0f0` | X520 port 0 (ixgbe) | Test pair — Wiitek module (SNR end) |
-| `enp1s0f1` | X520 port 1 (ixgbe) | Test pair — FS module (length end) |
+| `enp1s0f0np0` | ConnectX-5 port 0 (mlx5) | Test pair — Wiitek module (SN …X256U). **Linked at 10G** to its partner over the long (~45 m) cable |
+| `enp1s0f1np1` | ConnectX-5 port 1 (mlx5) | Test pair — Wiitek module (SN …X039U). **Linked at 10G** |
 | `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots). **The FS cannot live here**: X710 firmware disables Rx/Tx on the honest FS EEPROM ("unsupported SFP module type") and i40e has no `allow_unsupported_sfp` — the noise pair needs lying modules. Seated: Fibergaga (f0) + Wiitek SN …X170U (f1), both qualified, noise cycle live |
 | `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark |
 
@@ -26,10 +40,10 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet
 
 | Item | Status | Notes |
 |---|---|---|
-| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Mixed test pair: port 0 Wiitek (SN WAMZ012606X039U), port 1 FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) |
-| ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (nics/connectx-5/). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) |
-| Replacement Wiiteks | One in the X520 test pair (port 0), rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
-| FS SFP-10G-T-100 ×2 | One in the X520 test pair (port 1), one on the shelf (rejected by the X710 — ixgbe-only) | BCM84891L, documented, robust. The mixed FS+Wiitek pair is the configuration under evaluation: length from the FS ECD, IEEE per-pair SNR margins from the Wiitek — the best of both worlds a 2× FS pair cannot give (the BCM never populates IEEE SNR) |
+| ConnectX-5 | **Installed** in the single PCIe slot — retired as product NIC, to swap out for the X520 | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
+| X520-DA2 | Out of the box — **the product NIC under the smoothed-bucket rate plan** (nics/x520/); goes back into the slot with FS+Wiitek | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven — the HEAD implementation targets exactly this |
+| Replacement Wiiteks | Two in the CX-5 test pair (…X256U / …X039U), one in the X710 noise pair (…X170U), rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
+| FS SFP-10G-T-100 ×2 | Both on the shelf — one returns to the test pair with the X520 | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration under the X520 plan, and is what HEAD implements |
 | Fibergaga SFP-10G-T-30M | In the X710 noise pair (f0) | Aquantia, RollBall, the documented oracle |
 | 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
 | E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
@@ -62,7 +76,7 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
 - **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
 - **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.
 
-## Open items
-
+- **The NIC decision is made: the X520, via the smoothed-bucket rate plan** (nics/x520/, "The product plan"). Next steps, in order: (1) the audit — confirm the rate buckets are the only consumer of hardware RX stamps (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`; `probe.go`'s shared-PHC lean; "late" must be sequence-number-based); (2) measure the host read-stall distribution to size the smoothing window; (3) re-key buckets to read time, add the backward excess-fill, drop the `rx_filter=ALL` hard check (`ts.go`). Hardware: put the X520 back in the slot, CX-5 out, FS+Wiitek seated as at HEAD. The CX-5 mailbox-quantum mystery and the E810 bit-bang idea stay recorded in their device pages but are retired.
+- **Length returns as a goal** — the X520 plan restores the FS (ECD length) alongside the Wiitek (SNR), the configuration HEAD already implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
 - **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
 - **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).