BCM module diagnostics replace timestamp probes: bringup forces EEE off and runs ECD (re-run on every reset, re-baselining past its blip), 1 Hz SNR margin + corrected counters on panel and console; X520 bench divergences bypassed and tracked in open-questions

docs/goals.md

@@ -8,8 +8,8 @@ The target of measurement is the cable, not throughput. Line rate is a means to
 
 1. **Loss and error attribution** — reception gaps, link errors, NIC/driver counters as first-class output alongside application loss. Baseline loss must be exactly zero before a run counts; any host-side loss masks real cable faults.
 2. **Noise tolerance** — a deliberately-bad "noise" cable intertwined with the test cable, driven by link up/down cycling, stresses the cable under test with alien crosstalk.
-3. **Per-pair SNR** from the module PHYs — the leading indicator of a marginal cable before it drops frames. IEEE 802.3an standard registers on the Marvells; the BCM leaves those unpopulated and reports through its vendor command handler instead (modules/fs/).
-4. **Cable length** — sanity check and fault localization. The BCM ECD is the product path: per-pair lengths in meters, healthy pairs included, proven meter-accurate on the bench (modules/fs/).
+3. **Per-pair SNR** from the module PHYs — the leading indicator of a marginal cable before it drops frames. IEEE 802.3an standard registers on the Marvells; the BCM leaves those unpopulated and reports through its vendor command handler instead (modules/fs/). Shown as margin over the operating point, classified good/marginal/bad, so nobody needs the magic numbers to read it (`phy.go`).
+4. **Cable length** — sanity check and fault localization. The BCM ECD is the product path: per-pair lengths in meters, healthy pairs included, proven meter-accurate on the bench (modules/fs/); run at bringup and on every reset, with pair-swap and fault verdicts on the panel.
 5. **Pre-FEC error visibility** — corrected-error counters that move before post-FEC loss appears. Vendor-specific; located on the Aquantia (modules/fibergaga/), unlocated elsewhere. Standard latched PCS counters (errored blocks, BER, block-lock loss) are the working proxy under noise stress.
 
 ## Design preferences that shaped the tool

docs/modules/fs/README.md

@@ -27,7 +27,7 @@ Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-
 
 | Capability | How | Observed |
 |---|---|---|
-| Per-pair SNR | `CMD_GET_SNR` 0x8030, **invoked bare** (writing the documented DATA1 display flag returns zeros; IEEE 1.133–1.140 never populate — constant 0x8080) | DATA2–5 = SNR A–D ×0.1 dB; ≈ 27–32 dB absolute on bench, 0.1–0.4 dB jitter. 10GBASE-T operating point ≈ 26.5 dB, so margin ≈ value − 26.5 |
+| Per-pair SNR | `CMD_GET_SNR` 0x8030, **invoked bare** (writing the documented DATA1 display flag returns zeros; IEEE 1.133–1.140 never populate — constant 0x8080) | DATA2–5 = SNR A–D ×0.1 dB; ≈ 27–32 dB absolute on bench, 0.1–0.4 dB jitter. 10GBASE-T operating point ≈ 26.5 dB, so margin ≈ value − 26.5. cabletest shows the margin classified green ≥ 3 dB / amber ≥ 1 dB / red below — provisional thresholds until the graded-noise correlation run |
 | Die temperature | `CMD_GET_CURRENT_TEMP` 0x8031 | ~68–70 °C on bench |
 | Supply rails | `GET_CURRENT_VOLTAGE` 0x802F | 0.8 V and 1.88 V rails, tenths of mV |
 | Error counters | IEEE PCS 3.32/3.33 — block lock, latched errored-block/BER, clear-on-read | The noise-stress error proxy |
@@ -37,7 +37,7 @@ Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-
 
 | Item | Command | State |
 |---|---|---|
-| EEE / AutogrEEEn | 0x8008/0x8009 | Once read 0x0047 (10G AutogrEEEn variable latency + 5G/1G native — local-only, invisible in IEEE 7.60/7.61, which read 0); later reads 0. **Forced all-off** via SET with explicit params `(0, 0, 0x7A12, 0x480, 0)` + AN restart, verified. cabletest should apply this defensively at bringup |
+| EEE / AutogrEEEn | 0x8008/0x8009 | Once read 0x0047 (10G AutogrEEEn variable latency + 5G/1G native — local-only, invisible in IEEE 7.60/7.61, which read 0); later reads 0. **Forced all-off** via SET with explicit params `(0, 0, 0x7A12, 0x480, 0)` + AN restart, verified. cabletest applies this defensively at every bringup and verifies 7.60 reads 0 after relink (`phy.go`) |
 | EEE wire-truth | arm 0x801A after link-up, read 0x801B | Zero LPI events/duration on idle link; repeat under traffic |
 | Fast retrain | 0x800A (datasheet titles it EMI_MODE; description is fast retrain) | Enabled 10G/5G/2.5G; IEEE 1.147 = 0x0019, count bits zero. Keep enabled; read the 1.147 count per run — a marginal cable that fast-retrains still gets counted |
 | Pair map | 0x8000 | DATA2 = 0x00E4 = identity (A/B/C/D straight through) — MDI wiring verification works |

docs/open-questions.md

@@ -21,3 +21,13 @@ The register question is answered (post-FEC vs corrected-by-iteration histogram
 ## 4. Wiitek VCT — pursue or leave dead?
 
 No confirmed-safe path exists (every candidate lands in the µC danger window). The open decision is whether the capability is worth the NDA route or a sacrificial unit — the product doesn't need it for length ([modules/wiitek/](modules/wiitek/README.md), [modules/README.md](modules/README.md)).
+
+## 5. X520 bench divergences — features to restore on the product NIC
+
+Running on the X520 (BCM development) required parking product-NIC capabilities the 82599 lacks. Each stays parked only until the ConnectX-5 is in; none is a settled design change:
+
+- **Hardware timestamp hard check bypassed** (`ts.go`): `rx_filter=ALL` failure now reports yellow and continues instead of stopping the run. On the X520 that means the per-frame-stamp rate buckets never fill and the panel/console rates read zero — the measurement doctrine (exact per-frame RX stamps as a hard host requirement) is intact in the docs and must return to fatal on the product NIC.
+- **RX error counter set is per-driver** (`counters.go`, `nicRxStatsByDriver`): the ice set is the reference — jabber, fragments, `illegal_bytes` (64b/66b decode errors) and MAC local/remote faults have **no ixgbe ethtool equivalent**, so those signals are simply absent on the bench. The mlx5 name set needs deriving on CX-5 arrival; the "as close to BER as the link reports" counters (illegal bytes, faults) are the ones to insist on finding there.
+- **TX interrupt moderation** (`system.go`): ixgbe's mixed rx/tx vectors reject a tx-specific value, so `checkCoalesce` falls back to rx-shared-with-tx on EINVAL. Generic and self-reporting, but verify the product NIC takes the full rx+tx pair (the fallback must never fire there).
+- **`testDriver` still names "ice"** (`main.go`): the default pair discovery has no working target — bench runs pass `-a`/`-b` explicitly. Point it at the product driver (mlx5_core) when the CX-5 lands.
+- **Module I2C transport is ixgbe-only** (`phy.go`, `openBCM`): the sff_i2c debugfs path. The CX-5 needs the MCIA answer (question 1) and a second transport arm.

docs/state.md

@@ -2,7 +2,7 @@
 
 ## Committed tree
 
-AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — nics/README.md for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (`probe.go`); framebuffer UI; harness.
+AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — nics/README.md for what that demands of the NIC; **temporarily bypassed** in `ts.go` so BCM work can run on the X520, which cannot stamp — the check reports yellow and the panel rates read zero there; restore to fatal for the product NIC); read-time-stamped NIC-counter rates; BCM module diagnostics (`phy.go`, over the patched-ixgbe `sff_i2c` debugfs, compound-op framing): bringup identifies both modules, forces EEE off with an AN restart, then runs the ECD — per-pair verdicts, lengths and pair maps are the length/wiring path — and every reset re-runs it, re-baselining the counters only after the diag's own link blip so it is never charged to the run; a 1 Hz poller feeds per-pair SNR margin (vs the ≈26.5 dB operating point; green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run) and the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count) to the panel and console; framebuffer UI; harness.
 
 ## Stashes
 
@@ -17,7 +17,7 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet
 
 | Interface | Device | Role (rules: hardware.md) |
 |---|---|---|
-| `enp1s0f0` | X520 port 0 (ixgbe) | Test pair — new Wiitek module |
+| `enp1s0f0` | X520 port 0 (ixgbe) | Test pair — FS module |
 | `enp1s0f1` | X520 port 1 (ixgbe) | Test pair — FS module |
 | `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots) |
 | `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark |
@@ -26,10 +26,10 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet
 
 | Item | Status | Notes |
 |---|---|---|
-| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Port 0 `enp1s0f0` = new Wiitek (SN WAMZ012606X039U); port 1 `enp1s0f1` = FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) |
+| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Both ports FS (port 1 SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) |
 | ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (nics/connectx-5/). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) |
-| Replacement Wiiteks | Arrived; one in X520 port 0 | Originals bricked by register exploration — modules/wiitek/ trap first |
-| FS SFP-10G-T-100 ×2 | In hand; one in X520 port 1 | BCM84891L, documented, robust |
+| Replacement Wiiteks | Arrived; on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
+| FS SFP-10G-T-100 ×2 | Both in the X520 test pair | BCM84891L, documented, robust. **2× FS at both ends is the expected product module config** unless mixed ends prove wanted |
 | Fibergaga SFP-10G-T-30M | In hand | Aquantia, RollBall, the documented oracle |
 | 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
 | E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
@@ -64,5 +64,5 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
 
 ## Open items
 
-- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. Remaining work is characterizing the link blip the run causes (length is a between-runs operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
+- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
 - **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).