X520 live as the product NIC: rate buckets re-keyed to read time (one shared clock, one commit per drained batch) with the backward smear as a settled window — each completed bucket enters once, donates excess to the nearest earlier deficits once, locks for display when nothing later can refill it (per-sample recompute double-counted excess and read >20G on a 20G wire); timestamp machinery deleted (ts.go, SO_TIMESTAMPING, rx_filter=ALL check) after the audit confirmed the buckets were its sole consumer; testDriver ixgbe; RollBall mailbox hardened against orphaned-command completions (never sample status before a poll gap, value from the same block read, devad/reg echo required — a killed session's 7.60 read answered a PHY ID as 0x0006); full hardware pass: 20.00G flat both directions, zero errors, ECD 42m, SNR +7.4dB

This commit is contained in:
flamingcow
2026-08-17 09:35:58 -07:00
parent 06de095d1c
commit 3d7105073e
15 changed files with 313 additions and 404 deletions
+5 -19
View File
@@ -2,18 +2,14 @@
The genuinely open, thinking-worthy problems — nothing here is resolved. Analyses and settled facts live in the device docs; each entry points at the doc that holds its detail. Read `state.md` first for placement context.
## 1. Can ConnectX-5 MCIA reach the BCM? (decide on arrival)
The decisive unknown for product diagnostics coverage ([nics/connectx-5/](nics/connectx-5/README.md)):
- Does firmware MCIA accept **arbitrary I2C device addresses**? The register format allows it (8-bit field, per-access status — verified in `mlx5_ifc.h`); firmware acceptance is the unknown. The module side is settled: the BCM bridge tolerates MCIA's offset-write-then-read framing, proven on hardware ([nics/connectx-5/](nics/connectx-5/README.md)).
- Does MCIA emit **one offset byte per write** (which would let SMI command frames be synthesized exactly), and STOP-separated or repeated-START reads?
- If firmware says no: product diagnostics via the Marvell/Aquantia modules only.
## 2. ECD run disturbance
## 1. ECD run disturbance
The BCM ECD run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear). Until the disturbance is characterized, length measurement is a between-runs operation, not a during-run one ([modules/fs/](modules/fs/README.md)).
## 2. Smear window sizing
`smearWindow` (4 buckets = 64 ms) is a chosen default, not a measured one. Measure the host read-stall distribution under load on the X520 datapath, then size the window comfortably past the worst stall — it is also the display latency, so no larger than needed ([nics/x520/](nics/x520/README.md)).
## 3. Pre-FEC correlation run
The register question is answered (post-FEC vs corrected-by-iteration histogram located); the graded-noise correlation run that verifies the expected ordering has not happened. Design and instrument: [modules/fibergaga/](modules/fibergaga/README.md).
@@ -21,13 +17,3 @@ The register question is answered (post-FEC vs corrected-by-iteration histogram
## 4. Wiitek VCT — pursue or leave dead?
No confirmed-safe path exists (every candidate lands in the µC danger window). The open decision is whether the capability is worth the NDA route or a sacrificial unit — the product doesn't need it for length ([modules/wiitek/](modules/wiitek/README.md), [modules/README.md](modules/README.md)).
## 5. X520 bench divergences — features to restore on the product NIC
Running on the X520 (BCM development) required parking product-NIC capabilities the 82599 lacks. Each stays parked only until the ConnectX-5 is in; none is a settled design change:
- **Hardware timestamp hard check bypassed** (`ts.go`): `rx_filter=ALL` failure now reports yellow and continues instead of stopping the run. On the X520 that means the per-frame-stamp rate buckets never fill and the panel/console rates read zero — the measurement doctrine (exact per-frame RX stamps as a hard host requirement) is intact in the docs and must return to fatal on the product NIC.
- **RX error counter set is per-driver** (`counters.go`, `nicRxStatsByDriver`): the ice set is the reference — jabber, fragments, `illegal_bytes` (64b/66b decode errors) and MAC local/remote faults have **no ixgbe ethtool equivalent**, so those signals are simply absent on the bench. The mlx5 name set needs deriving on CX-5 arrival; the "as close to BER as the link reports" counters (illegal bytes, faults) are the ones to insist on finding there.
- **TX interrupt moderation** (`system.go`): ixgbe's mixed rx/tx vectors reject a tx-specific value, so `checkCoalesce` falls back to rx-shared-with-tx on EINVAL. Generic and self-reporting, but verify the product NIC takes the full rx+tx pair (the fallback must never fire there).
- **`testDriver` still names "ice"** (`main.go`): the default pair discovery has no working target — bench runs pass `-a`/`-b` explicitly. Point it at the product driver (mlx5_core) when the CX-5 lands.
- **Module I2C transport is ixgbe-only** (`phy.go`, `openBCM`): the sff_i2c debugfs path. The CX-5 needs the MCIA answer (question 1) and a second transport arm.