theater/cabletest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

X520 live as the product NIC: rate buckets re-keyed to read time (one shared clock, one commit per drained batch) with the backward smear as a settled window — each completed bucket enters once, donates excess to the nearest earlier deficits once, locks for display when nothing later can refill it (per-sample recompute double-counted excess and read >20G on a 20G wire); timestamp machinery deleted (ts.go, SO_TIMESTAMPING, rx_filter=ALL check) after the audit confirmed the buckets were its sole consumer; testDriver ixgbe; RollBall mailbox hardened against orphaned-command completions (never sample status before a poll gap, value from the same block read, devad/reg echo required — a killed session's 7.60 read answered a PHY ID as 0x0006); full hardware pass: 20.00G flat both directions, zero errors, ECD 42m, SNR +7.4dB

Browse Source
This commit is contained in:
flamingcow
2026-08-17 09:35:58 -07:00
parent 06de095d1c
commit 3d7105073e
15 changed files with 313 additions and 404 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/state.md
+21 -24
View File
@@ -1,14 +1,14 @@
# Current state
## Where the code is right now, and the plan
## Where the code is right now
**The committed tree (HEAD) is the X520/BCM/ECD implementation** — AF_PACKET datapath, the mixed FS+Wiitek diagnostics, the ECD length path — and **the plan is to build on exactly that**: the smoothed-bucket rate design (nics/x520/, "The product plan") dissolves the per-frame hardware-timestamp requirement that exiled the X520, so it returns as the product NIC with the FS+Wiitek pair (length + SNR) intact. The work, once the timestamp-consumer audit clears: re-key the receive buckets from MAC-stamp time to read time, add the backward excess-fill pass, and drop the `rx_filter=ALL` hard check in `ts.go`. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
**The committed tree is the X520 product implementation**: AF_PACKET datapath with read-time rate buckets and the backward excess-fill smear (nics/x520/, "The smoothed-bucket rate" — rate is a headline, never a verdict; `smearWindow` 4 buckets = 64 ms display latency, resize after measuring the stall distribution: open-questions), the mixed FS+Wiitek diagnostics with the BCM ECD length path, `testDriver` = ixgbe. The hardware-timestamp machinery is gone — the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`) — after the audit confirmed the rate buckets were its sole consumer. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
## The stashed Wiitek/CX-5 implementation (parts bin)
All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.
AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively — the `ts.go` bypass is now a no-op since the check passes); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
Module diagnostics (`phy.go` + `phy_mcia.go`): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe `sff_i2c` debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because `/dev/fwctl` is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).
@@ -24,32 +24,29 @@ Indices shift as stashes are pushed/popped — match by message, not number.
## The box
- Single usable PCIe slot (Gen4 x8): holds the ConnectX-5 (MCX512A-ACU, dual SFP28), trained Gen3 ×8 (8 GT/s, 63 Gb/s). The X520 is out of the box.
- Single usable PCIe slot (Gen4 x8): holds the **X520-DA2** (dual SFP+). Verify 5 GT/s ×8 trained at bringup (`lspci -vv`). The ConnectX-5 is out of the box.
- X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
- Many CPU cores; goroutine-heavy designs welcome.
- mstflint (AUR — not in the Arch repos) provides `mstlink`/`mstreg`/`mstconfig` for the CX-5.
- mstflint (AUR — not in the Arch repos) provides `mstlink`/`mstreg`/`mstconfig` for the CX-5, if it ever returns.
| Interface | Device | Role (rules: hardware.md) |
|---|---|---|
| `enp1s0f0np0` | ConnectX-5 port 0 (mlx5) | Test pair — Wiitek module (SN …X256U). **Linked at 10G** to its partner over the long (~45 m) cable |
| `enp1s0f1np1` | ConnectX-5 port 1 (mlx5) | Test pair — Wiitek module (SN …X039U). **Linked at 10G** |
| `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots). **The FS cannot live here**: X710 firmware disables Rx/Tx on the honest FS EEPROM ("unsupported SFP module type") and i40e has no `allow_unsupported_sfp` — the noise pair needs lying modules. Seated: Fibergaga (f0) + Wiitek SN …X170U (f1), both qualified, noise cycle live |
| `enp1s0f0` + second port | X520 (ixgbe) | Test pair — FS + Wiitek, dispatched by EEPROM PN so either port may hold either module. **Stock ixgbe refuses the FS's honest EEPROM and its port has no netdev until `./load-ixgbe`** (allow_unsupported_sfp=1 + the sff_i2c/ETQF patches); the tool needs exactly 2 ixgbe interfaces |
| `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots) — two Wiiteks, both qualified by the X710. The FS cannot live here (firmware rejects the honest EEPROM, no `allow_unsupported_sfp` in i40e) |
| `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark |
## Hardware
| Item | Status | Notes |
|---|---|---|
| ConnectX-5 | **Installed** in the single PCIe slot — retired as product NIC, to swap out for the X520 | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
| X520-DA2 | Out of the box — **the product NIC under the smoothed-bucket rate plan** (nics/x520/); goes back into the slot with FS+Wiitek | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven — the HEAD implementation targets exactly this |
| Replacement Wiiteks | Two in the CX-5 test pair (…X256U / …X039U), one in the X710 noise pair (…X170U), rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
| FS SFP-10G-T-100 ×2 | Both on the shelf — one returns to the test pair with the X520 | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration under the X520 plan, and is what HEAD implements |
| Fibergaga SFP-10G-T-30M | In the X710 noise pair (f0) | Aquantia, RollBall, the documented oracle |
| X520-DA2 | **Installed** in the single PCIe slot — **the product NIC** (nics/x520/) | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven; the smoothed-bucket rate design removes the need for hardware RX stamps |
| ConnectX-5 | Out of the box — retired | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
| FS SFP-10G-T-100 ×2 | One in the X520 test pair, one on the shelf | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration |
| Replacement Wiiteks | One in the X520 test pair, two in the X710 noise pair, rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
| Fibergaga SFP-10G-T-30M | On the shelf | Aquantia, RollBall, the documented oracle; the alternate length path if the pair mix ever changes |
| 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
| E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
**The X520 is disqualified as the product NIC — measurement, not a narrow gap.** Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, nics/x520/). Per-queue counters were explored and **rejected** — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.
## X520 diagnostics path — validated on hardware
In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`:
@@ -60,13 +57,12 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
- **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
- **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill.
## Bringup plan
## Bringup on the swapped box
1. ~~Card installed, modules seated, `./load-ixgbe`, `sff_i2c` present.~~ Done.
2. ~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
5. ~~RX steering: program/verify the ETQF path.~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (`~/work/phydiag-work/etqfbench`). `system.go` runs unchanged.
1. `./load-ixgbe` — until then the FS port has no netdev (stock driver, error -95) and `driverPair("ixgbe")` cannot find 2 interfaces. Verify `sff_i2c` present for both ports and 5 GT/s ×8 trained.
2. ~~Run.~~ Done — full pass on hardware: both modules identified and dispatched, ECD at startup (all pairs ok, ~42 m headline), steady 20.00–20.04 G both directions with zero lost/corrupt/link/internal over a 90 s run, SNR +7.4–7.5 dB, noise cycle live. SETTINGS self-reports the ixgbe shape (rx=8160/tx=4096 taken as-is; coalesce tx-shares-rx).
3. Measure the host read-stall distribution under load and resize `smearWindow` if 64 ms is mis-sized (open-questions).
4. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
## RX flow-steering on ixgbe
@@ -76,7 +72,8 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
- **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
- **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.
- **The NIC decision is made: the X520, via the smoothed-bucket rate plan** (nics/x520/, "The product plan"). Next steps, in order: (1) the audit — confirm the rate buckets are the only consumer of hardware RX stamps (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`; `probe.go`'s shared-PHC lean; "late" must be sequence-number-based); (2) measure the host read-stall distribution to size the smoothing window; (3) re-key buckets to read time, add the backward excess-fill, drop the `rx_filter=ALL` hard check (`ts.go`). Hardware: put the X520 back in the slot, CX-5 out, FS+Wiitek seated as at HEAD. The CX-5 mailbox-quantum mystery and the E810 bit-bang idea stay recorded in their device pages but are retired.
- **Length returns as a goal** — the X520 plan restores the FS (ECD length) alongside the Wiitek (SNR), the configuration HEAD already implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
## Standing goals
- **Length is a product goal again** — the FS (ECD length) rides alongside the Wiitek (SNR) in the test pair, which the tree implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then — open-questions).
- **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).