theater/cabletest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resolve X520 open questions on hardware: ETQF raw-L2 steering and single-hold compound I2C proven; 82599 has no all-packet exact-timestamp path so it is disqualified as product NIC (diagnostics-only), counter route rejected for a cable tester

Browse Source
This commit is contained in:
flamingcow
2026-08-12 20:36:31 -07:00
parent 910379e614
commit 5588ebec6f
5 changed files with 50 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/state.md
+8 -12
View File
@@ -34,13 +34,14 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet
| 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
| E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
**The X520 cannot be the product NIC**: rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement) and the 82599 stamps PTP only. It carries two interim campaigns: the diagnostics bring-up (certain arbitrary I2C framing) and the 82599 bucket-timestamping exploration (open-questions.md §2). If that exploration validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX.
**The X520 is disqualified as the product NIC — measurement, not a narrow gap.** Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, open-questions.md §2). Per-queue counters were explored and **rejected** — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.
## X520 diagnostics path — validated on hardware
In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`:
- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w <addr8> <bytes…>` / `r <addr8> <n>`). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`.
- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w <addr8> <bytes…>` / `r <addr8> <n>` / `x <waddr> <raddr> <delay_us> <n> <wbytes…>` single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: **ETQF steering** (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in `ixgbe_ethtool.c`, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); **`reg_ops` read-returns-value** and a **`ts_bench`** debugfs command (in-kernel latch poll/re-arm for the §2 pokes). `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`.
- **Bench tools** (`~/work/phydiag-work/`): `etqfbench/` (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), `x520poke/` (Go; drives `reg_ops`/`ts_bench` for the timestamp/counter pokes), `compound_test.py` (proves the `x` compound op against the FS BCM).
- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via `CMD_GET_SNR = 0x8030` invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
- **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
- **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill.
@@ -51,20 +52,15 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
2. ~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
5. RX steering: program/verify the ETQF path (open-questions.md §1) before cabletest traffic runs.
5. ~~RX steering: program/verify the ETQF path (open-questions.md §1).~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (`~/work/phydiag-work/etqfbench`). `system.go` runs unchanged.
## RX flow-steering on ixgbe
cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW — so the insert is rejected and the tool fails at startup (no fallback, by design). This affects the datapath regardless of module type.
**Solved on hardware.** cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (`ixgbe_ethtool.c`), so `system.go` runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (`~/work/phydiag-work/etqfbench`). Full analysis in open-questions.md §1.
Leads, most promising first (full analysis in open-questions.md §1):
- **ETQF/ETQS L2 EtherType queue filters — the answer on paper.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588); already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto ETQF slots so `system.go` is unchanged. Bench-confirm.
- **Flow Director flex-byte match — ruled out on paper.** fdir classifies IPv4/IPv6 only; the flex word narrows an IP match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding.
- **MAC → VMDq pool steering** — distinct dest MACs → pools → queues, fully raw. Fallback if ETQF disappoints.
- **RSS** — hashes IP tuples only; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline).
Fallbacks if raw-L2 steering can't work: minimal bare-IPv4 framing steered by IP_USER_FLOW, or single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable as a fallback (open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice.
- **ETQF/ETQS L2 EtherType queue filters — proven.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
- **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
- **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.
## Open items