cabletest/docs/state.md

Current state

Committed tree

AF_PACKET raw sockets everywhere (sock.go); flow-director steering; per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL as a hard host check — nics/README.md for what that demands of the NIC; temporarily bypassed in ts.go so BCM work can run on the X520, which cannot stamp — the check reports yellow and the panel rates read zero there; restore to fatal for the product NIC); read-time-stamped NIC-counter rates; test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix (the modules' jumbo path is exercised, not assumed); module diagnostics (phy.go, over the patched-ixgbe sff_i2c debugfs; module type dispatched from the EEPROM vendor PN before any protocol touch, with a BCM client — compound-op framing, one loop goroutine per module owning every transport touch, all work time-division-multiplexed with the firmware's own mailbox client: each request admitted only inside a 3.4 s edge-locked window after an observed internal temp poll, re-acquiring phase whenever it was lost, which ends the stale-read poisoning of the corrected channel — and a RollBall client for the Wiitek — unlock/page/mailbox in Go, whitelist-guarded so any register outside the proven-safe set panics before touching hardware; corrected charges and link-down reads print their raw registers so events self-attribute — modules/fs/ and modules/wiitek/ for the trap lists and validation): bringup identifies both modules and forces EEE off, jumbo on and the master/slave roles on every BCM end every boot (A master / B slave when both ends are BCM; with a non-BCM partner the BCM is forced slave and the partner auto-resolves master — the proven mixed combination) — no trustworthy readback exists and no cable is guaranteed to probe through — and every AN restart forces the enable bit (7.0 |= 0x1200: the ECD can leave 7.0.12 cleared, which mutes AN entirely and holds the link down through restarts and resets), with an explicit AN restart on the ECD end after every diag; the ECD — per-pair verdicts, lengths and pair maps are the length/wiring path — runs through one async path at startup and on every reset, never blocking the UI, with counters re-baselining only after the diag's own link blip so it is never charged to the run; a 1 Hz poller feeds per-pair SNR margin and the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count) to the panel and console — SNR comes only from ends that measure IEEE margins (the Wiitek's 1.133–136, the reason it is in the pair); the BCM's handler SNR is not polled (the derived number was never actionable and each GET costs handler traffic), so an all-BCM pair shows no SNR; green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run; framebuffer UI; harness.

Stashes

• stash@{0} — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
• stash@{1} — phydiag campaign: python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, campaign log. Working copies live in ~/work/phydiag-work/; the knowledge is in these docs; the code should eventually move into the repo properly.

The box

• Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
• X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
• Many CPU cores; goroutine-heavy designs welcome.

Interface	Device	Role (rules: hardware.md)
enp1s0f0	X520 port 0 (ixgbe)	Test pair — Wiitek module (SNR end)
enp1s0f1	X520 port 1 (ixgbe)	Test pair — FS module (length end)
enp3s0f0np0 / enp3s0f1np1	X710 (i40e)	Noise pair (has been enp4s0f* across reboots). The FS cannot live here: X710 firmware disables Rx/Tx on the honest FS EEPROM ("unsupported SFP module type") and i40e has no allow_unsupported_sfp — the noise pair needs lying modules. Seated: Fibergaga (f0) + Wiitek SN …X170U (f1), both qualified, noise cycle live
enp88s0	igc	LAN uplink, default route; sibling enp89s0 is dark

Hardware

Item	Status	Notes
X520-DA2	Installed in the single PCIe slot (E810 out); PCIe 5 GT/s ×8	Mixed test pair: port 0 Wiitek (SN WAMZ012606X039U), port 1 FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs allow_unsupported_sfp=1 — the FS trips qualification (hardware.md)
ConnectX-5	Ordered (dual SFP28, PCIe x8) — the product NIC candidate	mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (nics/connectx-5/). Arrival notes: ports may ship in InfiniBand mode (mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2); SFP+ drops into SFP28 cages at 10G; check mlx5_ib vs channel changes (the irdma lesson)
Replacement Wiiteks	One in the X520 test pair (port 0), rest on the shelf	Originals bricked by register exploration — modules/wiitek/ trap first
FS SFP-10G-T-100 ×2	One in the X520 test pair (port 1), one on the shelf (rejected by the X710 — ixgbe-only)	BCM84891L, documented, robust. The mixed FS+Wiitek pair is the configuration under evaluation: length from the FS ECD, IEEE per-pair SNR margins from the Wiitek — the best of both worlds a 2× FS pair cannot give (the BCM never populates IEEE SNR)
Fibergaga SFP-10G-T-30M	In the X710 noise pair (f0)	Aquantia, RollBall, the documented oracle
10Gtek	In hand	Claims SFP-10G-SR, still copper RJ45; filler, not in the test set
E810	Out of the box	Patched ice + sff_i2c remains useful only if it returns for read-side work

The X520 is disqualified as the product NIC — measurement, not a narrow gap. Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, nics/x520/). Per-queue counters were explored and rejected — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.

X520 diagnostics path — validated on hardware

In ~/work/ alongside the phydiag artifacts, ready to fold into the repo's kernel/:

• Patched ixgbe (~/work/ixgbe-sff/): ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + sff_i2c debugfs file (w <addr8> <bytes…> / r <addr8> <n> / x <waddr> <raddr> <delay_us> <n> <wbytes…> single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: ETQF steering (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in ixgbe_ethtool.c, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); reg_ops read-returns-value and a ts_bench debugfs command (in-kernel latch poll/re-arm for the timestamp bench pokes). load-ixgbe rebuilds + swaps the module, passing allow_unsupported_sfp=1.
• Bench tools (~/work/phydiag-work/): etqfbench/ (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), x520poke/ (Go; drives reg_ops/ts_bench for the timestamp/counter pokes), compound_test.py (proves the x compound op against the FS BCM), bcm_mcia_emu.py (proves the BCM bridge tolerates MCIA offset-write-then-read framing — nics/connectx-5/).
• BCM client (~/work/phydiag-work/bcm_ixgbe.py): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via CMD_GET_SNR = 0x8030 invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
• Exploration probes (bcm_explore.py, bcm_eee_off.py): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
• RollBall client (~/work/phydiag-work/rollball_ixgbe.py): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The *.0x??64 brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup plan

1. Card installed, modules seated, ./load-ixgbe, sff_i2c present. Done.
2. FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off. Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
5. RX steering: program/verify the ETQF path. Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (~/work/phydiag-work/etqfbench). system.go runs unchanged.

RX flow-steering on ixgbe

Solved on hardware. cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (ixgbe_ethtool.c), so system.go runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (~/work/phydiag-work/etqfbench). Full analysis in nics/x520/.

• ETQF/ETQS L2 EtherType queue filters — proven. Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
• Flow Director flex-byte match — ruled out on hardware. fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) only narrows an IP match.
• Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.

Open items

• BCM ECD works (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (phy.go). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
• Pre-FEC verification on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).