Current state

Committed tree

AF_PACKET raw sockets everywhere (sock.go); flow-director steering; per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL as a hard host check — nics/README.md for what that demands of the NIC; temporarily bypassed in ts.go so BCM work can run on the X520, which cannot stamp — the check reports yellow and the panel rates read zero there; restore to fatal for the product NIC); read-time-stamped NIC-counter rates; test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix (the modules' jumbo path is exercised, not assumed); BCM module diagnostics (phy.go, over the patched-ixgbe sff_i2c debugfs, compound-op framing; one loop goroutine per module owns every transport touch, and all work is time-division-multiplexed with the firmware's own mailbox client — each request admitted only inside a 3.4 s edge-locked window after an observed internal temp poll, re-acquiring phase whenever it was lost — which ends the stale-read poisoning of the corrected channel; corrected charges and link-down reads print their raw registers so events self-attribute — modules/fs/ for the full trap list and validation): bringup identifies both modules and forces EEE off, jumbo on and the master/slave roles (A master / B slave — left to AN the role is a per-training lottery) every boot — no trustworthy readback exists and no cable is guaranteed to probe through; the ECD — per-pair verdicts, lengths and pair maps are the length/wiring path — runs through one async path at startup and on every reset, never blocking the UI, with counters re-baselining only after the diag's own link blip so it is never charged to the run; a 1 Hz poller feeds per-pair SNR margin (vs the ≈26.5 dB operating point; green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run) and the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count) to the panel and console; framebuffer UI; harness.

Stashes

stash@{0} — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
stash@{1} — phydiag campaign: python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, campaign log. Working copies live in ~/work/phydiag-work/; the knowledge is in these docs; the code should eventually move into the repo properly.

The box

Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
Many CPU cores; goroutine-heavy designs welcome.

Interface	Device	Role (rules: hardware.md)
`enp1s0f0`	X520 port 0 (ixgbe)	Test pair — FS module
`enp1s0f1`	X520 port 1 (ixgbe)	Test pair — FS module
`enp3s0f0np0` / `enp3s0f1np1`	X710 (i40e)	Noise pair (has been `enp4s0f*` across reboots)
`enp88s0`	igc	LAN uplink, default route; sibling `enp89s0` is dark

Hardware

Item	Status	Notes
X520-DA2	Installed in the single PCIe slot (E810 out); PCIe 5 GT/s ×8	Both ports FS (port 1 SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the FS trips qualification (hardware.md)
ConnectX-5	Ordered (dual SFP28, PCIe x8) — the product NIC candidate	mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (nics/connectx-5/). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson)
Replacement Wiiteks	Arrived; on the shelf	Originals bricked by register exploration — modules/wiitek/ trap first
FS SFP-10G-T-100 ×2	Both in the X520 test pair	BCM84891L, documented, robust. 2× FS at both ends is the expected product module config unless mixed ends prove wanted
Fibergaga SFP-10G-T-30M	In hand	Aquantia, RollBall, the documented oracle
10Gtek	In hand	Claims SFP-10G-SR, still copper RJ45; filler, not in the test set
E810	Out of the box	Patched ice + `sff_i2c` remains useful only if it returns for read-side work

The X520 is disqualified as the product NIC — measurement, not a narrow gap. Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, nics/x520/). Per-queue counters were explored and rejected — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.

X520 diagnostics path — validated on hardware

In ~/work/ alongside the phydiag artifacts, ready to fold into the repo's kernel/:

Patched ixgbe (~/work/ixgbe-sff/): ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + sff_i2c debugfs file (w <addr8> <bytes…> / r <addr8> <n> / x <waddr> <raddr> <delay_us> <n> <wbytes…> single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: ETQF steering (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in ixgbe_ethtool.c, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); reg_ops read-returns-value and a ts_bench debugfs command (in-kernel latch poll/re-arm for the timestamp bench pokes). load-ixgbe rebuilds + swaps the module, passing allow_unsupported_sfp=1.
Bench tools (~/work/phydiag-work/): etqfbench/ (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), x520poke/ (Go; drives reg_ops/ts_bench for the timestamp/counter pokes), compound_test.py (proves the x compound op against the FS BCM), bcm_mcia_emu.py (proves the BCM bridge tolerates MCIA offset-write-then-read framing — nics/connectx-5/).
BCM client (~/work/phydiag-work/bcm_ixgbe.py): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via CMD_GET_SNR = 0x8030 invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
Exploration probes (bcm_explore.py, bcm_eee_off.py): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
RollBall client (~/work/phydiag-work/rollball_ixgbe.py): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The *.0x??64 brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup plan

~~Card installed, modules seated, ./load-ixgbe, sff_i2c present.~~ Done.
~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
~~RX steering: program/verify the ETQF path.~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (~/work/phydiag-work/etqfbench). system.go runs unchanged.

RX flow-steering on ixgbe

Solved on hardware. cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (ixgbe_ethtool.c), so system.go runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (~/work/phydiag-work/etqfbench). Full analysis in nics/x520/.

ETQF/ETQS L2 EtherType queue filters — proven. Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
Flow Director flex-byte match — ruled out on hardware. fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) only narrows an IP match.
Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.

Open items

BCM ECD works (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (phy.go). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
Pre-FEC verification on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).

10 KiB Raw Blame History Unescape Escape