Current state

Where the code is right now

The committed tree is the X520 product implementation: AF_PACKET datapath with read-time rate buckets and the backward excess-fill smear (nics/x520/, "The smoothed-bucket rate" — rate is a headline, never a verdict; smearWindow 4 buckets = 64 ms display latency, resize after measuring the stall distribution: open-questions), the mixed FS+Wiitek diagnostics with the BCM ECD length path, testDriver = ixgbe. The hardware-timestamp machinery is gone — the rx_filter=ALL check, the SO_TIMESTAMPING request and the per-frame cmsg parse (formerly ts.go) — after the audit confirmed the rate buckets were its sole consumer. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).

The stashed Wiitek/CX-5 implementation (parts bin)

All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.

AF_PACKET raw sockets everywhere (sock.go); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL, which the CX-5 satisfies natively); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (counters.go); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.

Module diagnostics (phy.go + phy_mcia.go): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe sff_i2c debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because /dev/fwctl is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).

Bringup is read-only for the Wiitek pair (both auto-resolve master/slave and advertise no EEE — verified, nothing forced), so it never drops the carrier and the run starts immediately with no wait for link. A ~1 Hz poller feeds per-pair SNR margin from both ends (each Wiitek's IEEE 1.133–136 addressed by its own card-global MCIA module index — the headline: worst pair across both ends) plus the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count); green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run. Framebuffer UI; harness.

Stashes

Indices shift as stashes are pushed/popped — match by message, not number.

"CX-5 Wiitek: MCIA transport…" (currently stash@{0}) — the whole Wiitek-pair rewrite: phy_mcia.go (VSC/ICMD MCIA transport), transport-interface phy.go, dropped BCM/ECD/length, ntuple + mlx5 counters, per-port card-global MCIA module index. Carries live debug instrumentation and rbCmdPoll lowered to 2 ms from the mailbox-latency investigation — not shippable as-is, clean it before committing. Also carries the pre-existing uncommitted ui.go/ui_test.go edits that were entangled in the same files.
"AF_XDP both directions…" (currently stash@{1}) — AF_XDP conversion (hand-rolled XSK + XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
"phydiag campaign…" (currently stash@{2}) — python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, campaign log. Working copies live in ~/work/phydiag-work/; the knowledge is in these docs; the code should eventually move into the repo properly.

The box

Single usable PCIe slot (Gen4 x8): holds the X520-DA2 (dual SFP+). Verify 5 GT/s ×8 trained at bringup (lspci -vv). The ConnectX-5 is out of the box.
X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
Many CPU cores; goroutine-heavy designs welcome.
mstflint (AUR — not in the Arch repos) provides mstlink/mstreg/mstconfig for the CX-5, if it ever returns.

Interface	Device	Role (rules: hardware.md)
`enp1s0f0` + second port	X520 (ixgbe)	Test pair — FS + Wiitek, dispatched by EEPROM PN so either port may hold either module. Stock ixgbe refuses the FS's honest EEPROM and its port has no netdev until `./load-ixgbe` (allow_unsupported_sfp=1 + the sff_i2c/ETQF patches); the tool needs exactly 2 ixgbe interfaces
`enp3s0f0np0` / `enp3s0f1np1`	X710 (i40e)	Noise pair (has been `enp4s0f*` across reboots) — two Wiiteks, both qualified by the X710. The FS cannot live here (firmware rejects the honest EEPROM, no `allow_unsupported_sfp` in i40e)
`enp88s0`	igc	LAN uplink, default route; sibling `enp89s0` is dark

Hardware

Item	Status	Notes
X520-DA2	Installed in the single PCIe slot — the product NIC (nics/x520/)	Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven; the smoothed-bucket rate design removes the need for hardware RX stamps
ConnectX-5	Out of the box — retired	Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status"
FS SFP-10G-T-100 ×2	One in the X520 test pair, one on the shelf	BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration
Replacement Wiiteks	One in the X520 test pair, two in the X710 noise pair, rest on the shelf	Originals bricked by register exploration — modules/wiitek/ trap first
Fibergaga SFP-10G-T-30M	On the shelf	Aquantia, RollBall, the documented oracle; the alternate length path if the pair mix ever changes
10Gtek	In hand	Claims SFP-10G-SR, still copper RJ45; filler, not in the test set
E810	Out of the box	Patched ice + `sff_i2c` remains useful only if it returns for read-side work

X520 diagnostics path — validated on hardware

In ~/work/ alongside the phydiag artifacts, ready to fold into the repo's kernel/:

Patched ixgbe (~/work/ixgbe-sff/): ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + sff_i2c debugfs file (w <addr8> <bytes…> / r <addr8> <n> / x <waddr> <raddr> <delay_us> <n> <wbytes…> single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: ETQF steering (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in ixgbe_ethtool.c, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); reg_ops read-returns-value and a ts_bench debugfs command (in-kernel latch poll/re-arm for the timestamp bench pokes). load-ixgbe rebuilds + swaps the module, passing allow_unsupported_sfp=1.
Bench tools (~/work/phydiag-work/): etqfbench/ (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), x520poke/ (Go; drives reg_ops/ts_bench for the timestamp/counter pokes), compound_test.py (proves the x compound op against the FS BCM), bcm_mcia_emu.py (proves the BCM bridge tolerates MCIA offset-write-then-read framing — nics/connectx-5/).
BCM client (~/work/phydiag-work/bcm_ixgbe.py): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via CMD_GET_SNR = 0x8030 invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
Exploration probes (bcm_explore.py, bcm_eee_off.py): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
RollBall client (~/work/phydiag-work/rollball_ixgbe.py): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The *.0x??64 brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup on the swapped box

./load-ixgbe — until then the FS port has no netdev (stock driver, error -95) and driverPair("ixgbe") cannot find 2 interfaces. Verify sff_i2c present for both ports and 5 GT/s ×8 trained.
~~Run.~~ Done — full pass on hardware: both modules identified and dispatched, ECD at startup (all pairs ok, ~42 m headline), steady 20.00–20.04 G both directions with zero lost/corrupt/link/internal over a 90 s run, SNR +7.4–7.5 dB, noise cycle live. SETTINGS self-reports the ixgbe shape (rx=8160/tx=4096 taken as-is; coalesce tx-shares-rx).
Measure the host read-stall distribution under load and resize smearWindow if 64 ms is mis-sized (open-questions).
Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.

RX flow-steering on ixgbe

Solved on hardware. cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (ixgbe_ethtool.c), so system.go runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (~/work/phydiag-work/etqfbench). Full analysis in nics/x520/.

ETQF/ETQS L2 EtherType queue filters — proven. Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
Flow Director flex-byte match — ruled out on hardware. fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) only narrows an IP match.
Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.

Standing goals

Length is a product goal again — the FS (ECD length) rides alongside the Wiitek (SNR) in the test pair, which the tree implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
BCM ECD works (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (phy.go). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then — open-questions).
Pre-FEC verification on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).

12 KiB Raw Blame History Unescape Escape