Current state

Where the code is right now

The committed tree is the X520 product implementation: AF_PACKET datapath with read-time rate buckets and the backward excess-fill smear (nics/x520/, "The smoothed-bucket rate" — rate is a headline, never a verdict; smearWindow 4 buckets = 64 ms display latency, resize after measuring the stall distribution: open-questions), the mixed FS+Wiitek diagnostics with the BCM ECD length path, testDriver = ixgbe. The hardware-timestamp machinery is gone — the rx_filter=ALL check, the SO_TIMESTAMPING request and the per-frame cmsg parse (formerly ts.go) — after the audit confirmed the rate buckets were its sole consumer. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).

The stashed Wiitek/CX-5 implementation (parts bin)

All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.

AF_PACKET raw sockets everywhere (sock.go); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL, which the CX-5 satisfies natively); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (counters.go); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.

Module diagnostics (phy.go + phy_mcia.go): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe sff_i2c debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because /dev/fwctl is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).

Bringup is read-only for the Wiitek pair (both auto-resolve master/slave and advertise no EEE — verified, nothing forced), so it never drops the carrier and the run starts immediately with no wait for link. A ~1 Hz poller feeds per-pair SNR margin from both ends (each Wiitek's IEEE 1.133–136 addressed by its own card-global MCIA module index — the headline: worst pair across both ends) plus the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count); green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run. Framebuffer UI; harness.

Stashes

Indices shift as stashes are pushed/popped — match by message, not number.

"CX-5 Wiitek: MCIA transport…" (currently stash@{0}) — the whole Wiitek-pair rewrite: phy_mcia.go (VSC/ICMD MCIA transport), transport-interface phy.go, dropped BCM/ECD/length, ntuple + mlx5 counters, per-port card-global MCIA module index. Carries live debug instrumentation and rbCmdPoll lowered to 2 ms from the mailbox-latency investigation — not shippable as-is, clean it before committing. Also carries the pre-existing uncommitted ui.go/ui_test.go edits that were entangled in the same files.
"AF_XDP both directions…" (currently stash@{1}) — AF_XDP conversion (hand-rolled XSK + XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
"phydiag campaign…" (currently stash@{2}) — python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, campaign log. Working copies live in ~/work/phydiag-work/; the knowledge is in these docs; the code should eventually move into the repo properly.

The box

Single usable PCIe slot (Gen4 x8): holds the X520-DA2 (dual SFP+). Verify 5 GT/s ×8 trained at bringup (lspci -vv). The ConnectX-5 is out of the box.
X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
Many CPU cores; goroutine-heavy designs welcome.
mstflint (AUR — not in the Arch repos) provides mstlink/mstreg/mstconfig for the CX-5, if it ever returns.

Interface	Device	Role (rules: hardware.md)
`enp1s0f0` + second port	X520 (ixgbe)	Test pair — FS + Wiitek, dispatched by EEPROM PN so either port may hold either module. Stock ixgbe refuses the FS's honest EEPROM and its port has no netdev until `./load-ixgbe` (allow_unsupported_sfp=1 + the sff_i2c/ETQF patches); the tool needs exactly 2 ixgbe interfaces
`enp3s0f0np0` / `enp3s0f1np1`	X710 (i40e)	Noise pair (has been `enp4s0f*` across reboots) — two Wiiteks, both qualified by the X710. The FS cannot live here (firmware rejects the honest EEPROM, no `allow_unsupported_sfp` in i40e)
`enp88s0`	igc	LAN uplink, default route; sibling `enp89s0` is dark

Hardware

Item	Status	Notes
X520-DA2	Installed in the single PCIe slot — the product NIC (nics/x520/)	Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven; the smoothed-bucket rate design removes the need for hardware RX stamps
ConnectX-5	Out of the box — retired	Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status"
FS SFP-10G-T-100 ×2	One in the X520 test pair, one on the shelf	BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration
Replacement Wiiteks	One in the X520 test pair, two in the X710 noise pair, rest on the shelf	Originals bricked by register exploration — modules/wiitek/ trap first
Fibergaga SFP-10G-T-30M	On the shelf	Aquantia, RollBall, the documented oracle; the alternate length path if the pair mix ever changes
10Gtek	In hand	Claims SFP-10G-SR, still copper RJ45; filler, not in the test set
E810	Out of the box	Patched ice + `sff_i2c` remains useful only if it returns for read-side work

X520 diagnostics path — validated on hardware

In ~/work/ alongside the phydiag artifacts, ready to fold into the repo's kernel/:

Patched ixgbe (~/work/ixgbe-sff/): ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + sff_i2c debugfs file (w <addr8> <bytes…> / r <addr8> <n> / x <waddr> <raddr> <delay_us> <n> <wbytes…> single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: ETQF steering (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in ixgbe_ethtool.c, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); reg_ops read-returns-value and a ts_bench debugfs command (in-kernel latch poll/re-arm for the timestamp bench pokes). load-ixgbe rebuilds + swaps the module, passing allow_unsupported_sfp=1.
Bench tools (~/work/phydiag-work/): etqfbench/ (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), x520poke/ (Go; drives reg_ops/ts_bench for the timestamp/counter pokes), compound_test.py (proves the x compound op against the FS BCM), bcm_mcia_emu.py (proves the BCM bridge tolerates MCIA offset-write-then-read framing — nics/connectx-5/).
BCM client (~/work/phydiag-work/bcm_ixgbe.py): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via CMD_GET_SNR = 0x8030 invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
Exploration probes (bcm_explore.py, bcm_eee_off.py): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
RollBall client (~/work/phydiag-work/rollball_ixgbe.py): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The *.0x??64 brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup on the swapped box

./load-ixgbe — until then the FS port has no netdev (stock driver, error -95) and driverPair("ixgbe") cannot find 2 interfaces. Verify sff_i2c present for both ports and 5 GT/s ×8 trained.
~~Run.~~ Done — full pass on hardware: both modules identified and dispatched, ECD at startup (all pairs ok, ~42 m headline), steady 20.00–20.04 G both directions with zero lost/corrupt/link/internal over a 90 s run, SNR +7.4–7.5 dB, noise cycle live. SETTINGS self-reports the ixgbe shape (rx=8160/tx=4096 taken as-is; coalesce tx-shares-rx).
Measure the host read-stall distribution under load and resize smearWindow if 64 ms is mis-sized (open-questions).
Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.

RX flow-steering on ixgbe

Solved on hardware. cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (ixgbe_ethtool.c), so system.go runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (~/work/phydiag-work/etqfbench). Full analysis in nics/x520/.

ETQF/ETQS L2 EtherType queue filters — proven. Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
Flow Director flex-byte match — ruled out on hardware. fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) only narrows an IP match.
Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.

Standing goals

Length is a product goal again — the FS (ECD length) rides alongside the Wiitek (SNR) in the test pair, which the tree implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
BCM ECD works (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset; while a measure runs, every failure counter is suppressed at its source (worker error adds, loss-window write-offs, NIC-poller accumulation, socket drops, module latch priming), so the blip is never counted anywhere rather than counted and reverted (phy.go cableDiag.measuring). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then — open-questions).
Pre-FEC verification on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).

12 KiB Raw Blame History Unescape Escape