Current state

Committed tree

AF_PACKET raw sockets everywhere (sock.go); flow-director steering; per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL as a hard host check — open-questions.md §2 for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (probe.go); framebuffer UI; harness.

Stashes

stash@{0} — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
stash@{1} — phydiag campaign: python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, campaign log. Working copies live in ~/work/phydiag-work/; the knowledge is in these docs; the code should eventually move into the repo properly.

The box

Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
Many CPU cores; goroutine-heavy designs welcome.

Interface	Device	Role (rules: hardware.md)
`enp1s0f0`	X520 port 0 (ixgbe)	Test pair — new Wiitek module
`enp1s0f1`	X520 port 1 (ixgbe)	Test pair — FS module
`enp3s0f0np0` / `enp3s0f1np1`	X710 (i40e)	Noise pair (has been `enp4s0f*` across reboots)
`enp88s0`	igc	LAN uplink, default route; sibling `enp89s0` is dark

Hardware

Item	Status	Notes
X520-DA2	Installed in the single PCIe slot (E810 out); PCIe 5 GT/s ×8	Port 0 `enp1s0f0` = new Wiitek (SN WAMZ012606X039U); port 1 `enp1s0f1` = FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the FS trips qualification (hardware.md)
ConnectX-5	Ordered (dual SFP28, PCIe x8) — the product NIC candidate	mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (open-questions.md §6). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson)
Replacement Wiiteks	Arrived; one in X520 port 0	Originals bricked by register exploration — modules/wiitek/ trap first
FS SFP-10G-T-100 ×2	In hand; one in X520 port 1	BCM84891L, documented, robust
Fibergaga SFP-10G-T-30M	In hand	Aquantia, RollBall, the documented oracle
10Gtek	In hand	Claims SFP-10G-SR, still copper RJ45; filler, not in the test set
E810	Out of the box	Patched ice + `sff_i2c` remains useful only if it returns for read-side work

The X520 is disqualified as the product NIC — measurement, not a narrow gap. Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, open-questions.md §2). Per-queue counters were explored and rejected — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.

X520 diagnostics path — validated on hardware

In ~/work/ alongside the phydiag artifacts, ready to fold into the repo's kernel/:

Patched ixgbe (~/work/ixgbe-sff/): ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + sff_i2c debugfs file (w <addr8> <bytes…> / r <addr8> <n> / x <waddr> <raddr> <delay_us> <n> <wbytes…> single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: ETQF steering (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in ixgbe_ethtool.c, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); reg_ops read-returns-value and a ts_bench debugfs command (in-kernel latch poll/re-arm for the §2 pokes). load-ixgbe rebuilds + swaps the module, passing allow_unsupported_sfp=1.
Bench tools (~/work/phydiag-work/): etqfbench/ (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), x520poke/ (Go; drives reg_ops/ts_bench for the timestamp/counter pokes), compound_test.py (proves the x compound op against the FS BCM).
BCM client (~/work/phydiag-work/bcm_ixgbe.py): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via CMD_GET_SNR = 0x8030 invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
Exploration probes (bcm_explore.py, bcm_eee_off.py): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
RollBall client (~/work/phydiag-work/rollball_ixgbe.py): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The *.0x??64 brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup plan

~~Card installed, modules seated, ./load-ixgbe, sff_i2c present.~~ Done.
~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
~~RX steering: program/verify the ETQF path (open-questions.md §1).~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (~/work/phydiag-work/etqfbench). system.go runs unchanged.

RX flow-steering on ixgbe

Solved on hardware. cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (ixgbe_ethtool.c), so system.go runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (~/work/phydiag-work/etqfbench). Full analysis in open-questions.md §1.

ETQF/ETQS L2 EtherType queue filters — proven. Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
Flow Director flex-byte match — ruled out on hardware. fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) only narrows an IP match.
Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.

Open items

BCM ECD works (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. Remaining work is characterizing the link blip the run causes (length is a between-runs operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
Pre-FEC verification on the Aquantia — counters documented (modules/fibergaga/); needs the graded-noise correlation run (open-questions.md §5).
X710 PTP path-delay length measurement: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, short-cable calibration); scoped but unbuilt — the committed probe.go is the filter-all variant (raw-frame probes, needs all-packet rx stamping, E810-only); the X710/X520 variant means PTP-shaped probes. Superseded for the product if PHY DSP length pans out.

8.5 KiB Raw Blame History Unescape Escape