Current state

Committed tree

AF_PACKET raw sockets everywhere (sock.go); flow-director steering; per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL as a hard host check — open-questions.md §2 for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (probe.go); framebuffer UI; harness.

Stashes

stash@{0} — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
stash@{1} — phydiag campaign: python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, campaign log. Working copies live in ~/work/phydiag-work/; the knowledge is in these docs; the code should eventually move into the repo properly.

The box

Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
Many CPU cores; goroutine-heavy designs welcome.

Interface	Device	Role (rules: hardware.md)
`enp1s0f0`	X520 port 0 (ixgbe)	Test pair — new Wiitek module
`enp1s0f1`	X520 port 1 (ixgbe)	Test pair — FS module
`enp3s0f0np0` / `enp3s0f1np1`	X710 (i40e)	Noise pair (has been `enp4s0f*` across reboots)
`enp88s0`	igc	LAN uplink, default route; sibling `enp89s0` is dark

Hardware

Item	Status	Notes
X520-DA2	Installed in the single PCIe slot (E810 out); PCIe 5 GT/s ×8	Port 0 `enp1s0f0` = new Wiitek (SN WAMZ012606X039U); port 1 `enp1s0f1` = FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the FS trips qualification (hardware.md)
ConnectX-5	Ordered (dual SFP28, PCIe x8) — the product NIC candidate	mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (open-questions.md §6). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson)
Replacement Wiiteks	Arrived; one in X520 port 0	Originals bricked by register exploration — modules/wiitek/ trap first
FS SFP-10G-T-100 ×2	In hand; one in X520 port 1	BCM84891L, documented, robust
Fibergaga SFP-10G-T-30M	In hand	Aquantia, RollBall, the documented oracle
10Gtek	In hand	Claims SFP-10G-SR, still copper RJ45; filler, not in the test set
E810	Out of the box	Patched ice + `sff_i2c` remains useful only if it returns for read-side work

The X520 cannot be the product NIC: rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement) and the 82599 stamps PTP only. It carries two interim campaigns: the diagnostics bring-up (certain arbitrary I2C framing) and the 82599 bucket-timestamping exploration (open-questions.md §2). If that exploration validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX.

X520 diagnostics path — validated on hardware

In ~/work/ alongside the phydiag artifacts, ready to fold into the repo's kernel/:

Patched ixgbe (~/work/ixgbe-sff/): ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + sff_i2c debugfs file (w <addr8> <bytes…> / r <addr8> <n>). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read now releases SDA after each ACK, mirroring stock's own release block. load-ixgbe rebuilds + swaps the module, passing allow_unsupported_sfp=1.
BCM client (~/work/phydiag-work/bcm_ixgbe.py): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via CMD_GET_SNR = 0x8030 invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
Exploration probes (bcm_explore.py, bcm_eee_off.py): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
RollBall client (~/work/phydiag-work/rollball_ixgbe.py): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The *.0x??64 brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup plan

~~Card installed, modules seated, ./load-ixgbe, sff_i2c present.~~ Done.
~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
RX steering: program/verify the ETQF path (open-questions.md §1) before cabletest traffic runs.

RX flow-steering on ixgbe

cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). ixgbe's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW — so the insert is rejected and the tool fails at startup (no fallback, by design). This affects the datapath regardless of module type.

Leads, most promising first (full analysis in open-questions.md §1):

ETQF/ETQS L2 EtherType queue filters — the answer on paper. Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588); already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto ETQF slots so system.go is unchanged. Bench-confirm.
Flow Director flex-byte match — ruled out on paper. fdir classifies IPv4/IPv6 only; the flex word narrows an IP match but can't create a non-IP one, FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) notwithstanding.
MAC → VMDq pool steering — distinct dest MACs → pools → queues, fully raw. Fallback if ETQF disappoints.
RSS — hashes IP tuples only; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline).

Fallbacks if raw-L2 steering can't work: minimal bare-IPv4 framing steered by IP_USER_FLOW, or single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable as a fallback (open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice.

Open items

BCM ECD works (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. Remaining work is characterizing the link blip the run causes (length is a between-runs operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
Pre-FEC verification on the Aquantia — counters documented (modules/fibergaga/); needs the graded-noise correlation run (open-questions.md §5).
X710 PTP path-delay length measurement: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, short-cable calibration); scoped but unbuilt — the committed probe.go is the filter-all variant (raw-frame probes, needs all-packet rx stamping, E810-only); the X710/X520 variant means PTP-shaped probes. Superseded for the product if PHY DSP length pans out.

7.8 KiB Raw Blame History Unescape Escape