AF_PACKET raw sockets everywhere (`sock.go`), flow-director steering, per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — see open-questions.md §2 for what that demands of the NIC), read-time-stamped NIC-counter rates, hardware-timestamped length probes (`probe.go`), framebuffer UI, harness. The zero-copy AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata) lives uncommitted in `stash@{0}`; it was built for the era when the test path ran on the E810, whose datapath delivers per-packet MAC timestamps. It is an experiment, not a requirement.
`stash@{1}` holds the phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, and a STATUS.md campaign log. Working copies extracted to `~/work/phydiag-work/` during the module work. The knowledge from that campaign is now in these docs; the code should eventually move into the repo properly.
- **Intel X520-DA2 installed** in the box's single PCIe slot (E810 out; PCIe trained 5 GT/s ×8). Port 0 `enp1s0f0` holds a new Wiitek (SN WAMZ012606X039U), port 1 `enp1s0f1` the FS (SN S2433774168), cable between them linked at 10G. Stock ixgbe runs with `allow_unsupported_sfp=1` — mandatory, and it's the *FS* that needs it (honest 10GBASE-T EEPROM fails Intel qualification; see hardware.md). The patched `sff_i2c` driver is not yet loaded. The X520 was chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md.
- **ConnectX-5 ordered (dual SFP28, PCIe x8) — the product NIC candidate.** Rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement); the 82599 stamps PTP frames only, and with raw-L2 steering relaxed (encap acceptable), mlx5 is the one driver that meets the full set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: the MCIA diagnostics questions in open-questions.md §6 (BCM at I2C 0x56, SMI read data phase). On arrival: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`), SFP+ modules drop into SFP28 cages at 10G, and check whether `mlx5_ib` autoloading interferes with channel changes (the irdma lesson).
- **The X520 arrives first and carries two campaigns**: the diagnostics bring-up (certain arbitrary I2C framing for proving the BCM/RollBall clients), and the 82599 bucket-timestamping exploration in open-questions.md §2 — per-queue hardware counters as bucket content plus latch-anchor sampling. If that validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX.
- **Replacement Wiitek (Marvell CUX3610) modules arrived** — one is in X520 port 0. Both original units were bricked by register exploration — see modules.md for the trap before ever touching one.
In `~/work/` alongside the ice-sff/phydiag-work artifacts, ready to fold into the repo's `kernel/`:
- **Patched ixgbe** (`~/work/ixgbe-sff/`): adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions built from the existing bit-bang primitives, swfw-semaphore bracketed) and an `sff_i2c` debugfs file with `w <addr8> <bytes…>` / `r <addr8> <n>` commands, result read back from the fd. **Proven on the card after one real bug**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible until now); `raw_read` now releases SDA after each ACK, mirroring the release block stock uses after clocking out a byte. `load-ixgbe` rebuilds + swaps the module and passes `allow_unsupported_sfp=1` (mandatory for the FS).
- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): BCM SMI framing proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, and per-pair SNR ≈ [32, 27.5, 30, 27.6] dB on the bench cable via `CMD_GET_SNR = 0x8030` invoked with no DATA1 write (see transports.md for the code-table trap and the pause-mode side quest).
- **RollBall client for the Wiiteks** (`~/work/phydiag-work/rollball_ixgbe.py`): the same `sff_i2c` transport, RollBall unlock/page/mailbox, per-pair IEEE SNR (PMA 1.133–1.136), untested on this card. The `*.0x??64` (high-byte ≥ 0x80) brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill.
1. Card installed, modules seated, `./load-ixgbe` with `allow_unsupported_sfp=1` (mandatory — the FS's honest 10GBASE-T EEPROM fails Intel qualification and kills the port probe; `ixgbe_main.c:165`), `sff_i2c` present. Done.
2. FS/BCM proven: SMI transport, EEPROM/PHY-ID/STATUS sanity, command handler (temp, voltage, per-pair SNR). The IEEE-register SNR path is dead on the BCM — the command handler is the source. Done.
3. Replacement Wiiteks: IEEE-standard registers only (SNR, latches) unless/until a documented recipe exists for more; the VCT templates in modules.md are candidates for single-shot targeted probes on a sacrificial unit only.
## Open: RX flow-steering on ixgbe needs serious investigation before the X520 carries test traffic
cabletest fans RX across queues by steering **raw ethertype** (0x88b5 base) to distinct queues with Flow Director rules (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). The obvious path fails: ixgbe/82599's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — **no ETHER_FLOW** — so an ETHER_FLOW rule insert is rejected (EINVAL) and the tool fails at startup (no fallback, by design). This is a datapath issue affecting *both* module types, independent of diagnostics.
But 82599 has hardware steering machinery the current code doesn't use, and **there is good reason to think we can keep true raw-L2 ethertype steering without falling back to encapsulation.** This needs bench investigation on the actual card; do not assume the answer is UDP encap. Leads, most promising first (full analysis in open-questions.md §1):
- **L2 EtherType queue filters (ETQF/ETQS) — the answer on paper.** A dedicated ethertype→RX-queue stage checked at L2 ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588), already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots so `system.go` works unchanged. Bench-confirm on the card.
- **Flow Director flex-byte match — ruled out on paper.** 82599 fdir classifies IPv4/IPv6 only; the flex word narrows an IP-typed match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test.
- **MAC → VMDq pool steering.** 82599 assigns unicast MAC (RAR) entries to VMDq pools and pools to RX queues (`hw_set_rar` VMDq pool/queue selection). Distinct dest MACs per stream → distinct queues, fully raw Ethernet, no IP at all. Fallback if ETQF disappoints.
- **RSS on a flexible field** — 82599 RSS hashes IP tuples, not raw L2; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline).
Fallbacks if raw-L2 steering genuinely can't be made to work: minimal bare-IPv4 framing (no UDP) steered by IP_USER_FLOW src/dst IP (lighter than the old UDP-encap plan), or single-queue RX (caps near the old ~1.6 Mpps single-NAPI ceiling, loses small-frame headroom and per-stream isolation). Encap is acceptable as a fallback (see open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice.
- **ECD register chapter**: the one missing document for BCM cable length. FS has been responsive; the narrow ask is "which registers invoke ECD / report the DSP cable-length estimate, and does it run with link up?"
- **Pre-FEC classification** on the Aquantia (3.e820): needs a marginal channel; cabletest's noise stress will provide one once diagnostics are integrated.
- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, 1 ft calibration cancels PHY latency); scoped but unbuilt — the committed `probe.go` is the *filter-all* variant of the same idea (raw-frame probes, needs all-packet rx stamping, so E810-only); the X710/X520 variant means reshaping probes as PTP frames. Superseded for the product if BCM ECD/DSP length pans out.
