AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — open-questions.md §2 for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (`probe.go`); framebuffer UI; harness.
-`stash@{0}` — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
-`stash@{1}` — phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
## The box
- Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival.
- X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
- Many CPU cores; goroutine-heavy designs welcome.
| Interface | Device | Role (rules: hardware.md) |
|---|---|---|
| `enp1s0f0` | X520 port 0 (ixgbe) | Test pair — new Wiitek module |
| `enp1s0f1` | X520 port 1 (ixgbe) | Test pair — FS module |
| `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots) |
| `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark |
## Hardware
| Item | Status | Notes |
|---|---|---|
| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Port 0 `enp1s0f0` = new Wiitek (SN WAMZ012606X039U); port 1 `enp1s0f1` = FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) |
| ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (open-questions.md §6). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) |
| Replacement Wiiteks | Arrived; one in X520 port 0 | Originals bricked by register exploration — modules/wiitek/ trap first |
| FS SFP-10G-T-100 ×2 | In hand; one in X520 port 1 | BCM84891L, documented, robust |
| Fibergaga SFP-10G-T-30M | In hand | Aquantia, RollBall, the documented oracle |
| 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
| E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
**The X520 cannot be the product NIC**: rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement) and the 82599 stamps PTP only. It carries two interim campaigns: the diagnostics bring-up (certain arbitrary I2C framing) and the 82599 bucket-timestamping exploration (open-questions.md §2). If that exploration validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX.
In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`:
- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w <addr8> <bytes…>` / `r <addr8> <n>`). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`.
- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via `CMD_GET_SNR = 0x8030` invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
- **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
- **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill.
2.~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW — so the insert is rejected and the tool fails at startup (no fallback, by design). This affects the datapath regardless of module type.
Leads, most promising first (full analysis in open-questions.md §1):
- **ETQF/ETQS L2 EtherType queue filters — the answer on paper.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588); already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto ETQF slots so `system.go` is unchanged. Bench-confirm.
- **Flow Director flex-byte match — ruled out on paper.** fdir classifies IPv4/IPv6 only; the flex word narrows an IP match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding.
- **MAC → VMDq pool steering** — distinct dest MACs → pools → queues, fully raw. Fallback if ETQF disappoints.
- **RSS** — hashes IP tuples only; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline).
Fallbacks if raw-L2 steering can't work: minimal bare-IPv4 framing steered by IP_USER_FLOW, or single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable as a fallback (open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice.
- **ECD register chapter** from FS — the one missing document for BCM cable length; the wider ask list is in modules/fs/.
- **Pre-FEC verification** on the Aquantia — counters documented (modules/fibergaga/); needs the graded-noise correlation run (open-questions.md §5).
- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, short-cable calibration); scoped but unbuilt — the committed `probe.go` is the *filter-all* variant (raw-frame probes, needs all-packet rx stamping, E810-only); the X710/X520 variant means PTP-shaped probes. Superseded for the product if PHY DSP length pans out.
