cabletest/docs/state.md

Current state

Committed tree

AF_PACKET raw sockets everywhere (sock.go), flow-director steering, per-packet-MAC-rx-stamped rate buckets (SO_TIMESTAMPING cmsg, rx_filter=ALL as a hard host check — see open-questions.md §2 for what that demands of the NIC), read-time-stamped NIC-counter rates, hardware-timestamped length probes (probe.go), framebuffer UI, harness. The zero-copy AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata) lives uncommitted in stash@{0}; it was built for the era when the test path ran on the E810, whose datapath delivers per-packet MAC timestamps. It is an experiment, not a requirement.

stash@{1} holds the phydiag campaign: python transport clients (i40e_aq.py, rollball_i40e.py, sff.py), scanners, and a STATUS.md campaign log. Working copies extracted to ~/work/phydiag-work/ during the module work. The knowledge from that campaign is now in these docs; the code should eventually move into the repo properly.

Hardware in flight

• Intel X520-DA2 installed in the box's single PCIe slot (E810 out; PCIe trained 5 GT/s ×8). Port 0 enp1s0f0 holds a new Wiitek (SN WAMZ012606X039U), port 1 enp1s0f1 the FS (SN S2433774168), cable between them linked at 10G. Stock ixgbe runs with allow_unsupported_sfp=1 — mandatory, and it's the FS that needs it (honest 10GBASE-T EEPROM fails Intel qualification; see hardware.md). The patched sff_i2c driver is not yet loaded. The X520 was chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md.
• ConnectX-5 ordered (dual SFP28, PCIe x8) — the product NIC candidate. Rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement); the 82599 stamps PTP frames only, and with raw-L2 steering relaxed (encap acceptable), mlx5 is the one driver that meets the full set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: the MCIA diagnostics questions in open-questions.md §6 (BCM at I2C 0x56, SMI read data phase). On arrival: ports may ship in InfiniBand mode (mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2), SFP+ modules drop into SFP28 cages at 10G, and check whether mlx5_ib autoloading interferes with channel changes (the irdma lesson).
• The X520 arrives first and carries two campaigns: the diagnostics bring-up (certain arbitrary I2C framing for proving the BCM/RollBall clients), and the 82599 bucket-timestamping exploration in open-questions.md §2 — per-queue hardware counters as bucket content plus latch-anchor sampling. If that validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX.
• Replacement Wiitek (Marvell CUX3610) modules arrived — one is in X520 port 0. Both original units were bricked by register exploration — see modules.md for the trap before ever touching one.
• In hand (all copper RJ45 10GBASE-T modules with cloned/lying fiber EEPROMs — see hardware.md; none are actually fiber):
  • 2× FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented diagnostics).
  • 1× Fibergaga SFP-10G-T-30M (Aquantia, RollBall, the proven oracle module).
  • 1× 10Gtek, EEPROM claims SFP-10G-SR — still a copper RJ45 module; filler, not part of the test set.

X520 prep — built ahead of the card, untested (no hardware yet)

In ~/work/ alongside the ice-sff/phydiag-work artifacts, ready to fold into the repo's kernel/ once validated on real hardware:

• Patched ixgbe (~/work/ixgbe-sff/, built against the running kernel, vermagic matches): adds ixgbe_i2c_raw_write/ixgbe_i2c_raw_read in ixgbe_phy.c (arbitrary START…STOP transactions built from the existing bit-bang primitives, swfw-semaphore bracketed) and an sff_i2c debugfs file with w <addr8> <bytes…> / r <addr8> <n> commands, result read back from the fd. Modeled on the proven ice sff_i2c; ixgbe has no firmware to refuse, so arbitrary framing works. load-ixgbe rebuilds + swaps the module.
• BCM client (~/work/phydiag-work/bcm_ixgbe.py): BCM SMI framing on those two ops (read = write [001+devad,RegH,RegL], >1 ms delay, read 2 B; write = 5-byte frame) plus the full command handler (_wait_idle → DATA → CMD|bit15 → poll PASS/ERROR → read DATA), CMD_GET_SNR decoded to per-pair dB, and EEPROM/PHY-ID/STATUS sanity reads.
• RollBall client for the Wiiteks (~/work/phydiag-work/rollball_ixgbe.py): the same sff_i2c transport, RollBall unlock/page/mailbox, per-pair IEEE SNR (PMA 1.133–1.136). The *.0x??64 (high-byte ≥ 0x80) brick blacklist is a hard guard that raises before touching hardware — the client structurally cannot repeat the kill.

Bringup plan when the X520 arrives

1. Card installed and modules seated (done). Next: ./load-ixgbe with allow_unsupported_sfp=1 (mandatory — the FS's honest 10GBASE-T EEPROM fails Intel qualification and kills the port probe; ixgbe_main.c:165), confirm sff_i2c appears.
2. FS/BCM modules first: bcm_ixgbe.py sanity reads (EEPROM, PHY ID, STATUS) to prove the SMI transport, then the command handler (CMD_GET_SNR vs the IEEE registers) on the test cable set.
3. Replacement Wiiteks: IEEE-standard registers only (SNR, latches) unless/until a documented recipe exists for more; the VCT templates in modules.md are candidates for single-shot targeted probes on a sacrificial unit only.
4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).

Open: RX flow-steering on ixgbe needs serious investigation before the X520 carries test traffic

cabletest fans RX across queues by steering raw ethertype (0x88b5 base) to distinct queues with Flow Director rules (system.go, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). The obvious path fails: ixgbe/82599's ixgbe_flowspec_to_flow_type accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW — so an ETHER_FLOW rule insert is rejected (EINVAL) and the tool fails at startup (no fallback, by design). This is a datapath issue affecting both module types, independent of diagnostics.

But 82599 has hardware steering machinery the current code doesn't use, and there is good reason to think we can keep true raw-L2 ethertype steering without falling back to encapsulation. This needs bench investigation on the actual card; do not assume the answer is UDP encap. Leads, most promising first (full analysis in open-questions.md §1):

• L2 EtherType queue filters (ETQF/ETQS) — the answer on paper. A dedicated ethertype→RX-queue stage checked at L2 ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588), already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots so system.go works unchanged. Bench-confirm on the card.
• Flow Director flex-byte match — ruled out on paper. 82599 fdir classifies IPv4/IPv6 only; the flex word narrows an IP-typed match but can't create a non-IP one, FDIRCTRL_FLEX_SHIFT = 0x6 (offset 12 = ethertype) notwithstanding — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test.
• MAC → VMDq pool steering. 82599 assigns unicast MAC (RAR) entries to VMDq pools and pools to RX queues (hw_set_rar VMDq pool/queue selection). Distinct dest MACs per stream → distinct queues, fully raw Ethernet, no IP at all. Fallback if ETQF disappoints.
• RSS on a flexible field — 82599 RSS hashes IP tuples, not raw L2; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline).

Fallbacks if raw-L2 steering genuinely can't be made to work: minimal bare-IPv4 framing (no UDP) steered by IP_USER_FLOW src/dst IP (lighter than the old UDP-encap plan), or single-queue RX (caps near the old ~1.6 Mpps single-NAPI ceiling, loses small-frame headroom and per-stream isolation). Encap is acceptable as a fallback (see open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice.

Open items

• ECD register chapter: the one missing document for BCM cable length. FS has been responsive; the narrow ask is "which registers invoke ECD / report the DSP cable-length estimate, and does it run with link up?"
• Pre-FEC classification on the Aquantia (3.e820): needs a marginal channel; cabletest's noise stress will provide one once diagnostics are integrated.
• X710 PTP path-delay length measurement: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, 1 ft calibration cancels PHY latency); scoped but unbuilt — the committed probe.go is the filter-all variant of the same idea (raw-frame probes, needs all-packet rx stamping, so E810-only); the X710/X520 variant means reshaping probes as PTP frames. Superseded for the product if BCM ECD/DSP length pans out.
• E810 leaves the box when the X520 arrives; the patched ice + sff_i2c remains useful only if an E810 returns for read-side work.