Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons

docs/README.md

@@ -0,0 +1,12 @@
+# cabletest docs
+
+Project knowledge that moves and versions with the repo.
+
+- [goals.md](goals.md) — what cabletest is for and what it must output.
+- [state.md](state.md) — where things are now: committed vs stashed, hardware in flight, bringup plan, open items.
+- [hardware.md](hardware.md) — the box, interfaces, host tuning, how to run.
+- [transports.md](transports.md) — how to reach module-PHY diagnostics, why the NIC choice governs it, and the register-level protocols.
+- [modules.md](modules.md) — per-module diagnostics, register maps, cable-length physics, and the Marvell brick hazard.
+- [measurement.md](measurement.md) — how to measure correctly; performance results and dead ends.
+
+Vendor PDFs (BCM84891L transport + command handler, Marvell 88E151x datasheet) live alongside these.

docs/goals.md

@@ -0,0 +1,16 @@
+# Goals
+
+cabletest finds Ethernet cables that misbehave under sustained full-duplex 10 Gbit load. It exchanges raw Ethernet frames (no IP, no ARP) across a variety of sizes and payloads between two directly-cabled ports on one host, and attributes every problem it can to the cable rather than the host. It runs as PID 1 on a dedicated appliance with a framebuffer UI; development happens on the same machine under Arch.
+
+The target of measurement is the cable, not throughput. Line rate is a means to stress the physical layer. The outputs that matter, roughly in order:
+
+1. **Loss and error attribution** — reception gaps, link errors, NIC/driver counters as first-class output alongside application loss. Baseline loss must be exactly zero before a run counts; any host-side loss masks real cable faults.
+2. **Noise tolerance** — a deliberately-bad "noise" cable intertwined with the test cable, driven by link up/down cycling, stresses the cable under test with alien crosstalk.
+3. **Per-pair SNR margin** from the module PHYs (IEEE 802.3an standard registers) — the leading indicator of a marginal cable before it drops frames.
+4. **Cable length** — sanity check and fault localization. Sources, in preference order:
+   - Module PHY DSP estimate — works on a linked cable.
+   - PHY TDR — localizes opens/shorts both-ended; healthy-cable length only single-ended.
+   - NIC timestamp path-delay — works linked, module-independent; needs per-packet or PTP-latch hardware timestamps and a short-cable calibration.
+5. **Pre-FEC error visibility** — corrected-error counters that move before post-FEC loss appears. Vendor-specific; still unlocated on the module PHYs we can talk to. Standard latched PCS counters (errored blocks, BER, block-lock loss) are the working proxy under noise stress.
+
+Design preferences that shaped the tool: plain sockets first, escalate only after measuring (AF_PACKET reached line rate on the shipping mix; the AF_XDP experiment is parked); pre-populated frame contents; goroutine-heavy is fine (many cores to burn); DPDK is off the table — it would kill the sysfs counters and link management the tester depends on.

docs/hardware.md

@@ -0,0 +1,39 @@
+# Hardware and host
+
+## The box
+Single usable PCIe slot (Gen4 x8, currently the E810 — being replaced by the X520). The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.
+
+## Interfaces
+- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order). Currently the X710 pair `enp4s0f0np0` / `enp4s0f1np1` (i40e); becomes the X520 pair when it arrives.
+- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. Was the i40e pair during the E810-test-path era.
+- **`enp89s0` (igc)**: this box's LAN uplink with the default route. Never repurpose or down it.
+- Both test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.
+
+## The media lies
+`ethtool` reports `Port: FIBRE` / `10000baseSR` on the test ports, and the modules' own EEPROMs claim fiber identities (SR, 850 nm, LC connector, multimode fiber lengths, even fake optical DOM). All false. **Every test module is a copper RJ45 10GBASE-T module with a cloned/lying EEPROM** — none are fiber. The cheap RJ45 SFP+ modules clone a real optical module's EEPROM to pass NIC compatibility checks, so a module whose part number reads `SFP-10G-SR` is still 10GBASE-T copper. The real media is a 10GBASE-T PHY inside each module.
+
+Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE.
+
+## Host tuning (resets every reboot; re-apply before trusting results)
+Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults.
+
+1. **CPU governor → `performance`** on all cores.
+   - Biggest single contributor.
+   - Under powersave, cores idle at 400 MHz and ramp too slowly, dropping frames at startup.
+2. **Disable adaptive coalescing.**
+   - `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`.
+   - Syntax/values differ on ixgbe — re-derive for the X520.
+3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair).
+   - Effect was not isolated in testing; don't claim it as a proven win.
+4. **Socket buffers** need no sysctl change.
+   - The tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) to bypass the 4 MB `rmem_max` clamp.
+
+Verify a clean run shows `tx frames == rx frames` exactly before a run counts.
+
+## irdma (dev host only)
+`irdma` autoloads and binds the ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active"), which fails the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.
+
+## Running
+Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a enp1s0f0np0 -b enp1s0f1np1`. It starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1) and do not follow every run with a pgrep orphan check — the group kill is the guarantee. The tell for two competing instances (an orphan, or the user's own hand-run) is `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs — ask rather than killing something that might be theirs. `./harness` also dumps the panel to `shots/<offset>.png` for remote assessment.
+
+Test runs use the **default config only** — no `-sizes`/`-streams` overrides, and no editing the hardcoded `frameSizes`/`numStreams`/`batchSize` (that's the same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.

docs/measurement.md

@@ -0,0 +1,29 @@
+# Measurement lessons
+
+Hard-won rules about measuring correctly. Violating these produces numbers that look plausible and are wrong.
+
+## Rate buckets: stamp at read time, never at ticker-fire time
+`direction.capture()` stamps each bucket with `time.Now()` taken immediately after reading the counters — never a timestamp passed in from the sampler ticker. A rate is Δcounters/Δt; the sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator, and because E[1/x] > 1/E[x] the error biases the rate **upward**, it doesn't cancel. Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high; read-time stamping read within 2%. A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it. Never share one timestamp across directions "so buckets share an instant" — nothing needs it and it reintroduces the skew.
+
+## NIC counters are not monotonic
+`/sys/class/net/*/statistics/*` run from boot, not process start, and reset to zero on driver stats resets. In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run, and a backward step underflows `now - base` to ~2^64. Accumulate only forward motion into a process-local total, then take every baseline from that total — never from a raw reading. Establish all baselines in one place before traffic starts. Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug.
+
+## Only zero-baseline-loss runs count; small frames measure the host
+Any nonzero baseline loss masks real cable faults, so a run counts only when `tx frames == rx frames` exactly with a clean cable. Small-frame runs are CPU/host-bound, not cable-bound, so cable conclusions drawn from them are false. Keep to the default config (see hardware.md) and judge from steady state — the first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) that can read far below line rate and is not residual error.
+
+## Measured performance (AF_PACKET committed path, ice/E810 era)
+- Full size mix at 7 flow-director streams: line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup.
+- 64 B only: pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops), so only ~3.8 Gb/s.
+- RX cannot be parallelized by RSS (hardware RSS on ice can't hash raw ethertypes) — Flow Director steering by ethertype to distinct queues is what gives multiple NAPI contexts. rxnfc/fdir programming has sharp edges; see the rxnfc notes below.
+- Dead ends, measured and not to be re-attempted without new hardware (the binding constraint is total CPU across ~28 goroutines on 20 threads):
+  - **Splitting tx senders from rx streams** — raises tx but collapses rx, since rx scales with queue count, capped at 7 on that NIC.
+  - **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at tx.
+  - **Batch sizes above 64** — no gain, worse loss.
+- The tx `sendmmsg` busy-spin on ENOBUFS is not worth chasing: it only exists when the tx ring is full, which means the wire is the ceiling, so recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
+
+The AF_XDP experiment (stashed) removed the tx frame-generation ceiling (64 B tx ~14.5 Mpps/dir) and moved the bottleneck to RX (~4 Mpps/dir on AF_PACKET, ~13.5 with AF_XDP RX). It required per-packet MAC rx timestamps for honest buckets, delivered via an XDP-metadata kfunc — available on the E810 datapath, not the X710. With the test path off the E810 this whole path is parked; read-time stamping covers bucketing regardless of NIC.
+
+## rxnfc / Flow Director programming (ice), three traps that each cost a debugging round
+1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement, so trusting its display gives an inverted rule that silently matches nothing.
+2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192) on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
+3. **ice rejects `RX_CLS_LOC_ANY`** with ENOSPC — allocate a free location yourself (capacity from `ETHTOOL_GRXCLSRLCNT`'s `data`; the CLI allocates downward from the top). When a rule inserts but steers nothing, diff the raw bytes of a CLI-made known-good rule against yours.

docs/modules.md

@@ -0,0 +1,55 @@
+# Module-PHY diagnostics and the register maps
+
+## DANGER: Marvell CUX3610 (Wiitek UF-RJ45-10G-100) modules brick on certain reads
+Both original Wiiteks were bricked **permanently** by register exploration — the death survives cold power cycles and reseats. Read this before touching any replacement.
+
+The CUX3610 has an integrated microcontroller (Marvell Alaska-M, PHY ID 0x002B0BF4, firmware 12.1.5.0) that boots from onboard non-volatile storage. The high vendor register windows (3.0x8000+, 31.0xF000+) reached over RollBall are **µC-mapped peripheral space, not PHY-core MDIO**. A read of a trapped peripheral crashes the µC mid-operation and corrupts its persistent boot state, so it comes up FATAL forever after.
+
+**The trap: any register whose low byte is 0x64 in a high window (register high-byte ≥ 0x80), on any devad.** Module 1 died reading 3.0x8064, module 2 reading 31.0xF064. The mailbox writes the address bytes, then the µC dies executing the access — the command never even needs to complete. 3.0x8064 is a known QCA808x CDT (cable-diagnostic) status register; the ODM firmware appears to emulate/intercept a CDT block there and crash on a cold read. A sloppy `(reg & 0xFF) == 0x64` decode explains both deaths with one bug.
+
+Rules for any Marvell/RollBall module:
+- **Never sweep vendor register windows. Never read `*.0x??64` (high-byte ≥ 0x80).**
+- Single targeted reads only, on registers with a documented reason.
+- Recovery after a brick needs a vendor firmware image + the boot-download protocol (registers 1.0xC050 boot status, 1.0xD0F0–D0F3 download) — we have no CUX3610 image, so bricks are effectively dead. A ghost i2c device at 7-bit 0x57 (returns zeros) is the ROM bootloader waiting for a download.
+
+## What the Marvell modules give (safely)
+Proven on the CUX3610 before they died, and on the Fibergaga oracle — all IEEE 802.3an standard registers, safe on any Marvell module:
+- **Per-pair SNR margin** — PMA 1.133–1.136 current, 1.137–1.140 minimum.
+  - On the CUX3610 the value is `reg − 0x8000` in 0.1 dB (saw A 9.1 / B 7.1 / C 7.7 / D 8.9 dB).
+  - Retrain-dependent — only compare within one training session.
+- **Latched error/retrain detection** — PCS 3.32/3.33.
+  - Block-lock loss, BER saturate, errored blocks; clear-on-read.
+- **Link-partner / master-slave** — 7.33.
+- **FW version** 1.0xC011/12; **boot status** 1.0xC050.
+- **Forcing a retrain** (to refresh SNR minimums):
+  - RollBall write `7.0 |= bit9` (restart AN) — proven safe.
+  - An `ip link` bounce does not drop the copper line, so it won't retrain.
+
+Not available in safe space: pre-FEC counter, cable length. Those live in the µC vendor space that killed the modules.
+
+## Marvell cable-length register candidates (unverified, single-shot only)
+From kernel drivers for sibling parts — templates for a *future* single targeted probe on a sacrificial module, never a sweep, never near `*.0x??64`:
+
+- **TDR** from the 88Q2220 (`marvell-88q2xxx.c`, same 0x002b0bXX family, recipe cited from "the Marvell API"), all in MMD 3:
+  - TDR_RESET 0xFECA — init 0x0D90, clear 0x1D90.
+  - Calibration: 0xFED9 = 0x010E, 0xFEDA = 0x00EB, 0xFEDE = 0x0058.
+  - TDR_STATUS 0xFEDD — write 0x0002 to start; ~500 ms test.
+  - Read TDR_STATUS: bits[1:0] = 01 done; [7:4] verdict (0x7 OK / 0xE open / 0x3 short / 0x5 noise); [15:8] distance in meters.
+  - Single-pair on the Q2220; the 4-pair CUX3610 likely has per-pair variants nearby. Low bytes are nowhere near the 0x64 trap.
+- **QCA-style CDT** (`qcom/qcom.h`):
+  - 3.0x8064 CDT_STATUS, 3.0x8065–0x8068 per-pair results, 3.0x8074–0x807E thresholds.
+  - This IS the trap address — only ever entered via the full start sequence, never a cold status read; treat as off-limits absent strong evidence.
+- **Next doc-mining lead** for the 4-pair pattern: `gtAdvVct.c` in the Marvell qd-dsdt GPL dumps (e.g. github.com/andy928/xpenology `arch/arm/plat-feroceon/mv_hal/qd-dsdt/`).
+
+## Cable-length physics (applies to both vendors)
+- **TDR** (classic VCT/CDT/ECD):
+  - Locates opens/shorts both-ended.
+  - But *healthy-cable length* needs a far-end reflection, and a plugged-in terminated far end absorbs the pulse — so "OK" pairs report no distance. Effectively single-ended for length.
+  - Every kernel implementation only reports distance for *fault* results.
+- **DSP estimate**:
+  - Both vendors' PHYs compute linked-cable length continuously from insertion loss during training (Marvell "proactively determines cable length"; the BCM's limited-reach power mode switches "based on cable length" while linked).
+  - Works in the product's both-ends-plugged topology — the open question is only whether the estimate is exposed in a readable register.
+- **NIC timestamp path-delay** — the module-independent fallback if the DSP length isn't reachable:
+  - Works linked.
+  - Needs per-packet or PTP-latch hardware timestamps.
+  - Needs a short-cable calibration to cancel fixed PHY latency (~4.8 ns/m one way; 50 m ≈ 240 ns).

docs/state.md

@@ -0,0 +1,26 @@
+# Current state
+
+## Committed tree
+AF_PACKET raw sockets everywhere (`sock.go`), flow-director steering, read-time-stamped rate buckets, framebuffer UI, harness. The zero-copy AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata) lives uncommitted in `stash@{0}`; it was built for the era when the test path ran on the E810, whose datapath delivers per-packet MAC timestamps. It is an experiment, not a requirement.
+
+`stash@{1}` holds the phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, and a STATUS.md campaign log. Working copies extracted to `~/work/phydiag-work/` during the module work. The knowledge from that campaign is now in these docs; the code should eventually move into the repo properly.
+
+## Hardware in flight
+- **Intel X520-DA2 ordered** — replaces the E810 in the box's single PCIe slot. Chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md.
+- **Replacement Wiitek (Marvell CUX3610) modules ordered.** Both original units were bricked by register exploration — see modules.md for the trap before ever touching one.
+- **In hand** (all copper RJ45 10GBASE-T modules with cloned/lying fiber EEPROMs — see hardware.md; none are actually fiber):
+  - 2× FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented diagnostics).
+  - 1× Fibergaga SFP-10G-T-30M (Aquantia, RollBall, the proven oracle module).
+  - 1× 10Gtek, EEPROM claims SFP-10G-SR — still a copper RJ45 module; filler, not part of the test set.
+
+## Bringup plan when the X520 arrives
+1. ixgbe debugfs patch exposing raw I2C transactions (model: the ice `sff_i2c` patch in `kernel/`; ixgbe has no firmware to refuse, so arbitrary framing works).
+2. FS/BCM modules first: prove the SMI transport against known-value registers, then the command handler (CMD_GET_SNR vs IEEE registers), on the test cable set.
+3. Replacement Wiiteks: IEEE-standard registers only (SNR, latches) unless/until a documented recipe exists for more; the VCT templates in modules.md are candidates for single-shot targeted probes on a sacrificial unit only.
+4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs) and re-verify the rxnfc flow-steering code against ixgbe's fdir.
+
+## Open items
+- **ECD register chapter**: the one missing document for BCM cable length. FS has been responsive; the narrow ask is "which registers invoke ECD / report the DSP cable-length estimate, and does it run with link up?"
+- **Pre-FEC classification** on the Aquantia (3.e820): needs a marginal channel; cabletest's noise stress will provide one once diagnostics are integrated.
+- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, 1 ft calibration cancels PHY latency); scoped but unbuilt. Superseded for the product if BCM ECD/DSP length pans out.
+- E810 leaves the box when the X520 arrives; the patched ice + `sff_i2c` remains useful only if an E810 returns for read-side work.

docs/transports.md

@@ -0,0 +1,76 @@
+# Module-PHY transports
+
+The diagnostics we want (SNR, cable length, error counters) live inside the SFP module's PHY, reachable only over the module's I2C sideband. Which NIC carries the test path decides whether the host can talk to that PHY at all. This is the single biggest architectural lever, and it was learned the hard way.
+
+## The rule
+A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. NICs whose firmware mediates I2C impose limits that break the module command protocols.
+
+| NIC (driver) | Module I2C | Notes |
+|---|---|---|
+| **X520 / 82599 (ixgbe)** | host bit-bangs the bus — full control | `ixgbe_phy.c` wiggles I2CCTL pins directly; no firmware in the path. A debugfs patch gives arbitrary transactions. **The chosen NIC.** |
+| **Wangxun WX1820 (txgbe)** | kernel `sff`/phylink — `/dev/i2c-N`, zero patches | Architecturally ideal, but no AF_XDP (irrelevant now the committed path is AF_PACKET); younger driver — flow-steering/counter parity unverified. |
+| **ConnectX-4/5 (mlx5)** | firmware MCIA, but writes allowed | Offset model: RollBall fits; BCM SMI read data-phase uncertain. Has per-packet HW timestamps (would restore the timestamp length path). |
+| **E810 (ice)** | multi-byte framing, but **writes EPERM-blocked** | Intel policy, blanket across all device addresses, survives NVM update. Reads fine. Dead for anything needing writes. |
+| **X710 (i40e)** | 1-byte-offset I2C only | AQ 0x0628/0x0629 EXTERNAL_MODULE; reg_address caps at 0xFF. Can't frame multi-byte SMI. Writes work but framing doesn't. |
+
+### How the dead ends were proven
+- **i40e** — 1-byte-offset only, can't frame the SMI:
+  - reg_address > 0xFF returns AQ retval 14 (hard boundary).
+  - Emits at most `[dev, offset, data]` (2 payload bytes); the BCM SMI needs 3–5 byte frames in one transaction.
+  - MDIO interface modes don't reach the module — no PHY on the NIC MDIO pins; copper SFPs wire only I2C.
+- **E810** — frames multi-byte fine, but writes are policy-blocked:
+  - topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. Reads work perfectly.
+  - Every write returns aq 1 (EPERM), tested identically against both the EEPROM address 0x50 and the BCM address 0x56 — so the block is write-vs-read, not address-scoped.
+  - NVM 5.01 hardened it: the silent-drop on 3.10 became an explicit EPERM.
+
+## BCM84891L (FS SFP-10G-T-100) — the documented, safe path
+Docs are in this folder (`10GBase-T...BCM84891.pdf` transport, `BCM84891L-MDIO Command Descriptions-.pdf` command handler). Broadcom PHY, robust — survived the whole poking campaign un-bricked, unlike the Marvell modules.
+
+**Transport (I2C→MDIO SMI bridge), PHY at I2C 7-bit 0x56 (8-bit 0xAC wr / 0xAD rd):**
+- Clause-45 write: I2C-write to 0x56 the 5-byte frame `[000+DevAD, RegH, RegL, DataH, DataL]`.
+- Clause-45 read, in two transactions:
+  - I2C-write `[001+DevAD, RegH, RegL]` to 0x56.
+  - Delay **>1 ms**.
+  - I2C-read 2 bytes.
+- Single-byte reads see it as inert (returns 0) — the multi-byte DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead.
+
+**MDIO Command Handler** — status-gated handshake, inherently safe (the opposite of blind register pokes). Registers in MMD 0x1E:
+- **Registers**: CMD 0x4005, STATUS 0x4037, DATA1–5 0x4038–0x403C.
+- **STATUS codes**:
+  - CMD_RECEIVED 0x0001
+  - IN_PROGRESS 0x0002
+  - COMPLETE_PASS 0x0004
+  - COMPLETE_ERROR 0x0008
+  - SYSTEM_BUSY 0xBBBB
+- **Procedure**:
+  1. Poll STATUS until idle (not IN_PROGRESS/BUSY).
+  2. Write params to DATAn.
+  3. Write (cmdcode | bit15) to CMD.
+  4. Poll STATUS for PASS/ERROR.
+  5. Read DATAn for results.
+  - Poll ~100 ms; STATUS is frozen up to 2 s during 10GBASE-T training, so only run after link up.
+- **Commands**:
+  - **CMD_GET_SNR** (GET commands 0x8000+; detail §1.25.1.24): DATA2/10..DATA5/10 = per-pair SNR dB (channels A–D).
+  - Also pair swap/skew/polarity, die temp, voltage, 1588.
+  - **Enhanced Cable Diagnostics** (shorts/opens/**cable length**) is a headline feature, but its invocation is NOT in the command-code table (0x8000–0x805B) — it's a separate ECD register mechanism whose chapter isn't in the docs FS sent. This is the one outstanding ask to FS.
+
+**Correct BCM SMI framing over the E810 topo-I2C** (recorded in case a write-capable multi-byte transport reappears): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, then read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`.
+
+## RollBall (Marvell/Aquantia modules) over i40e — the proven oracle path
+Stock, unpatched, works today on the X710 noise pair; the control experiment that proved our client logic correct.
+
+- **AQ transport** via `/sys/kernel/debug/i40e/<pci>/command`:
+  - Command form: `send aq_cmd <flags> <opcode> 0 0 0 0 <param0> <param1> <param2> 0`, flags 0x2000.
+  - Opcode 0x0629 get / 0x0628 set PHY register.
+  - param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value.
+  - Result comes back in the dmesg "AQ desc WB" line, 9th field.
+  - Single-byte writes work here (unlike the E810). This 1-byte model drives RollBall (all messages ≤4 B) but cannot frame the BCM SMI.
+- **RollBall sequence**:
+  - Unlock: password 0xFF×4 at A2h 0x7B.
+  - Page-select: A2h 0x7F → 3.
+  - Mailbox: A2h 0x80 (cmd) / 0x81 (data), poll for DONE (0x04).
+  - This is a clause-45 read/write of the internal PHY. On ixgbe with true multi-byte I2C it becomes the kernel's own `mdio-i2c` path — cleaner than the i40e byte-at-a-time framing.
+- **Proven on the Fibergaga SFP-10G-T-30M** (Aquantia AQR, PHY ID 0x31c31c13), with link up:
+  - Per-pair SNR at PMA 1.133–136.
+  - PCS block-lock and errored-block counters at 3.32/3.33.
+  - `3.e820` is a clear-on-read error counter — pre- vs post-FEC unclassified, needs a marginal channel to move it.