cabletest/docs/open-questions.md

# Open questions

The genuinely open, thinking-worthy problems — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `transports.md`, `modules.md`, `measurement.md`); this assumes that context.

Short framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. We're swapping the E810 out for an Intel X520-DA2 (82599, `ixgbe`) so the host can bit-bang the SFP module I2C bus (the E810/i40e firmware blocked or couldn't frame it). Diagnostics clients and a raw-i2c kernel patch are prebuilt but untested. Test modules: FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented) and replacement Wiiteks (Marvell CUX3610, brick-prone). All modules are copper lying as fiber.

## 1. Raw-L2 RX steering on 82599 (gates whether cabletest runs on the X520 at all)
The tool steers raw ethertype 0x88b5 to per-queue NAPI contexts via Flow Director; ixgbe rejects ETHER_FLOW. But `FDIRCTRL_FLEX_SHIFT` is already set to 0x6 = byte offset 12 = the ethertype field, so the hardware samples it. Open: will a non-IP frame classify into an fdir flow_type the rule engine matches, and does pairing `flex_bytes` with a permissive base flow type need a driver change? Alternative: MAC→VMDq-pool steering (distinct dest MACs → queues, fully raw). Think through both against the 82599 datasheet before committing; UDP/bare-IP encap is the last resort we want to avoid. See the steering section of state.md.

**Answered on paper (2026-08, desk analysis of the driver source — needs only bench confirmation): use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.**
- The flex-byte lead is a dead end: 82599 Flow Director classifies IPv4/IPv6 packets only. The flex word *narrows* an IP-typed match; it cannot create a non-IP one, `FLEX_SHIFT` notwithstanding — non-IP frames never enter the fdir lookup. (One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.)
- ETQF[0–7]/ETQS[0–7] are a dedicated ethertype→RX-queue stage checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses exactly this machinery — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports which ETQF index matched (`RXDADV_PKTTYPE_ETQF`).
- Slot budget fits: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588) is worth reserving (item 2 needs it) → 7 free slots = exactly the 7 streams.
- Patch shape: teach the ixgbe ethtool ntuple path to map exact-ethertype ETHER_FLOW inserts onto free ETQF slots, so `system.go` works unchanged. We already carry a driver patch; this is far smaller than fdir surgery.
- Explains the no-filter baseline too: non-IP frames hash to RSS 0 → all land on queue 0.
- MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter). Encap stays unnecessary.

## 2. Can 82599 sustain zero-baseline-loss 2×10G full duplex, and what happens to timestamping?
It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss (see measurement.md). Separately: the cable-length-via-timestamp idea assumed two ports on one NIC sharing a PHC with usable TX+RX hardware timestamps — 82599's PTP is limited (largely PTP-only, sparse). Re-examine whether the timestamp path-delay length measurement is even viable on this NIC, or whether it dies with the E810.

**Throughput arithmetic says yes at the default config.** X520-DA2 is PCIe Gen2 x8: ~32 Gb/s raw per direction, ~25–26 Gb/s effective after 8b/10b + TLP overhead, against 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already, so it tests nothing new here. On install verify the link actually trained 5 GT/s ×8 (`lspci -vv`). Loss attribution survives the move: 82599 has missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters, so "prove host-side zero" still works.

**Timestamping: the shared-PHC assumption dies, the method doesn't (verified in `ixgbe_ptp.c`).** Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps *only* PTP frames — `HWTSTAMP_FILTER_ALL` is X550+ (`ixgbe_ptp.c:1043`; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences:
- Probe frames must be 1588 L2 event frames (ethertype 0x88F7). They coexist with line-rate 0x88b5 traffic since only 0x88F7 latches — length probing can run *during* the stress test. (0x88F7 recognition is ETQF index 3 — reserved in item 1.)
- Two SYSTIMEs but one crystal: relative drift ≈ 0. A two-way exchange over the same cable (A→B then B→A) cancels the unknown offset exactly, PTP-peer-delay style.
- 6.4 ns quantization ≈ 1.3 m of round trip — average many probes.

**The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives.** The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently.

## 3. Consolidate the cable-length strategy — currently three partial paths
- **BCM ECD/DSP register** — real, but the ECD chapter is missing from FS's docs. What's the fallback if FS doesn't deliver: mine the Broadcom SDK/patents, or lean on the QCA/Marvell CDT analogy?
- **Marvell VCT/DSP** — undocumented, and gated behind a firmware trap that permanently bricks the module on certain reads (`*.0x??64`, high window; see modules.md). Is there *any* safe route, or is it strictly vendor-docs-or-sacrificial-unit?
- **Timestamp path-delay** — see item 2.

Decide which is the product path vs. nice-to-have.

**Additions (2026-08):**
- **A fourth path exists in pure standard registers — no vendor docs at all.** The Clause 45 10GBASE-T PMA group we already read (1.133–1.140) also carries the negotiated TX power backoff / short-reach fields (~1.131 — verify the exact address in 802.3 Table 45–3 before use; all low-address space, nowhere near the `??64` trap). Power backoff is derived from measured channel insertion loss in 2 dB steps → a coarse length bucket on any compliant module. And SNR margin itself falls monotonically with insertion loss on a healthy cable: a small calibration table (known-length cables → margin per pair) likely satisfies goals.md's "sanity check" outright, leaving ECD needed only for fault *localization*.
- **BCM fallback, checked:** OpenBCM's `phy8481.c` covers the copper XGPHY family only through BCM8488x — no 84891, and no ECD code in the retrievable portion — and the kernel's Broadcom ECD (`bcm-phy-lib`) is the BCM54xx GbE register model; neither transfers. SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route, with the standard-register proxy above as the hedge.
- **The Aquantia-based oracle gets a documented path the Wiiteks can't have.** The Fibergaga's PHY family (AQR105/107/109) has its full register reference publicly mirrored (see item 5), reportedly including the cable-diagnostic block — pull the PDF and check before relying on it. The CUX3610 stays sacrificial-unit-only.
- **Proposed decision:** product path = standard-register proxy (SNR margin + power backoff) for length sanity; BCM ECD if/when FS delivers = adds fault localization; timestamp path-delay = pursue only if the item-2 retrain-stability experiment passes.

## 4. Two-master I2C safety + correctness of the untested clients
The `ixgbe` driver polls the SFP EEPROM (DOM/qualification) on its own; our `sff_i2c` transactions share the bus. The swfw semaphore *should* serialize us, but verify the reasoning. Also scrutinize framing assumptions we couldn't test:
- the BCM SMI read's >1 ms inter-transaction delay adequacy;
- the EEPROM-read assumption that the word-address pointer persists across STOP;
- whether RollBall's mailbox read (set-pointer-then-read as two separate transactions) matches what the module expects vs. the kernel `mdio-i2c` combined-transfer form.

**Verified against the source (2026-08) — per-op bracketing is real, and it is the weak point.** Each debugfs `w`/`r` acquires and releases the swfw semaphore individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`), and both clients compose their protocols from separate ops with userspace sleeps between them (`bcm_ixgbe.py` `mdio_read`: `w`, 3 ms sleep, `r`; `eeprom()`: pointer-write then read; same shape in `rollball_ixgbe.py`). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns:
- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`, so no mid-transaction bus corruption. But that traffic is event-driven — SFP identify after module/link events, link setup, `ethtool -m` — i.e. it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but *correlated with the interesting moments*.
- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the real risk is the other master *moving* it. An interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence (`eeprom()`, RollBall mailbox reads).
- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe — the driver never addresses 0x56 and I2C devices are address-filtered — but that's the one interleaving case with no proof.
- **One fix kills all of it**: add a compound debugfs op (write bytes, STOP, optional delay in µs, read n) executed under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds the semaphore longer during SFP identify. This is the same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the i2c bus across its multi-transfer sequences).
- **RollBall split transactions are already field-proven**: the i40e oracle path drove RollBall entirely with single-byte, separate-STOP AQ transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules; only the interleaving above needs fixing.
- **The >1 ms figure cannot be validated from experience** — the E810 never executed an SMI read (reads start with an I2C *write*, which was EPERM-blocked), so the whole SMI path including the delay is untested. Make it self-calibrating at bringup: sweep the delay 0.5→5 ms on a known-nonzero read (PHY ID), find where 0x0000 stops appearing, run at 3×; and treat 0x0000 results as retry-with-longer-delay, since 0 is also the bridge's not-ready signature.

## 5. Pre-FEC error visibility — design the experiment
No module exposes a pre-FEC counter in reachable space; the Aquantia `3.e820` clear-on-read counter is the best candidate but is unclassified (pre- vs post-FEC) because it won't move on a healthy short link. cabletest's own noise pair can create a marginal channel — design the correlation (module counter vs. host CRC/loss under graded noise) that would classify it and turn it into a real leading indicator.

**Answered from documentation (2026-08): `3.E820` is post-FEC, and the pre-FEC counters are documented next door.** The Aquantia AQR105/107/109 register reference is publicly mirrored (assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf — the December 2017 "AQR107-AQR109" datasheet) and names `3.E820`/`3.E821` as the **PCS Receive Vendor Uncorrected Frame Counter** 1/2 — uncorrectable LDPC frames, i.e. it moves *with* host CRC/loss, not before. The same register listings name corrected-frame counters in the same block (search results cite `3.E816–E817` corrected-error-frame counters and a `3.E840–E843` corrected-frames group, apparently broken out by LDPC iteration count — confirm exact names from the PDF). A corrected-by-iteration histogram is the best possible leading indicator: frames needing more LDPC iterations = margin eroding while everything is still corrected and the host sees nothing. First action: pull the PDF, extract the full 3.E8xx map, add the corrected counters to the oracle client.

The correlation experiment becomes verification rather than classification. Design points that stand regardless:
- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
- Run at high line utilization: an uncorrected LDPC frame that lands in idle/IFG leaves no host-visible trace, so low-load runs undercount the host side of the correlation.
- Expected ordering as margin falls: corrected counters rise first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1 counts.
- Log fast retrains alongside (Clause 45 fast-retrain status/count, ~1.147 — verify address) to separate FEC exhaustion from retrain hiccups.
- Clear-on-read discipline: exactly one reader during a run.
- Scope: this documents the Aquantia-based Fibergaga (the oracle) — which is fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out of this until documented.

## 6. Architecture sanity check
Is putting *both* test traffic and module diagnostics on the X520 right, given its steering/timestamp weaknesses, or is there a cleaner split (traffic on a NIC that steers well, diagnostics on the X520)? The tension: the modules under test are where the traffic physically flows, so splitting is awkward — but worth pressure-testing the assumption.

**Resolved (2026-08): co-location is forced, not chosen.** The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md). Each X520 weakness has a within-card answer — steering → ETQF (item 1), timestamps → PTP-shaped probes on-card or the X710 fallback (item 2). The only scenario that reopens this question is the X520 failing zero-baseline-loss (item 2), and the response would be a different *single* card with host-drivable I2C (the txgbe WX1820 from transports.md), not a split.