cabletest/docs/open-questions.md

# Open questions

The genuinely open, thinking-worthy problems and their current answers — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `nics/`, `modules/`, `measurement.md`); this assumes that context.

Framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. Which cards and modules are in play: `state.md`.

## 1. Raw-L2 RX steering on 82599

**Status: solved and proven on hardware — ETQF steers raw ethertypes to per-queue contexts.** The committed tool steers raw ethertype 0x88b5+ to per-queue NAPI contexts; stock ixgbe rejects ETHER_FLOW ntuple rules. The patched ixgbe maps exact-ethertype ETHER_FLOW inserts onto ETQF slots (`~/work/ixgbe-sff/ixgbe_ethtool.c`, `ixgbe_add_etqf_entry`), so `system.go` runs unchanged.

- **ETQF/ETQS is a dedicated ethertype→RX-queue stage** checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses it — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`).
- **Bench-confirmed** (`~/work/phydiag-work/etqfbench`): 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each on their queue; an 8th unfiltered ethertype falls to queue 0; delete restores the fall-through. `ETHTOOL_GRXCLSRULE`/`GRXCLSRLALL` round-trip the ETQF rules so the tool's stale-rule cleanup sees and clears them.
- **Slot budget fits**: 8 filters; the patch reserves index 3 (1588, needed by §2) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes were moved behind the FCoE-enabled guard (`ixgbe_fcoe.c`) so the default config leaves slots 0/1/2 free; without that, FCoE CRC pre-claimed slot 2. Restore-on-link-up (`ixgbe_etqf_filter_restore`) and clear-on-close (`ixgbe_etqf_filter_exit`) are hooked so a ring-resize reset re-applies the rules.
- **Flow Director is a dead end for raw L2 — confirmed on hardware.** 82599 fdir classifies IPv4/IPv6 only; a flex-word rule (`FDIRCTRL_FLEX_SHIFT = 0x6` = byte offset 12 = the ethertype field) targeting a raw 0x88b5 stream steers nothing — every frame falls to queue 0 (`etqfbench -mode fdirneg`). Non-IP frames never enter the fdir lookup; the flex word only *narrows* an IP-typed match.
- Baseline behavior without filters: non-IP frames hash to RSS 0 → all land on queue 0.
- Fallbacks (unneeded now): MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix).

## 2. Zero-baseline loss and timestamping on the 82599

**Status: throughput yes; all-packet exact RX timestamps no — and there is no substitute. A cable tester needs each frame's exact arrival time; aggregate counters/rates are honest only when nothing is wrong, which is useless for fault detection. The 82599 has one serial PTP-only latch and nothing else, so the X520 is disqualified as the product measurement NIC (diagnostics-only). The ConnectX-5 is the product path.**

### Throughput

- PCIe Gen2 x8 ≈ 32 Gb/s raw, ~25–26 Gb/s/direction effective, vs 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already. Verify 5 GT/s ×8 trained (`lspci -vv`).
- Loss attribution survives: 82599 exposes missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" still works.

### Timestamping is two separate uses — keep them distinct

| Use | Mechanism (committed) | 82599 |
|---|---|---|
| 1. Rate bucketing — every received frame | Per-packet MAC RX stamps via `SO_TIMESTAMPING` cmsg (`rx.go` `rateRun.add`); `rx_filter=ALL` enforced as a hard host check (`ts.go`; failed checks abort, `system.go` `reportChecks`) | **Cannot** — PTP-only (`TYPE_ALL` is X550+, `ixgbe_ptp.c:1043`); i40e also rejects `FILTER_ALL` with -ERANGE (verified upstream). Only the E810 qualified |
| 2. Length probes — low volume | `probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry | Possible with PTP-shaped probes (below) |

**There is no software fallback for use 1** — software RX stamping was tried and cannot reach the needed precision. All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright. (measurement.md's read-time-stamping lesson covers the *NIC-counter* buckets, a separate system.)

### The full 82599 hardware space for use 1 — and why none of it substitutes for per-packet stamps

**A cable tester measures the receive precisely when it is misbehaving — loss bursts, late frames, timing excursions to correlate with pre-FEC errors and retrains. That signal lives in each frame's exact arrival time. Any aggregate (a rate, a count per window) is only honest when the receive is well-behaved, i.e. when there is nothing to measure. Exact per-frame timestamps are the requirement, full stop; nothing on the 82599 provides them for all packets.**

What X550 calls "timestamp all" is `TYPE_ALL` filtering **plus TSIP** (`TSYNCRXCTL_TSIP_UT_EN` — stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 has neither — confirmed on hardware: writing `TYPE_ALL` sticks in the register but never arms the latch on raw frames, and the `TSIP_UT_EN` bit will not even hold. All it has is one RX latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`) with attribute-capture (RXSATRL/H), no timestamp field in the RX descriptor.

The routes that were mapped here are all **rejected** for a cable tester:

1. ~~Per-queue hardware counters as bucket content.~~ **Rejected — it's a rate, not timestamps.** QPRC/QBRC deltas per window give aggregate throughput, honest only in zero-loss steady state (the FIFO smear that would perturb a boundary window is ~0 only when nothing is going wrong). It measured cleanly on the bench (RQSMR maps 1:1, read-clear, exact counts) but it cannot place a single frame in time, so it cannot see the fault a cable tester is built to catch. Measured facts retained below; the conclusion is dead.
2. ~~The latch as an identifiable sampler.~~ **Not all-packet.** One serial latch, PTP-only, ~383 k stamps/s from a dedicated in-kernel spin loop against ~1.78 M pps/dir line rate. Useful only for the low-volume length probes (below), never for stamping the 0x88b5 test traffic.
3. ~~Wire-determinism reconstruction.~~ **Rejected — same failure as counters.** Reconstructing per-frame wire times from the size schedule assumes saturation regularity, which a loss/retrain event breaks; it is blind at exactly the interesting moments.

**Verdict: the 82599 cannot do all-packet exact RX timestamping and has no path to it. The X520 is disqualified as the product measurement NIC — diagnostics / BCM-framing card only.** The ConnectX-5 (per-packet CQE stamps, `HWTSTAMP_FILTER_ALL`) is the product path (§6).

Bench-poke facts (measured, retained for the record even though the counter conclusion is dead):

- `TYPE_ALL` writable but inert for raw frames; `TSIP_UT_EN` will not hold — the latch is PTP-only silicon, not a hidden all-packet path.
- RQSMR maps queue→counter 1:1; QPRC/QBRC exact and read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`).
- Single PTP latch services ~383 k identifiable stamps/s; stream seq lands in `RXSATRH[31:16]` byte-swapped.

### Probes on the 82599

- The shared-PHC assumption dies, the method doesn't: each port has its own free-running SYSTIME (~6.4 ns granularity at 10G), but both share one crystal — relative drift ≈ 0, and a two-way exchange over the same cable cancels the unknown offset exactly, PTP-peer-delay style. `probe.go`'s existing both-direction averaging already provides this — but the two SYSTIMEs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first.
- Probe frames must be 1588 L2 event frames (0x88F7): only those latch, so probing coexists with line-rate 0x88b5 traffic (0x88F7 recognition is ETQF index 3, reserved in §1). 6.4 ns quantization ≈ 1.3 m of round trip — average many probes.

### The decisive unknown is the module PHYs — testable before anything is built

The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the timestamp length method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches — probes must be PTP-shaped there too): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. ≲ 10–20 ns → viable; much more → close §3's timestamp path permanently.

The bench pokes (measured facts above) were run through the patched driver's debugfs — `reg_ops` returns read values, and `ts_bench` runs the latch poll/re-arm loop in-kernel because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM` (drivers: `~/work/phydiag-work/x520poke`, `ts_bench` in `~/work/ixgbe-sff/ixgbe_debugfs.c`). What they establish for the 82599: no all-packet exact-timestamp path exists, so the only surviving use of the latch is the low-volume **length probes** (below), where sparse PTP-shaped anchors are exactly the model — not the line-rate 0x88b5 measurement.

## 3. Cable-length strategy

**Status: consolidated — PHY features are the product path, NIC timestamps the fallback. The BCM path works.**

| Path | Status |
|---|---|
| BCM ECD | **Working on the bench** — recipe recovered from the OpenBCM SDK (`phy8481.c` cable-diag, same register family as the 84891L's command handler) and validated: per-pair verdicts + per-pair lengths in meters on a plugged healthy cable. Calibration and run-disturbance characterization remain (modules/fs/) |
| Aquantia DSP/TDR (oracle only) | **Fully documented**: `1E.C884` length ±1 m, per-pair TDR verdicts and reflection distances (modules/fibergaga/) |
| Marvell VCT (Wiiteks) | Undocumented and gated behind the brick trap; sacrificial-unit-only single-shot templates (modules/wiitek/). The BCM ECD result layout (verdict nibbles + per-pair length registers) is a fresh analogy for future targeted probes |
| NIC timestamp path-delay | Module-independent; gated on the §2 retrain-stability experiment |

- **Decision:** product path = the PHY's own length machinery (BCM ECD, `1E.C884` on the oracle); NIC timestamp path-delay = fallback, pursued only if the §2 experiment passes. The ECD run blips the link, so length measurement is a between-runs operation, not a during-run one, unless characterization says otherwise.

## 4. Two-master I2C safety and the once-untested client assumptions

**Status: verified against source and hardware; the hardening item is built and proven.**

- **Per-op semaphore bracketing is real and is the weak point.** Each debugfs `w`/`r` acquires/releases swfw individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`); clients compose protocols from separate ops with userspace sleeps. Every multi-op sequence has windows where the driver can run its own I2C.
- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask` — no mid-transaction bus corruption. But driver traffic is event-driven (SFP identify after module/link events, link setup, `ethtool -m`) — it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but correlated with the interesting moments.
- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the risk is the other master *moving* the pointer — an interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence.
- **The BCM 0x56 bridge's pending read data** (0x56 7-bit = 0xAC/0xAD write/read) across the >1 ms window was the one interleaving case without proof; the compound `x` op removes the window rather than reasoning about it, so the case is now closed by construction.
- **The compound op is built and proven.** `sff_i2c` gains an `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`, `ixgbe_phy.c`): write, STOP, in-kernel delay, read — all under a single swfw hold, closing the interleave window entirely. The write/read primitives were refactored into locked/unlocked halves so `x` reuses them under one acquire. Validated on the FS BCM (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004). A ~3 ms hold is nothing — the driver holds swfw longer during SFP identify. The delay sweep corroborates the not-ready hazard the op closes: at delay 0 the bridged read returns stale data (0x0004 left over from the prior read), at ≥200 µs it reads correctly — precisely the stale-pointer/stale-data corruption an interleaved driver access would cause.
- **RollBall split transactions: field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules.
- **The >1 ms BCM delay**: validated on hardware at 3 ms. Treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.

## 5. Pre-FEC error visibility

**Status: answered from documentation — `3.E820` is post-FEC; the pre-FEC counters are the corrected-by-iteration histogram next door.** Register map: modules/fibergaga/. The correlation experiment is now verification, not classification.

Design points:

- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
- Run at high line utilization: an uncorrected LDPC frame landing in idle/IFG leaves no host-visible trace; low-load runs undercount the host side.
- Expected ordering as margin falls: corrected-iteration counters shift toward higher N first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1.
- Log fast retrains alongside (IEEE 1.147) to separate FEC exhaustion from retrain hiccups.
- Clear-on-read discipline: exactly one reader during a run.
- Scope: documents the Aquantia oracle — fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out until documented.

## 6. Architecture

**Status: resolved twice over — co-location is forced, and the single card is the ConnectX-5.**

- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation (§5), and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
- **The single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (§2) and raw-L2 steering is not (§1) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only; txgbe is also PTP-only (verified in upstream `wx_ptp.c`); **ConnectX-4/5 (mlx5) is the only candidate meeting the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (`probe.go`'s assumption holds), native ETHER_FLOW ntuple steering, mature driver.
- **Open on the ConnectX**: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only use 0x50/0x51) and whether the BCM SMI read data phase (raw 2-byte read, no offset write) can be framed. RollBall (0x51-resident, offset-model) fits.
- **If MCIA can't reach the BCM**: product diagnostics via the Marvell/Aquantia modules only, with the X520 kept as an offline BCM rig — its bit-banged framing stays the only *certain* SMI transport. The X520 is **not** a fallback product NIC: it has no all-packet exact-timestamp path (§2), and a cable tester's rate buckets require exactly that.