# Open questions

The genuinely open, thinking-worthy problems and their current answers — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `nics/`, `modules/`, `measurement.md`); this assumes that context.

Framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. Which cards and modules are in play: `state.md`.

## 1. Raw-L2 RX steering on 82599

**Status: answered on paper — use ETQF, bench-confirm.** The tool steers raw ethertype 0x88b5+ to per-queue NAPI contexts; ixgbe rejects ETHER_FLOW ntuple rules.

- **Flow Director is a dead end for raw L2.** 82599 fdir classifies IPv4/IPv6 only; the flex word (`FDIRCTRL_FLEX_SHIFT = 0x6` = byte offset 12 = the ethertype field) *narrows* an IP-typed match but cannot create a non-IP one — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.
- **ETQF/ETQS is a dedicated ethertype→RX-queue stage** checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses it — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`).
- **Slot budget fits**: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588, needed by §2) is reserved → 7 free = exactly the 7 streams.
- **Patch shape**: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots in the ixgbe ethtool path, so `system.go` works unchanged. Far smaller than fdir surgery; we already carry a driver patch.
- Baseline behavior without filters: non-IP frames hash to RSS 0 → all land on queue 0.
- Fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred (no framing change, no header overhead in the size mix).

## 2. Zero-baseline loss and timestamping on the 82599

**Status: throughput yes; per-packet timestamps no — exploration routes mapped, plus one decisive experiment runnable on the X710 ahead of any build-out.**

### Throughput

- PCIe Gen2 x8 ≈ 32 Gb/s raw, ~25–26 Gb/s/direction effective, vs 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already. Verify 5 GT/s ×8 trained (`lspci -vv`).
- Loss attribution survives: 82599 exposes missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" still works.

### Timestamping is two separate uses — keep them distinct

| Use | Mechanism (committed) | 82599 |
|---|---|---|
| 1. Rate bucketing — every received frame | Per-packet MAC RX stamps via `SO_TIMESTAMPING` cmsg (`rx.go` `rateRun.add`); `rx_filter=ALL` enforced as a hard host check (`ts.go`; failed checks abort, `system.go` `reportChecks`) | **Cannot** — PTP-only (`TYPE_ALL` is X550+, `ixgbe_ptp.c:1043`); i40e also rejects `FILTER_ALL` with -ERANGE (verified upstream). Only the E810 qualified |
| 2. Length probes — low volume | `probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry | Possible with PTP-shaped probes (below) |

**There is no software fallback for use 1** — software RX stamping was tried and cannot reach the needed precision. All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright. (measurement.md's read-time-stamping lesson covers the *NIC-counter* buckets, a separate system.)

### The full 82599 hardware space for use 1 — routes to honest buckets without per-packet stamps

What X550 calls "timestamp all" is `TYPE_ALL` filtering **plus TSIP** (`TSYNCRXCTL_TSIP_UT_EN` — stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 documents neither: one RX latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`), attribute-capture registers (RXSATRL/H, unused by the driver), no timestamp field in the RX descriptor. Routes, best first:

1. **Per-queue hardware counters as bucket content — likely sufficient.** QPRC/QBRC (16 slots; queues map via RQSMR; 7 streams fit) increment at queue write-back — *upstream of ring residence*, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate (more than one 16 ms bucket), while the RX FIFO ahead of the counters bounds smear at 512 KB ≈ 400 µs worst-case, ≈ 0 in zero-loss steady state. Implementation: an hrtimer sampler in the patched driver (1–16 ms cadence, snapshot slots + ktime into a page the tool maps) — hardware-counted content, read-time-labeled boundaries, the same honesty discipline measurement.md proved. Constraint: QPRC/QBRC are read-clear (`ixgbe_main.c:7777,7787`) — exactly one reader; the sampler owns the reads and feeds the driver's accumulators. Nothing lost: `rateRun` only ever carried frames/bytes per bucket.
2. **The latch as an identifiable sampler.** Reading RXSTMPH re-arms it. Anchor frames shaped as L2 PTP V2 events carrying our sequence counter in the PTP sequenceID field get MAC-time stamps; RXSATRL/H capture the identifying bytes. Even ~1 k stamps/s continuously validates route 1's boundaries and measures FIFO+ring residence live.
3. **Wire-determinism reconstruction — the reserve.** At saturation the TX ring drains at exactly line rate; per-frame wire times are computable from sequence numbers + the size schedule, anchored by sparse latch stamps. Breaks off-saturation; only if 1–2 fall short.

Bench pokes, first day on the card:

- Write `TYPE_ALL` (0x08) into TSYNCRXCTL and blast raw frames — defined silicon-wide in the header, documented only for X550; if the reserved encoding secretly arms the latch, route 2 sheds its PTP-shaping constraint (put the stream seq at the RXSATR capture offsets). Poke `TSIP_UT_EN` and look for a buffer prepend. Expected dead; minutes to know.
- RQSMR mapping, QPRC read-clear semantics, coexistence with the watchdog stats task.
- Latch service rate: sustainable identifiable stamps/s.
- Route-1 sampler prototype, validated against route-2 anchors and against software stamps at low load (where software is honest).

If route 1 validates, the X520 delivers honest buckets — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the MCIA fallback (§6).

### Probes on the 82599

- The shared-PHC assumption dies, the method doesn't: each port has its own free-running SYSTIME (~6.4 ns granularity at 10G), but both share one crystal — relative drift ≈ 0, and a two-way exchange over the same cable cancels the unknown offset exactly, PTP-peer-delay style. `probe.go`'s existing both-direction averaging already provides this — but the two SYSTIMEs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first.
- Probe frames must be 1588 L2 event frames (0x88F7): only those latch, so probing coexists with line-rate 0x88b5 traffic (0x88F7 recognition is ETQF index 3, reserved in §1). 6.4 ns quantization ≈ 1.3 m of round trip — average many probes.

### The decisive unknown is the module PHYs — testable before anything is built

The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the timestamp length method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches — probes must be PTP-shaped there too): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. ≲ 10–20 ns → viable; much more → close §3's timestamp path permanently.

## 3. Cable-length strategy

**Status: consolidated — PHY features are the product path, NIC timestamps the fallback. The BCM path works.**

| Path | Status |
|---|---|
| BCM ECD | **Working on the bench** — recipe recovered from the OpenBCM SDK (`phy8481.c` cable-diag, same register family as the 84891L's command handler) and validated: per-pair verdicts + per-pair lengths in meters on a plugged healthy cable. Calibration and run-disturbance characterization remain (modules/fs/) |
| Aquantia DSP/TDR (oracle only) | **Fully documented**: `1E.C884` length ±1 m, per-pair TDR verdicts and reflection distances (modules/fibergaga/) |
| Marvell VCT (Wiiteks) | Undocumented and gated behind the brick trap; sacrificial-unit-only single-shot templates (modules/wiitek/). The BCM ECD result layout (verdict nibbles + per-pair length registers) is a fresh analogy for future targeted probes |
| NIC timestamp path-delay | Module-independent; gated on the §2 retrain-stability experiment |

- **Decision:** product path = the PHY's own length machinery (BCM ECD, `1E.C884` on the oracle); NIC timestamp path-delay = fallback, pursued only if the §2 experiment passes. The ECD run blips the link, so length measurement is a between-runs operation, not a during-run one, unless characterization says otherwise.

## 4. Two-master I2C safety and the once-untested client assumptions

**Status: verified against source and hardware; one hardening item remains.**

- **Per-op semaphore bracketing is real and is the weak point.** Each debugfs `w`/`r` acquires/releases swfw individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`); clients compose protocols from separate ops with userspace sleeps. Every multi-op sequence has windows where the driver can run its own I2C.
- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask` — no mid-transaction bus corruption. But driver traffic is event-driven (SFP identify after module/link events, link setup, `ethtool -m`) — it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but correlated with the interesting moments.
- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the risk is the other master *moving* the pointer — an interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence.
- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe (the driver never addresses 0x56; I2C is address-filtered) — the one interleaving case without proof.
- **Remaining hardening item**: a compound debugfs op (write bytes, STOP, optional delay µs, read n) under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds it longer during SFP identify. Same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the bus across multi-transfer sequences).
- **RollBall split transactions: field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules.
- **The >1 ms BCM delay**: validated on hardware at 3 ms. Treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.

## 5. Pre-FEC error visibility

**Status: answered from documentation — `3.E820` is post-FEC; the pre-FEC counters are the corrected-by-iteration histogram next door.** Register map: modules/fibergaga/. The correlation experiment is now verification, not classification.

Design points:

- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
- Run at high line utilization: an uncorrected LDPC frame landing in idle/IFG leaves no host-visible trace; low-load runs undercount the host side.
- Expected ordering as margin falls: corrected-iteration counters shift toward higher N first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1.
- Log fast retrains alongside (IEEE 1.147) to separate FEC exhaustion from retrain hiccups.
- Clear-on-read discipline: exactly one reader during a run.
- Scope: documents the Aquantia oracle — fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out until documented.

## 6. Architecture

**Status: resolved twice over — co-location is forced, and the single card is the ConnectX-5.**

- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation (§5), and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
- **The single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (§2) and raw-L2 steering is not (§1) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only; txgbe is also PTP-only (verified in upstream `wx_ptp.c`); **ConnectX-4/5 (mlx5) is the only candidate meeting the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (`probe.go`'s assumption holds), native ETHER_FLOW ntuple steering, mature driver.
- **Open on the ConnectX**: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only use 0x50/0x51) and whether the BCM SMI read data phase (raw 2-byte read, no offset write) can be framed. RollBall (0x51-resident, offset-model) fits.
- **If MCIA can't reach the BCM**: product diagnostics via the Marvell/Aquantia modules only, with the X520 kept as an offline BCM rig — its bit-banged framing stays the only *certain* SMI transport — or, if the §2 bucket exploration validates, the X520 itself as fallback product NIC.