The genuinely open, thinking-worthy problems — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `transports.md`, `modules.md`, `measurement.md`); this assumes that context.
Short framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. We're swapping the E810 out for an Intel X520-DA2 (82599, `ixgbe`) so the host can bit-bang the SFP module I2C bus (the E810/i40e firmware blocked or couldn't frame it). Diagnostics clients and a raw-i2c kernel patch are prebuilt but untested. Test modules: FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented) and replacement Wiiteks (Marvell CUX3610, brick-prone). All modules are copper lying as fiber.
## 1. Raw-L2 RX steering on 82599 (gates whether cabletest runs on the X520 at all)
The tool steers raw ethertype 0x88b5 to per-queue NAPI contexts via Flow Director; ixgbe rejects ETHER_FLOW. But `FDIRCTRL_FLEX_SHIFT` is already set to 0x6 = byte offset 12 = the ethertype field, so the hardware samples it. Open: will a non-IP frame classify into an fdir flow_type the rule engine matches, and does pairing `flex_bytes` with a permissive base flow type need a driver change? Alternative: MAC→VMDq-pool steering (distinct dest MACs → queues, fully raw). Think through both against the 82599 datasheet before committing; UDP/bare-IP encap is the last resort we want to avoid. See the steering section of state.md.
**Answered on paper from the driver source — needs only bench confirmation: use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.**
- The flex-byte lead is a dead end: 82599 Flow Director classifies IPv4/IPv6 packets only. The flex word *narrows* an IP-typed match; it cannot create a non-IP one, `FLEX_SHIFT` notwithstanding — non-IP frames never enter the fdir lookup. (One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.)
- ETQF[0–7]/ETQS[0–7] are a dedicated ethertype→RX-queue stage checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses exactly this machinery — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports which ETQF index matched (`RXDADV_PKTTYPE_ETQF`).
- Slot budget fits: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588) is worth reserving (item 2 needs it) → 7 free slots = exactly the 7 streams.
- Patch shape: teach the ixgbe ethtool ntuple path to map exact-ethertype ETHER_FLOW inserts onto free ETQF slots, so `system.go` works unchanged. We already carry a driver patch; this is far smaller than fdir surgery.
- Explains the no-filter baseline too: non-IP frames hash to RSS 0 → all land on queue 0.
- MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter).
- UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred on the X520 (no framing change, no header overhead in the size mix), but if it disappoints on bench, encap is fine.
## 2. Can 82599 sustain zero-baseline-loss 2×10G full duplex, and what happens to timestamping?
It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss (see measurement.md). Separately: the cable-length-via-timestamp idea assumed two ports on one NIC sharing a PHC with usable TX+RX hardware timestamps — 82599's PTP is limited (largely PTP-only, sparse). Re-examine whether the timestamp path-delay length measurement is even viable on this NIC, or whether it dies with the E810.
**Throughput arithmetic says yes at the default config.** X520-DA2 is PCIe Gen2 x8: ~32 Gb/s raw per direction, ~25–26 Gb/s effective after 8b/10b + TLP overhead, against 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already, so it tests nothing new here. On install verify the link actually trained 5 GT/s ×8 (`lspci -vv`). Loss attribution survives the move: 82599 has missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters, so "prove host-side zero" still works.
**Timestamping is two separate uses — keep them distinct.**
- *Use 1: rate bucketing — every received frame.* The committed path buckets by per-packet MAC RX timestamps (`rx.go``rateRun.add` on the cmsg stamp), obtained via `SO_TIMESTAMPING` with `rx_filter=ALL` — and that filter is a hard host check (`ts.go`; any failed check aborts the run, `system.go``reportChecks`). Only NICs that can timestamp *all* RX packets qualify: the E810 could; i40e rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`), and 82599 is PTP-only (`TYPE_ALL` is X550+). As committed, the timestamp check therefore fails on both the interim X710 pair and the incoming X520. **There is no software fallback — software RX stamping was tried and cannot reach the needed precision.** All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright, and whether its other hardware can still produce honest buckets is worked through below — that answer drives the NIC choice in item 6. (measurement.md's read-time-stamping lesson is about the *NIC-counter* buckets, a separate system; it is not a substitute for the per-frame stamps.)
- *Use 2: length probes — low volume.* Also already built and committed (`probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry). It rides the same `rx_filter=ALL` for its receive stamps, so on PTP-only NICs the probes must be reshaped as L2 1588 event frames (0x88F7) under a PTP rx filter. The existing both-direction averaging also cancels a stable inter-port clock offset, so `probe.go`'s architecture survives the 82599's per-port clocks — but the two free-running SYSTIMs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first.
**The full 82599 timestamp-hardware space for use 1 — what exists, what might, and the routes to honest buckets without per-packet stamps.** What X550 calls "timestamp all" is two mechanisms together: `TYPE_ALL` filtering plus TSIP (`TSYNCRXCTL_TSIP_UT_EN` — the stamp *prepended into the packet buffer*, which is what makes per-packet delivery scale). The 82599 documents neither: its RX 1588 unit is a single latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`) plus attribute-capture registers (RXSATRL/H, unused by the driver), and its advanced RX descriptor has no timestamp field. Literal per-packet stamps are absent from the silicon. What remains, best first:
1.**Per-queue hardware counters as bucket content — likely sufficient, and the key insight is *where* they count.** QPRC/QBRC (16 slots; queues map to slots via RQSMR; 7 streams fit) increment at queue write-back — *upstream of ring residence*, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate, more than an entire 16 ms bucket, while the RX FIFO ahead of the counters bounds their smear at 512 KB ≈ 400 µs worst-case and ≈ 0 in zero-loss steady state. A sampler in the patched driver (hrtimer at 1–16 ms, snapshot all slots plus ktime into a page the tool maps) yields buckets whose *content* is hardware-counted and whose *boundaries* carry read-time labels — the same honesty discipline measurement.md already proved for counter rates, with µs kernel jitter against 16 ms buckets. Constraint: QPRC/QBRC are read-clear (`ixgbe_main.c:7777,7787`), so exactly one reader — the sampler owns the reads and feeds the driver's accumulators. Nothing of value is lost: `rateRun` only ever carried frames/bytes per bucket.
2.**The latch as an identifiable sampling instrument.** Reading RXSTMPH re-arms the latch. Anchor frames shaped as L2 PTP V2 events with our sequence counter placed in the PTP sequenceID field get true MAC-time stamps, and RXSATRL/H capture the identifying header bytes — each stamp attributable to a specific frame. Even ~1 k stamps/s of ground truth continuously validates route 1's boundaries and measures FIFO+ring residence live. The same machinery is the on-card probe path (below).
3.**Wire-determinism reconstruction — the reserve.** At saturation the TX ring drains at exactly line rate, so every frame's wire time is computable from sequence numbers plus the known size schedule, anchored by sparse latch stamps: exact per-frame times with no per-frame hardware. Breaks off-saturation; reach for it only if routes 1–2 fall short.
Bench pokes that decide the map, first day on the card:
- Write `TYPE_ALL` (0x08) into TSYNCRXCTL on the 82599 and blast raw frames — the encoding is defined silicon-wide in the driver header but documented only for X550; if the reserved encoding secretly arms the latch on raw frames, route 2 sheds its PTP-shaping constraint (RXSATR captures whatever bytes sit at the attribute offsets — put the stream seq there). Poke `TSIP_UT_EN` the same way and look for a buffer prepend. Expected dead; minutes to know.
- RQSMR mapping, QPRC read-clear semantics, and coexistence with the watchdog stats task.
- Latch service rate: sustainable identifiable stamps/s from a dedicated poller.
- Route-1 sampler prototype, validated against route-2 anchors, and against software stamps at low load where software is honest.
If route 1 validates, the X520 delivers honest buckets after all — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the fallback should MCIA fail to reach the BCM on the ConnectX (item 6).
**Probes: the shared-PHC assumption dies, the method doesn't (verified in `ixgbe_ptp.c`).** Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps *only* PTP frames — `HWTSTAMP_FILTER_ALL` is X550+ (`ixgbe_ptp.c:1043`; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences:
- Probe frames must be 1588 L2 event frames (ethertype 0x88F7). They coexist with line-rate 0x88b5 traffic since only 0x88F7 latches — length probing can run *during* the stress test. (0x88F7 recognition is ETQF index 3 — reserved in item 1.)
- Two SYSTIMEs but one crystal: relative drift ≈ 0. A two-way exchange over the same cable (A→B then B→A) cancels the unknown offset exactly, PTP-peer-delay style.
- 6.4 ns quantization ≈ 1.3 m of round trip — average many probes.
**The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives.** The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now — but probes must be PTP-shaped here too, since i40e also stamps only PTP): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently.
## 3. Consolidate the cable-length strategy — currently three partial paths
- **BCM ECD/DSP register** — real, but the ECD chapter is missing from FS's docs. What's the fallback if FS doesn't deliver: mine the Broadcom SDK/patents, or lean on the QCA/Marvell CDT analogy?
- **Marvell VCT/DSP** — undocumented, and gated behind a firmware trap that permanently bricks the module on certain reads (`*.0x??64`, high window; see modules.md). Is there *any* safe route, or is it strictly vendor-docs-or-sacrificial-unit?
- **Timestamp path-delay** — see item 2.
Decide which is the product path vs. nice-to-have.
- **A fourth path exists in pure standard registers — no vendor docs at all.** The Clause 45 10GBASE-T PMA group we already read (1.133–1.140) also carries the negotiated TX power backoff / short-reach fields (~1.131 — verify the exact address in 802.3 Table 45–3 before use; all low-address space, nowhere near the `??64` trap). Power backoff is derived from measured channel insertion loss in 2 dB steps → a coarse length bucket on any compliant module. And SNR margin itself falls monotonically with insertion loss on a healthy cable: a small calibration table (known-length cables → margin per pair) likely satisfies goals.md's "sanity check" outright, leaving ECD needed only for fault *localization*.
- **BCM fallback, checked:** OpenBCM's `phy8481.c` covers the copper XGPHY family only through BCM8488x — no 84891, and no ECD code in the retrievable portion — and the kernel's Broadcom ECD (`bcm-phy-lib`) is the BCM54xx GbE register model; neither transfers. SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route, with the standard-register proxy above as the hedge.
- **The Aquantia-based oracle gets a fully documented path the Wiiteks can't have.** The Fibergaga's PHY family register reference is now vendored in this folder (`aquantia-aqr107-aqr109-datasheet.pdf`, see item 5), and it delivers the DSP estimate outright: **`1E.C884` "Global Status" bits 7:0 = estimated cable length in meters, ±1 m, RO, "estimated from the cable diagnostic engine."** Plus full TDR results: per-pair fault verdicts (open/short/mismatch/cross-pair) at `1E.C800`, distances of the four worst reflections per pair (±1 m) at `1E.C801–C807`, reflection impedance/magnitude at `1E.C880–C883`. Bench question: whether C884 is continuously valid on a linked cable or only after an explicit diag run. This also proves at least one vendor exposes the DSP length in a plain register — useful precedent for the FS/Broadcom ECD ask. The CUX3610 stays sacrificial-unit-only.
## 4. Two-master I2C safety + correctness of the untested clients
The `ixgbe` driver polls the SFP EEPROM (DOM/qualification) on its own; our `sff_i2c` transactions share the bus. The swfw semaphore *should* serialize us, but verify the reasoning. Also scrutinize framing assumptions we couldn't test:
- the BCM SMI read's >1 ms inter-transaction delay adequacy;
- the EEPROM-read assumption that the word-address pointer persists across STOP;
- whether RollBall's mailbox read (set-pointer-then-read as two separate transactions) matches what the module expects vs. the kernel `mdio-i2c` combined-transfer form.
**Verified against the source — per-op bracketing is real, and it is the weak point.** Each debugfs `w`/`r` acquires and releases the swfw semaphore individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`), and both clients compose their protocols from separate ops with userspace sleeps between them (`bcm_ixgbe.py``mdio_read`: `w`, 3 ms sleep, `r`; `eeprom()`: pointer-write then read; same shape in `rollball_ixgbe.py`). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns:
- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`, so no mid-transaction bus corruption. But that traffic is event-driven — SFP identify after module/link events, link setup, `ethtool -m` — i.e. it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but *correlated with the interesting moments*.
- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the real risk is the other master *moving* it. An interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence (`eeprom()`, RollBall mailbox reads).
- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe — the driver never addresses 0x56 and I2C devices are address-filtered — but that's the one interleaving case with no proof.
- **One fix kills all of it**: add a compound debugfs op (write bytes, STOP, optional delay in µs, read n) executed under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds the semaphore longer during SFP identify. This is the same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the i2c bus across its multi-transfer sequences).
- **RollBall split transactions are already field-proven**: the i40e oracle path drove RollBall entirely with single-byte, separate-STOP AQ transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules; only the interleaving above needs fixing.
- **The >1 ms figure cannot be validated from experience** — the E810 never executed an SMI read (reads start with an I2C *write*, which was EPERM-blocked), so the whole SMI path including the delay is untested. Make it self-calibrating at bringup: sweep the delay 0.5→5 ms on a known-nonzero read (PHY ID), find where 0x0000 stops appearing, run at 3×; and treat 0x0000 results as retry-with-longer-delay, since 0 is also the bridge's not-ready signature.
## 5. Pre-FEC error visibility — design the experiment
No module exposes a pre-FEC counter in reachable space; the Aquantia `3.e820` clear-on-read counter is the best candidate but is unclassified (pre- vs post-FEC) because it won't move on a healthy short link. cabletest's own noise pair can create a marginal channel — design the correlation (module counter vs. host CRC/loss under graded noise) that would classify it and turn it into a real leading indicator.
**Answered from documentation: `3.E820` is post-FEC, and the pre-FEC counters are documented next door.** The AQR107-AQR109 register reference (December 2017, rev 1.2.4) is vendored in this folder as `aquantia-aqr107-aqr109-datasheet.pdf` (source mirror: assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf). Verified against it:
-`3.E820`/`3.E821` = **PCS Receive Vendor Uncorrected Frame Counter** 1/2 (§6.7.225–226) — uncorrectable LDPC frames, i.e. it moves *with* host CRC/loss, not before.
-`3.E840–3.E84E` = **PCS Receive Vendor Corrected Frame N Iteration Counter** for N = 1…10 LDPC iterations (§6.7.228–242; N=1–5 are 32-bit register pairs, N=6–10 single 16-bit) — a full corrected-by-iteration histogram, the best possible leading indicator: mass shifting toward higher iteration counts = margin eroding while everything is still corrected and the host sees nothing.
The correlation experiment becomes verification rather than classification. Design points that stand regardless:
- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
- Run at high line utilization: an uncorrected LDPC frame that lands in idle/IFG leaves no host-visible trace, so low-load runs undercount the host side of the correlation.
- Expected ordering as margin falls: corrected counters rise first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1 counts.
- Log fast retrains alongside (Clause 45 fast-retrain status/count, ~1.147 — verify address) to separate FEC exhaustion from retrain hiccups.
- Clear-on-read discipline: exactly one reader during a run.
- Scope: this documents the Aquantia-based Fibergaga (the oracle) — which is fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out of this until documented.
Is putting *both* test traffic and module diagnostics on the X520 right, given its steering/timestamp weaknesses, or is there a cleaner split (traffic on a NIC that steers well, diagnostics on the X520)? The tension: the modules under test are where the traffic physically flows, so splitting is awkward — but worth pressure-testing the assumption.
**Resolved: co-location is forced, not chosen.** The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md).
**But the single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (item 2: software stamping was tried, not precise enough) and raw-L2 steering is not (item 1: encap acceptable) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only, so it cannot carry the committed measurement path; and among the transports.md candidates, txgbe is also PTP-only (verified in upstream `wx_ptp.c`), leaving **ConnectX-4/5 (mlx5) as the only candidate that can meet the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (probe.go's assumption holds), native ETHER_FLOW ntuple steering (no driver patch, no encap needed), mature driver. Its open questions are exactly the diagnostics ones transports.md flagged: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only ever use 0x50/0x51) and whether the BCM SMI read's data phase (raw 2-byte read, no offset write) can be framed at all. RollBall (0x51-resident, offset-model) fits. If MCIA can't reach the BCM, the fork is: product diagnostics via Marvell/Aquantia modules only, with the X520 kept as an offline BCM diagnostics rig — its arbitrary bit-banged framing stays the only *certain* SMI transport, and it remains the bring-up card for proving the clients either way.
