Open questions

The genuinely open, thinking-worthy problems — not mechanical tasks. Read the rest of docs/ first (state.md, transports.md, modules.md, measurement.md); this assumes that context.

Short framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. We're swapping the E810 out for an Intel X520-DA2 (82599, ixgbe) so the host can bit-bang the SFP module I2C bus (the E810/i40e firmware blocked or couldn't frame it). Diagnostics clients and a raw-i2c kernel patch are prebuilt but untested. Test modules: FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented) and replacement Wiiteks (Marvell CUX3610, brick-prone). All modules are copper lying as fiber.

1. Raw-L2 RX steering on 82599 (gates whether cabletest runs on the X520 at all)

The tool steers raw ethertype 0x88b5 to per-queue NAPI contexts via Flow Director; ixgbe rejects ETHER_FLOW. But FDIRCTRL_FLEX_SHIFT is already set to 0x6 = byte offset 12 = the ethertype field, so the hardware samples it. Open: will a non-IP frame classify into an fdir flow_type the rule engine matches, and does pairing flex_bytes with a permissive base flow type need a driver change? Alternative: MAC→VMDq-pool steering (distinct dest MACs → queues, fully raw). Think through both against the 82599 datasheet before committing; UDP/bare-IP encap is the last resort we want to avoid. See the steering section of state.md.

Answered on paper from the driver source — needs only bench confirmation: use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.

The flex-byte lead is a dead end: 82599 Flow Director classifies IPv4/IPv6 packets only. The flex word narrows an IP-typed match; it cannot create a non-IP one, FLEX_SHIFT notwithstanding — non-IP frames never enter the fdir lookup. (One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.)
ETQF[0–7]/ETQS[0–7] are a dedicated ethertype→RX-queue stage checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806). ixgbe already uses exactly this machinery — 0x88F7→1588 latch (ixgbe_ptp.c:1099), FCoE/FIP (ixgbe_fcoe.c:656,691), LLDP/FC under SR-IOV (ixgbe_sriov.c:1828) — and the RX descriptor reports which ETQF index matched (RXDADV_PKTTYPE_ETQF).
Slot budget fits: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588) is worth reserving (item 2 needs it) → 7 free slots = exactly the 7 streams.
Patch shape: teach the ixgbe ethtool ntuple path to map exact-ethertype ETHER_FLOW inserts onto free ETQF slots, so system.go works unchanged. We already carry a driver patch; this is far smaller than fdir surgery.
Explains the no-filter baseline too: non-IP frames hash to RSS 0 → all land on queue 0.
MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter).
UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred on the X520 (no framing change, no header overhead in the size mix), but if it disappoints on bench, encap is fine.

2. Can 82599 sustain zero-baseline-loss 2×10G full duplex, and what happens to timestamping?

It's older/Gen2 silicon. The whole tool depends on exactly zero host-side loss (see measurement.md). Separately: the cable-length-via-timestamp idea assumed two ports on one NIC sharing a PHC with usable TX+RX hardware timestamps — 82599's PTP is limited (largely PTP-only, sparse). Re-examine whether the timestamp path-delay length measurement is even viable on this NIC, or whether it dies with the E810.

Throughput arithmetic says yes at the default config. X520-DA2 is PCIe Gen2 x8: ~32 Gb/s raw per direction, ~25–26 Gb/s effective after 8b/10b + TLP overhead, against 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already, so it tests nothing new here. On install verify the link actually trained 5 GT/s ×8 (lspci -vv). Loss attribution survives the move: 82599 has missed-packet (RXMPC → rx_missed_errors) and per-queue drop (QPRDC) counters, so "prove host-side zero" still works.

Timestamping is two separate uses — keep them distinct.

Use 1: rate bucketing — every received frame. The committed path buckets by per-packet MAC RX timestamps (rx.go rateRun.add on the cmsg stamp), obtained via SO_TIMESTAMPING with rx_filter=ALL — and that filter is a hard host check (ts.go; any failed check aborts the run, system.go reportChecks). Only NICs that can timestamp all RX packets qualify: the E810 could; i40e rejects HWTSTAMP_FILTER_ALL with -ERANGE (verified in upstream i40e_ptp.c), and 82599 is PTP-only (TYPE_ALL is X550+). As committed, the timestamp check therefore fails on both the interim X710 pair and the incoming X520. There is no software fallback — software RX stamping was tried and cannot reach the needed precision. All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright, and whether its other hardware can still produce honest buckets is worked through below — that answer drives the NIC choice in item 6. (measurement.md's read-time-stamping lesson is about the NIC-counter buckets, a separate system; it is not a substitute for the per-frame stamps.)
Use 2: length probes — low volume. Also already built and committed (probe.go: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry). It rides the same rx_filter=ALL for its receive stamps, so on PTP-only NICs the probes must be reshaped as L2 1588 event frames (0x88F7) under a PTP rx filter. The existing both-direction averaging also cancels a stable inter-port clock offset, so probe.go's architecture survives the 82599's per-port clocks — but the two free-running SYSTIMs start at driver load and can sit seconds apart, outside the probeMaxDelay accept window, so they need a one-time offset estimate (or phc2sys) first.

The full 82599 timestamp-hardware space for use 1 — what exists, what might, and the routes to honest buckets without per-packet stamps. What X550 calls "timestamp all" is two mechanisms together: TYPE_ALL filtering plus TSIP (TSYNCRXCTL_TSIP_UT_EN — the stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 documents neither: its RX 1588 unit is a single latch (RXSTMPL/H, locked until RXSTMPH is read, ixgbe_ptp.c:742) plus attribute-capture registers (RXSATRL/H, unused by the driver), and its advanced RX descriptor has no timestamp field. Literal per-packet stamps are absent from the silicon. What remains, best first:

Per-queue hardware counters as bucket content — likely sufficient, and the key insight is where they count. QPRC/QBRC (16 slots; queues map to slots via RQSMR; 7 streams fit) increment at queue write-back — upstream of ring residence, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate, more than an entire 16 ms bucket, while the RX FIFO ahead of the counters bounds their smear at 512 KB ≈ 400 µs worst-case and ≈ 0 in zero-loss steady state. A sampler in the patched driver (hrtimer at 1–16 ms, snapshot all slots plus ktime into a page the tool maps) yields buckets whose content is hardware-counted and whose boundaries carry read-time labels — the same honesty discipline measurement.md already proved for counter rates, with µs kernel jitter against 16 ms buckets. Constraint: QPRC/QBRC are read-clear (ixgbe_main.c:7777,7787), so exactly one reader — the sampler owns the reads and feeds the driver's accumulators. Nothing of value is lost: rateRun only ever carried frames/bytes per bucket.
The latch as an identifiable sampling instrument. Reading RXSTMPH re-arms the latch. Anchor frames shaped as L2 PTP V2 events with our sequence counter placed in the PTP sequenceID field get true MAC-time stamps, and RXSATRL/H capture the identifying header bytes — each stamp attributable to a specific frame. Even ~1 k stamps/s of ground truth continuously validates route 1's boundaries and measures FIFO+ring residence live. The same machinery is the on-card probe path (below).
Wire-determinism reconstruction — the reserve. At saturation the TX ring drains at exactly line rate, so every frame's wire time is computable from sequence numbers plus the known size schedule, anchored by sparse latch stamps: exact per-frame times with no per-frame hardware. Breaks off-saturation; reach for it only if routes 1–2 fall short.

Bench pokes that decide the map, first day on the card:

Write TYPE_ALL (0x08) into TSYNCRXCTL on the 82599 and blast raw frames — the encoding is defined silicon-wide in the driver header but documented only for X550; if the reserved encoding secretly arms the latch on raw frames, route 2 sheds its PTP-shaping constraint (RXSATR captures whatever bytes sit at the attribute offsets — put the stream seq there). Poke TSIP_UT_EN the same way and look for a buffer prepend. Expected dead; minutes to know.
RQSMR mapping, QPRC read-clear semantics, and coexistence with the watchdog stats task.
Latch service rate: sustainable identifiable stamps/s from a dedicated poller.
Route-1 sampler prototype, validated against route-2 anchors, and against software stamps at low load where software is honest.

If route 1 validates, the X520 delivers honest buckets after all — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the fallback should MCIA fail to reach the BCM on the ConnectX (item 6).

Probes: the shared-PHC assumption dies, the method doesn't (verified in ixgbe_ptp.c). Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps only PTP frames — HWTSTAMP_FILTER_ALL is X550+ (ixgbe_ptp.c:1043; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences:

Probe frames must be 1588 L2 event frames (ethertype 0x88F7). They coexist with line-rate 0x88b5 traffic since only 0x88F7 latches — length probing can run during the stress test. (0x88F7 recognition is ETQF index 3 — reserved in item 1.)
Two SYSTIMEs but one crystal: relative drift ≈ 0. A two-way exchange over the same cable (A→B then B→A) cancels the unknown offset exactly, PTP-peer-delay style.
6.4 ns quantization ≈ 1.3 m of round trip — average many probes.

The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives. The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on every NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now — but probes must be PTP-shaped here too, since i40e also stamps only PTP): fixed cable, force N retrains (7.0 |= bit9), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently.

3. Consolidate the cable-length strategy — currently three partial paths

BCM ECD/DSP register — real, but the ECD chapter is missing from FS's docs. What's the fallback if FS doesn't deliver: mine the Broadcom SDK/patents, or lean on the QCA/Marvell CDT analogy?
Marvell VCT/DSP — undocumented, and gated behind a firmware trap that permanently bricks the module on certain reads (*.0x??64, high window; see modules.md). Is there any safe route, or is it strictly vendor-docs-or-sacrificial-unit?
Timestamp path-delay — see item 2.

Decide which is the product path vs. nice-to-have.

BCM fallback, checked: OpenBCM's phy8481.c covers the copper XGPHY family only through BCM8488x — no 84891, and no ECD code in the retrievable portion — and the kernel's Broadcom ECD (bcm-phy-lib) is the BCM54xx GbE register model; neither transfers. SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route.
The Aquantia-based oracle gets a fully documented path the Wiiteks can't have. The Fibergaga's PHY family register reference is now vendored in this folder (aquantia-aqr107-aqr109-datasheet.pdf, see item 5), and it delivers the DSP estimate outright: 1E.C884 "Global Status" bits 7:0 = estimated cable length in meters, ±1 m, RO, "estimated from the cable diagnostic engine." Plus full TDR results: per-pair fault verdicts (open/short/mismatch/cross-pair) at 1E.C800, distances of the four worst reflections per pair (±1 m) at 1E.C801–C807, reflection impedance/magnitude at 1E.C880–C883. Bench question: whether C884 is continuously valid on a linked cable or only after an explicit diag run. This also proves at least one vendor exposes the DSP length in a plain register — useful precedent for the FS/Broadcom ECD ask. The CUX3610 stays sacrificial-unit-only.
Proposed decision: product path = the PHY's own length machinery — BCM ECD if/when FS delivers, the documented 1E.C884 estimate on the Aquantia oracle; NIC timestamp path-delay = the module-independent fallback, pursued only if the item-2 retrain-stability experiment passes.

4. Two-master I2C safety + correctness of the untested clients

The ixgbe driver polls the SFP EEPROM (DOM/qualification) on its own; our sff_i2c transactions share the bus. The swfw semaphore should serialize us, but verify the reasoning. Also scrutinize framing assumptions we couldn't test:

the BCM SMI read's >1 ms inter-transaction delay adequacy;
the EEPROM-read assumption that the word-address pointer persists across STOP;
whether RollBall's mailbox read (set-pointer-then-read as two separate transactions) matches what the module expects vs. the kernel mdio-i2c combined-transfer form.

Verified against the source — per-op bracketing is real, and it is the weak point. Each debugfs w/r acquires and releases the swfw semaphore individually (~/work/ixgbe-sff/ixgbe_phy.c:63–108), and both clients compose their protocols from separate ops with userspace sleeps between them (bcm_ixgbe.py mdio_read: w, 3 ms sleep, r; eeprom(): pointer-write then read; same shape in rollball_ixgbe.py). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns:

Serialization itself holds: the driver's own SFP traffic uses the same byte primitives under the same phy_semaphore_mask, so no mid-transaction bus corruption. But that traffic is event-driven — SFP identify after module/link events, link setup, ethtool -m — i.e. it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but correlated with the interesting moments.
Pointer persistence, resolved: AT24-style word pointers do persist across STOP; the real risk is the other master moving it. An interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence (eeprom(), RollBall mailbox reads).
The BCM 0x56 bridge's pending read data across the >1 ms window is probably safe — the driver never addresses 0x56 and I2C devices are address-filtered — but that's the one interleaving case with no proof.
One fix kills all of it: add a compound debugfs op (write bytes, STOP, optional delay in µs, read n) executed under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds the semaphore longer during SFP identify. This is the same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (mdio-i2c locks the i2c bus across its multi-transfer sequences).
RollBall split transactions are already field-proven: the i40e oracle path drove RollBall entirely with single-byte, separate-STOP AQ transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules; only the interleaving above needs fixing.
The >1 ms figure cannot be validated from experience — the E810 never executed an SMI read (reads start with an I2C write, which was EPERM-blocked), so the whole SMI path including the delay is untested. Make it self-calibrating at bringup: sweep the delay 0.5→5 ms on a known-nonzero read (PHY ID), find where 0x0000 stops appearing, run at 3×; and treat 0x0000 results as retry-with-longer-delay, since 0 is also the bridge's not-ready signature.

5. Pre-FEC error visibility — design the experiment

No module exposes a pre-FEC counter in reachable space; the Aquantia 3.e820 clear-on-read counter is the best candidate but is unclassified (pre- vs post-FEC) because it won't move on a healthy short link. cabletest's own noise pair can create a marginal channel — design the correlation (module counter vs. host CRC/loss under graded noise) that would classify it and turn it into a real leading indicator.

Answered from documentation: 3.E820 is post-FEC, and the pre-FEC counters are documented next door. The AQR107-AQR109 register reference (December 2017, rev 1.2.4) is vendored in this folder as aquantia-aqr107-aqr109-datasheet.pdf (source mirror: assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf). Verified against it:

3.E820/3.E821 = PCS Receive Vendor Uncorrected Frame Counter 1/2 (§6.7.225–226) — uncorrectable LDPC frames, i.e. it moves with host CRC/loss, not before.
3.E840–3.E84E = PCS Receive Vendor Corrected Frame N Iteration Counter for N = 1…10 LDPC iterations (§6.7.228–242; N=1–5 are 32-bit register pairs, N=6–10 single 16-bit) — a full corrected-by-iteration histogram, the best possible leading indicator: mass shifting toward higher iteration counts = margin eroding while everything is still corrected and the host sees nothing.
3.E816 = PCS Receive Vendor RS Corrected Error Frame Counter (§6.7.223) — likely the system-side/RS path, secondary.

First action: add the corrected-iteration counters to the oracle client.

The correlation experiment becomes verification rather than classification. Design points that stand regardless:

Dose axis = measured SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
Run at high line utilization: an uncorrected LDPC frame that lands in idle/IFG leaves no host-visible trace, so low-load runs undercount the host side of the correlation.
Expected ordering as margin falls: corrected counters rise first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1 counts.
Log fast retrains alongside (Clause 45 fast-retrain status/count, ~1.147 — verify address) to separate FEC exhaustion from retrain hiccups.
Clear-on-read discipline: exactly one reader during a run.
Scope: this documents the Aquantia-based Fibergaga (the oracle) — which is fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out of this until documented.

6. Architecture sanity check

Is putting both test traffic and module diagnostics on the X520 right, given its steering/timestamp weaknesses, or is there a cleaner split (traffic on a NIC that steers well, diagnostics on the X520)? The tension: the modules under test are where the traffic physically flows, so splitting is awkward — but worth pressure-testing the assumption.

Resolved: co-location is forced, not chosen. The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in its cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md).

But the single card is not the X520. Two constraints — all-packet hardware RX stamps are non-negotiable (item 2: software stamping was tried, not precise enough) and raw-L2 steering is not (item 1: encap acceptable) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only, so it cannot carry the committed measurement path; and among the transports.md candidates, txgbe is also PTP-only (verified in upstream wx_ptp.c), leaving ConnectX-4/5 (mlx5) as the only candidate that can meet the full set: CQE-stamps every RX packet (HWTSTAMP_FILTER_ALL), one shared clock across both ports (probe.go's assumption holds), native ETHER_FLOW ntuple steering (no driver patch, no encap needed), mature driver. Its open questions are exactly the diagnostics ones transports.md flagged: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only ever use 0x50/0x51) and whether the BCM SMI read's data phase (raw 2-byte read, no offset write) can be framed at all. RollBall (0x51-resident, offset-model) fits. If MCIA can't reach the BCM, the fork is: product diagnostics via Marvell/Aquantia modules only, with the X520 kept as an offline BCM diagnostics rig — its arbitrary bit-banged framing stays the only certain SMI transport, and it remains the bring-up card for proving the clients either way.

22 KiB Raw Blame History Unescape Escape