diff --git a/docs/open-questions.md b/docs/open-questions.md index 5516ee9..de788f2 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -7,9 +7,26 @@ Short framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and ## 1. Raw-L2 RX steering on 82599 (gates whether cabletest runs on the X520 at all) The tool steers raw ethertype 0x88b5 to per-queue NAPI contexts via Flow Director; ixgbe rejects ETHER_FLOW. But `FDIRCTRL_FLEX_SHIFT` is already set to 0x6 = byte offset 12 = the ethertype field, so the hardware samples it. Open: will a non-IP frame classify into an fdir flow_type the rule engine matches, and does pairing `flex_bytes` with a permissive base flow type need a driver change? Alternative: MAC→VMDq-pool steering (distinct dest MACs → queues, fully raw). Think through both against the 82599 datasheet before committing; UDP/bare-IP encap is the last resort we want to avoid. See the steering section of state.md. +**Answered on paper (2026-08, desk analysis of the driver source — needs only bench confirmation): use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.** +- The flex-byte lead is a dead end: 82599 Flow Director classifies IPv4/IPv6 packets only. The flex word *narrows* an IP-typed match; it cannot create a non-IP one, `FLEX_SHIFT` notwithstanding — non-IP frames never enter the fdir lookup. (One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.) +- ETQF[0–7]/ETQS[0–7] are a dedicated ethertype→RX-queue stage checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses exactly this machinery — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports which ETQF index matched (`RXDADV_PKTTYPE_ETQF`). +- Slot budget fits: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588) is worth reserving (item 2 needs it) → 7 free slots = exactly the 7 streams. +- Patch shape: teach the ixgbe ethtool ntuple path to map exact-ethertype ETHER_FLOW inserts onto free ETQF slots, so `system.go` works unchanged. We already carry a driver patch; this is far smaller than fdir surgery. +- Explains the no-filter baseline too: non-IP frames hash to RSS 0 → all land on queue 0. +- MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter). Encap stays unnecessary. + ## 2. Can 82599 sustain zero-baseline-loss 2×10G full duplex, and what happens to timestamping? It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss (see measurement.md). Separately: the cable-length-via-timestamp idea assumed two ports on one NIC sharing a PHC with usable TX+RX hardware timestamps — 82599's PTP is limited (largely PTP-only, sparse). Re-examine whether the timestamp path-delay length measurement is even viable on this NIC, or whether it dies with the E810. +**Throughput arithmetic says yes at the default config.** X520-DA2 is PCIe Gen2 x8: ~32 Gb/s raw per direction, ~25–26 Gb/s effective after 8b/10b + TLP overhead, against 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already, so it tests nothing new here. On install verify the link actually trained 5 GT/s ×8 (`lspci -vv`). Loss attribution survives the move: 82599 has missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters, so "prove host-side zero" still works. + +**Timestamping: the shared-PHC assumption dies, the method doesn't (verified in `ixgbe_ptp.c`).** Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps *only* PTP frames — `HWTSTAMP_FILTER_ALL` is X550+ (`ixgbe_ptp.c:1043`; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences: +- Probe frames must be 1588 L2 event frames (ethertype 0x88F7). They coexist with line-rate 0x88b5 traffic since only 0x88F7 latches — length probing can run *during* the stress test. (0x88F7 recognition is ETQF index 3 — reserved in item 1.) +- Two SYSTIMEs but one crystal: relative drift ≈ 0. A two-way exchange over the same cable (A→B then B→A) cancels the unknown offset exactly, PTP-peer-delay style. +- 6.4 ns quantization ≈ 1.3 m of round trip — average many probes. + +**The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives.** The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently. + ## 3. Consolidate the cable-length strategy — currently three partial paths - **BCM ECD/DSP register** — real, but the ECD chapter is missing from FS's docs. What's the fallback if FS doesn't deliver: mine the Broadcom SDK/patents, or lean on the QCA/Marvell CDT analogy? - **Marvell VCT/DSP** — undocumented, and gated behind a firmware trap that permanently bricks the module on certain reads (`*.0x??64`, high window; see modules.md). Is there *any* safe route, or is it strictly vendor-docs-or-sacrificial-unit? @@ -17,14 +34,40 @@ It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss Decide which is the product path vs. nice-to-have. +**Additions (2026-08):** +- **A fourth path exists in pure standard registers — no vendor docs at all.** The Clause 45 10GBASE-T PMA group we already read (1.133–1.140) also carries the negotiated TX power backoff / short-reach fields (~1.131 — verify the exact address in 802.3 Table 45–3 before use; all low-address space, nowhere near the `??64` trap). Power backoff is derived from measured channel insertion loss in 2 dB steps → a coarse length bucket on any compliant module. And SNR margin itself falls monotonically with insertion loss on a healthy cable: a small calibration table (known-length cables → margin per pair) likely satisfies goals.md's "sanity check" outright, leaving ECD needed only for fault *localization*. +- **BCM fallback, checked:** OpenBCM's `phy8481.c` covers the copper XGPHY family only through BCM8488x — no 84891, and no ECD code in the retrievable portion — and the kernel's Broadcom ECD (`bcm-phy-lib`) is the BCM54xx GbE register model; neither transfers. SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route, with the standard-register proxy above as the hedge. +- **The Aquantia-based oracle gets a documented path the Wiiteks can't have.** The Fibergaga's PHY family (AQR105/107/109) has its full register reference publicly mirrored (see item 5), reportedly including the cable-diagnostic block — pull the PDF and check before relying on it. The CUX3610 stays sacrificial-unit-only. +- **Proposed decision:** product path = standard-register proxy (SNR margin + power backoff) for length sanity; BCM ECD if/when FS delivers = adds fault localization; timestamp path-delay = pursue only if the item-2 retrain-stability experiment passes. + ## 4. Two-master I2C safety + correctness of the untested clients The `ixgbe` driver polls the SFP EEPROM (DOM/qualification) on its own; our `sff_i2c` transactions share the bus. The swfw semaphore *should* serialize us, but verify the reasoning. Also scrutinize framing assumptions we couldn't test: - the BCM SMI read's >1 ms inter-transaction delay adequacy; - the EEPROM-read assumption that the word-address pointer persists across STOP; - whether RollBall's mailbox read (set-pointer-then-read as two separate transactions) matches what the module expects vs. the kernel `mdio-i2c` combined-transfer form. +**Verified against the source (2026-08) — per-op bracketing is real, and it is the weak point.** Each debugfs `w`/`r` acquires and releases the swfw semaphore individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`), and both clients compose their protocols from separate ops with userspace sleeps between them (`bcm_ixgbe.py` `mdio_read`: `w`, 3 ms sleep, `r`; `eeprom()`: pointer-write then read; same shape in `rollball_ixgbe.py`). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns: +- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`, so no mid-transaction bus corruption. But that traffic is event-driven — SFP identify after module/link events, link setup, `ethtool -m` — i.e. it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but *correlated with the interesting moments*. +- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the real risk is the other master *moving* it. An interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence (`eeprom()`, RollBall mailbox reads). +- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe — the driver never addresses 0x56 and I2C devices are address-filtered — but that's the one interleaving case with no proof. +- **One fix kills all of it**: add a compound debugfs op (write bytes, STOP, optional delay in µs, read n) executed under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds the semaphore longer during SFP identify. This is the same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the i2c bus across its multi-transfer sequences). +- **RollBall split transactions are already field-proven**: the i40e oracle path drove RollBall entirely with single-byte, separate-STOP AQ transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules; only the interleaving above needs fixing. +- **The >1 ms figure cannot be validated from experience** — the E810 never executed an SMI read (reads start with an I2C *write*, which was EPERM-blocked), so the whole SMI path including the delay is untested. Make it self-calibrating at bringup: sweep the delay 0.5→5 ms on a known-nonzero read (PHY ID), find where 0x0000 stops appearing, run at 3×; and treat 0x0000 results as retry-with-longer-delay, since 0 is also the bridge's not-ready signature. + ## 5. Pre-FEC error visibility — design the experiment No module exposes a pre-FEC counter in reachable space; the Aquantia `3.e820` clear-on-read counter is the best candidate but is unclassified (pre- vs post-FEC) because it won't move on a healthy short link. cabletest's own noise pair can create a marginal channel — design the correlation (module counter vs. host CRC/loss under graded noise) that would classify it and turn it into a real leading indicator. +**Answered from documentation (2026-08): `3.E820` is post-FEC, and the pre-FEC counters are documented next door.** The Aquantia AQR105/107/109 register reference is publicly mirrored (assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf — the December 2017 "AQR107-AQR109" datasheet) and names `3.E820`/`3.E821` as the **PCS Receive Vendor Uncorrected Frame Counter** 1/2 — uncorrectable LDPC frames, i.e. it moves *with* host CRC/loss, not before. The same register listings name corrected-frame counters in the same block (search results cite `3.E816–E817` corrected-error-frame counters and a `3.E840–E843` corrected-frames group, apparently broken out by LDPC iteration count — confirm exact names from the PDF). A corrected-by-iteration histogram is the best possible leading indicator: frames needing more LDPC iterations = margin eroding while everything is still corrected and the host sees nothing. First action: pull the PDF, extract the full 3.E8xx map, add the corrected counters to the oracle client. + +The correlation experiment becomes verification rather than classification. Design points that stand regardless: +- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic. +- Run at high line utilization: an uncorrected LDPC frame that lands in idle/IFG leaves no host-visible trace, so low-load runs undercount the host side of the correlation. +- Expected ordering as margin falls: corrected counters rise first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1 counts. +- Log fast retrains alongside (Clause 45 fast-retrain status/count, ~1.147 — verify address) to separate FEC exhaustion from retrain hiccups. +- Clear-on-read discipline: exactly one reader during a run. +- Scope: this documents the Aquantia-based Fibergaga (the oracle) — which is fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out of this until documented. + ## 6. Architecture sanity check Is putting *both* test traffic and module diagnostics on the X520 right, given its steering/timestamp weaknesses, or is there a cleaner split (traffic on a NIC that steers well, diagnostics on the X520)? The tension: the modules under test are where the traffic physically flows, so splitting is awkward — but worth pressure-testing the assumption. + +**Resolved (2026-08): co-location is forced, not chosen.** The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md). Each X520 weakness has a within-card answer — steering → ETQF (item 1), timestamps → PTP-shaped probes on-card or the X710 fallback (item 2). The only scenario that reopens this question is the X520 failing zero-baseline-loss (item 2), and the response would be a different *single* card with host-drivable I2C (the txgbe WX1820 from transports.md), not a split. diff --git a/docs/state.md b/docs/state.md index ff22c15..47d6f27 100644 --- a/docs/state.md +++ b/docs/state.md @@ -28,10 +28,11 @@ In `~/work/` alongside the ice-sff/phydiag-work artifacts, ready to fold into th ## Open: RX flow-steering on ixgbe needs serious investigation before the X520 carries test traffic cabletest fans RX across queues by steering **raw ethertype** (0x88b5 base) to distinct queues with Flow Director rules (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). The obvious path fails: ixgbe/82599's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — **no ETHER_FLOW** — so an ETHER_FLOW rule insert is rejected (EINVAL) and the tool fails at startup (no fallback, by design). This is a datapath issue affecting *both* module types, independent of diagnostics. -But 82599 has hardware steering machinery the current code doesn't use, and **there is good reason to think we can keep true raw-L2 ethertype steering without falling back to encapsulation.** This needs bench investigation on the actual card; do not assume the answer is UDP encap. Leads, most promising first: -- **Flow Director flex-byte match.** The driver exposes a 2-byte `flex_bytes` filter (via the `user-def`/FLOW_EXT `vlan_etype` field), and `ixgbe_reinit_fdir_tables_82599` already programs `FDIRCTRL_FLEX_SHIFT = 0x6` — offset 12 bytes, **exactly the ethertype field** of an Ethernet frame. So the hardware is already sampling the ethertype. Open question to settle on hardware: whether a non-IP raw frame classifies into an fdir flow_type the rule engine will match, given the driver requires an IP base flow_type in the rule spec — may need a small driver change to pair flex_bytes with a permissive/L2 flow type. -- **MAC → VMDq pool steering.** 82599 assigns unicast MAC (RAR) entries to VMDq pools and pools to RX queues (`hw_set_rar` VMDq pool/queue selection). Distinct dest MACs per stream → distinct queues, fully raw Ethernet, no IP at all. Worth prototyping alongside the flex-byte path. -- **RSS on a flexible field** — 82599 RSS hashes IP tuples, not raw L2, so unlikely to help without encap; noted only to rule out. +But 82599 has hardware steering machinery the current code doesn't use, and **there is good reason to think we can keep true raw-L2 ethertype steering without falling back to encapsulation.** This needs bench investigation on the actual card; do not assume the answer is UDP encap. Leads, most promising first (full analysis in open-questions.md §1): +- **L2 EtherType queue filters (ETQF/ETQS) — the answer on paper.** A dedicated ethertype→RX-queue stage checked at L2 ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588), already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots so `system.go` works unchanged. Bench-confirm on the card. +- **Flow Director flex-byte match — ruled out on paper.** 82599 fdir classifies IPv4/IPv6 only; the flex word narrows an IP-typed match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test. +- **MAC → VMDq pool steering.** 82599 assigns unicast MAC (RAR) entries to VMDq pools and pools to RX queues (`hw_set_rar` VMDq pool/queue selection). Distinct dest MACs per stream → distinct queues, fully raw Ethernet, no IP at all. Fallback if ETQF disappoints. +- **RSS on a flexible field** — 82599 RSS hashes IP tuples, not raw L2; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline). Fallbacks if raw-L2 steering genuinely can't be made to work: minimal bare-IPv4 framing (no UDP) steered by IP_USER_FLOW src/dst IP (lighter than the old UDP-encap plan), or single-queue RX (caps near the old ~1.6 Mpps single-NAPI ceiling, loses small-frame headroom and per-stream isolation). These are last resorts, not the plan.