diff --git a/docs/aquantia-aqr107-aqr109-datasheet.pdf b/docs/aquantia-aqr107-aqr109-datasheet.pdf new file mode 100644 index 0000000..b41af51 Binary files /dev/null and b/docs/aquantia-aqr107-aqr109-datasheet.pdf differ diff --git a/docs/measurement.md b/docs/measurement.md index f1f1412..a606223 100644 --- a/docs/measurement.md +++ b/docs/measurement.md @@ -21,7 +21,7 @@ Any nonzero baseline loss masks real cable faults, so a run counts only when `tx - **Batch sizes above 64** — no gain, worse loss. - The tx `sendmmsg` busy-spin on ENOBUFS is not worth chasing: it only exists when the tx ring is full, which means the wire is the ceiling, so recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable). -The AF_XDP experiment (stashed) removed the tx frame-generation ceiling (64 B tx ~14.5 Mpps/dir) and moved the bottleneck to RX (~4 Mpps/dir on AF_PACKET, ~13.5 with AF_XDP RX). It required per-packet MAC rx timestamps for honest buckets, delivered via an XDP-metadata kfunc — available on the E810 datapath, not the X710. With the test path off the E810 this whole path is parked; read-time stamping covers bucketing regardless of NIC. +The AF_XDP experiment (stashed) removed the tx frame-generation ceiling (64 B tx ~14.5 Mpps/dir) and moved the bottleneck to RX (~4 Mpps/dir on AF_PACKET, ~13.5 with AF_XDP RX). It required per-packet MAC rx timestamps for honest buckets, delivered via an XDP-metadata kfunc — available on the E810 datapath, not the X710. With the test path off the E810 this whole path is parked. Note the committed AF_PACKET path *also* buckets by per-packet MAC rx stamps — via the `SO_TIMESTAMPING` cmsg, with `rx_filter=ALL` enforced as a hard host check — so the stash's novelty was the delivery mechanism and throughput, not the use of hardware stamps. Read-time stamping is how the *NIC-counter* buckets are labeled (the lesson above), not a substitute for the per-frame stamps; NICs without all-packet rx timestamping (X710, 82599) fail the host check and need a bucketing fallback — see open-questions.md §2. ## rxnfc / Flow Director programming (ice), three traps that each cost a debugging round 1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement, so trusting its display gives an inverted rule that silently matches nothing. diff --git a/docs/open-questions.md b/docs/open-questions.md index de788f2..dba69b3 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -7,25 +7,44 @@ Short framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and ## 1. Raw-L2 RX steering on 82599 (gates whether cabletest runs on the X520 at all) The tool steers raw ethertype 0x88b5 to per-queue NAPI contexts via Flow Director; ixgbe rejects ETHER_FLOW. But `FDIRCTRL_FLEX_SHIFT` is already set to 0x6 = byte offset 12 = the ethertype field, so the hardware samples it. Open: will a non-IP frame classify into an fdir flow_type the rule engine matches, and does pairing `flex_bytes` with a permissive base flow type need a driver change? Alternative: MAC→VMDq-pool steering (distinct dest MACs → queues, fully raw). Think through both against the 82599 datasheet before committing; UDP/bare-IP encap is the last resort we want to avoid. See the steering section of state.md. -**Answered on paper (2026-08, desk analysis of the driver source — needs only bench confirmation): use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.** +**Answered on paper from the driver source — needs only bench confirmation: use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.** - The flex-byte lead is a dead end: 82599 Flow Director classifies IPv4/IPv6 packets only. The flex word *narrows* an IP-typed match; it cannot create a non-IP one, `FLEX_SHIFT` notwithstanding — non-IP frames never enter the fdir lookup. (One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.) - ETQF[0–7]/ETQS[0–7] are a dedicated ethertype→RX-queue stage checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses exactly this machinery — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports which ETQF index matched (`RXDADV_PKTTYPE_ETQF`). - Slot budget fits: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588) is worth reserving (item 2 needs it) → 7 free slots = exactly the 7 streams. - Patch shape: teach the ixgbe ethtool ntuple path to map exact-ethertype ETHER_FLOW inserts onto free ETQF slots, so `system.go` works unchanged. We already carry a driver patch; this is far smaller than fdir surgery. - Explains the no-filter baseline too: non-IP frames hash to RSS 0 → all land on queue 0. -- MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter). Encap stays unnecessary. +- MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter). +- UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred on the X520 (no framing change, no header overhead in the size mix), but if it disappoints on bench, encap is fine. ## 2. Can 82599 sustain zero-baseline-loss 2×10G full duplex, and what happens to timestamping? It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss (see measurement.md). Separately: the cable-length-via-timestamp idea assumed two ports on one NIC sharing a PHC with usable TX+RX hardware timestamps — 82599's PTP is limited (largely PTP-only, sparse). Re-examine whether the timestamp path-delay length measurement is even viable on this NIC, or whether it dies with the E810. **Throughput arithmetic says yes at the default config.** X520-DA2 is PCIe Gen2 x8: ~32 Gb/s raw per direction, ~25–26 Gb/s effective after 8b/10b + TLP overhead, against 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already, so it tests nothing new here. On install verify the link actually trained 5 GT/s ×8 (`lspci -vv`). Loss attribution survives the move: 82599 has missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters, so "prove host-side zero" still works. -**Timestamping: the shared-PHC assumption dies, the method doesn't (verified in `ixgbe_ptp.c`).** Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps *only* PTP frames — `HWTSTAMP_FILTER_ALL` is X550+ (`ixgbe_ptp.c:1043`; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences: +**Timestamping is two separate uses — keep them distinct.** +- *Use 1: rate bucketing — every received frame.* The committed path buckets by per-packet MAC RX timestamps (`rx.go` `rateRun.add` on the cmsg stamp), obtained via `SO_TIMESTAMPING` with `rx_filter=ALL` — and that filter is a hard host check (`ts.go`; any failed check aborts the run, `system.go` `reportChecks`). Only NICs that can timestamp *all* RX packets qualify: the E810 could; i40e rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`), and 82599 is PTP-only (`TYPE_ALL` is X550+). As committed, the timestamp check therefore fails on both the interim X710 pair and the incoming X520. **There is no software fallback — software RX stamping was tried and cannot reach the needed precision.** All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright, and whether its other hardware can still produce honest buckets is worked through below — that answer drives the NIC choice in item 6. (measurement.md's read-time-stamping lesson is about the *NIC-counter* buckets, a separate system; it is not a substitute for the per-frame stamps.) +- *Use 2: length probes — low volume.* Also already built and committed (`probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry). It rides the same `rx_filter=ALL` for its receive stamps, so on PTP-only NICs the probes must be reshaped as L2 1588 event frames (0x88F7) under a PTP rx filter. The existing both-direction averaging also cancels a stable inter-port clock offset, so `probe.go`'s architecture survives the 82599's per-port clocks — but the two free-running SYSTIMs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first. + +**The full 82599 timestamp-hardware space for use 1 — what exists, what might, and the routes to honest buckets without per-packet stamps.** What X550 calls "timestamp all" is two mechanisms together: `TYPE_ALL` filtering plus TSIP (`TSYNCRXCTL_TSIP_UT_EN` — the stamp *prepended into the packet buffer*, which is what makes per-packet delivery scale). The 82599 documents neither: its RX 1588 unit is a single latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`) plus attribute-capture registers (RXSATRL/H, unused by the driver), and its advanced RX descriptor has no timestamp field. Literal per-packet stamps are absent from the silicon. What remains, best first: + +1. **Per-queue hardware counters as bucket content — likely sufficient, and the key insight is *where* they count.** QPRC/QBRC (16 slots; queues map to slots via RQSMR; 7 streams fit) increment at queue write-back — *upstream of ring residence*, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate, more than an entire 16 ms bucket, while the RX FIFO ahead of the counters bounds their smear at 512 KB ≈ 400 µs worst-case and ≈ 0 in zero-loss steady state. A sampler in the patched driver (hrtimer at 1–16 ms, snapshot all slots plus ktime into a page the tool maps) yields buckets whose *content* is hardware-counted and whose *boundaries* carry read-time labels — the same honesty discipline measurement.md already proved for counter rates, with µs kernel jitter against 16 ms buckets. Constraint: QPRC/QBRC are read-clear (`ixgbe_main.c:7777,7787`), so exactly one reader — the sampler owns the reads and feeds the driver's accumulators. Nothing of value is lost: `rateRun` only ever carried frames/bytes per bucket. +2. **The latch as an identifiable sampling instrument.** Reading RXSTMPH re-arms the latch. Anchor frames shaped as L2 PTP V2 events with our sequence counter placed in the PTP sequenceID field get true MAC-time stamps, and RXSATRL/H capture the identifying header bytes — each stamp attributable to a specific frame. Even ~1 k stamps/s of ground truth continuously validates route 1's boundaries and measures FIFO+ring residence live. The same machinery is the on-card probe path (below). +3. **Wire-determinism reconstruction — the reserve.** At saturation the TX ring drains at exactly line rate, so every frame's wire time is computable from sequence numbers plus the known size schedule, anchored by sparse latch stamps: exact per-frame times with no per-frame hardware. Breaks off-saturation; reach for it only if routes 1–2 fall short. + +Bench pokes that decide the map, first day on the card: +- Write `TYPE_ALL` (0x08) into TSYNCRXCTL on the 82599 and blast raw frames — the encoding is defined silicon-wide in the driver header but documented only for X550; if the reserved encoding secretly arms the latch on raw frames, route 2 sheds its PTP-shaping constraint (RXSATR captures whatever bytes sit at the attribute offsets — put the stream seq there). Poke `TSIP_UT_EN` the same way and look for a buffer prepend. Expected dead; minutes to know. +- RQSMR mapping, QPRC read-clear semantics, and coexistence with the watchdog stats task. +- Latch service rate: sustainable identifiable stamps/s from a dedicated poller. +- Route-1 sampler prototype, validated against route-2 anchors, and against software stamps at low load where software is honest. + +If route 1 validates, the X520 delivers honest buckets after all — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the fallback should MCIA fail to reach the BCM on the ConnectX (item 6). + +**Probes: the shared-PHC assumption dies, the method doesn't (verified in `ixgbe_ptp.c`).** Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps *only* PTP frames — `HWTSTAMP_FILTER_ALL` is X550+ (`ixgbe_ptp.c:1043`; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences: - Probe frames must be 1588 L2 event frames (ethertype 0x88F7). They coexist with line-rate 0x88b5 traffic since only 0x88F7 latches — length probing can run *during* the stress test. (0x88F7 recognition is ETQF index 3 — reserved in item 1.) - Two SYSTIMEs but one crystal: relative drift ≈ 0. A two-way exchange over the same cable (A→B then B→A) cancels the unknown offset exactly, PTP-peer-delay style. - 6.4 ns quantization ≈ 1.3 m of round trip — average many probes. -**The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives.** The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently. +**The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives.** The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now — but probes must be PTP-shaped here too, since i40e also stamps only PTP): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently. ## 3. Consolidate the cable-length strategy — currently three partial paths - **BCM ECD/DSP register** — real, but the ECD chapter is missing from FS's docs. What's the fallback if FS doesn't deliver: mine the Broadcom SDK/patents, or lean on the QCA/Marvell CDT analogy? @@ -34,10 +53,9 @@ It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss Decide which is the product path vs. nice-to-have. -**Additions (2026-08):** - **A fourth path exists in pure standard registers — no vendor docs at all.** The Clause 45 10GBASE-T PMA group we already read (1.133–1.140) also carries the negotiated TX power backoff / short-reach fields (~1.131 — verify the exact address in 802.3 Table 45–3 before use; all low-address space, nowhere near the `??64` trap). Power backoff is derived from measured channel insertion loss in 2 dB steps → a coarse length bucket on any compliant module. And SNR margin itself falls monotonically with insertion loss on a healthy cable: a small calibration table (known-length cables → margin per pair) likely satisfies goals.md's "sanity check" outright, leaving ECD needed only for fault *localization*. - **BCM fallback, checked:** OpenBCM's `phy8481.c` covers the copper XGPHY family only through BCM8488x — no 84891, and no ECD code in the retrievable portion — and the kernel's Broadcom ECD (`bcm-phy-lib`) is the BCM54xx GbE register model; neither transfers. SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route, with the standard-register proxy above as the hedge. -- **The Aquantia-based oracle gets a documented path the Wiiteks can't have.** The Fibergaga's PHY family (AQR105/107/109) has its full register reference publicly mirrored (see item 5), reportedly including the cable-diagnostic block — pull the PDF and check before relying on it. The CUX3610 stays sacrificial-unit-only. +- **The Aquantia-based oracle gets a fully documented path the Wiiteks can't have.** The Fibergaga's PHY family register reference is now vendored in this folder (`aquantia-aqr107-aqr109-datasheet.pdf`, see item 5), and it delivers the DSP estimate outright: **`1E.C884` "Global Status" bits 7:0 = estimated cable length in meters, ±1 m, RO, "estimated from the cable diagnostic engine."** Plus full TDR results: per-pair fault verdicts (open/short/mismatch/cross-pair) at `1E.C800`, distances of the four worst reflections per pair (±1 m) at `1E.C801–C807`, reflection impedance/magnitude at `1E.C880–C883`. Bench question: whether C884 is continuously valid on a linked cable or only after an explicit diag run. This also proves at least one vendor exposes the DSP length in a plain register — useful precedent for the FS/Broadcom ECD ask. The CUX3610 stays sacrificial-unit-only. - **Proposed decision:** product path = standard-register proxy (SNR margin + power backoff) for length sanity; BCM ECD if/when FS delivers = adds fault localization; timestamp path-delay = pursue only if the item-2 retrain-stability experiment passes. ## 4. Two-master I2C safety + correctness of the untested clients @@ -46,7 +64,7 @@ The `ixgbe` driver polls the SFP EEPROM (DOM/qualification) on its own; our `sff - the EEPROM-read assumption that the word-address pointer persists across STOP; - whether RollBall's mailbox read (set-pointer-then-read as two separate transactions) matches what the module expects vs. the kernel `mdio-i2c` combined-transfer form. -**Verified against the source (2026-08) — per-op bracketing is real, and it is the weak point.** Each debugfs `w`/`r` acquires and releases the swfw semaphore individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`), and both clients compose their protocols from separate ops with userspace sleeps between them (`bcm_ixgbe.py` `mdio_read`: `w`, 3 ms sleep, `r`; `eeprom()`: pointer-write then read; same shape in `rollball_ixgbe.py`). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns: +**Verified against the source — per-op bracketing is real, and it is the weak point.** Each debugfs `w`/`r` acquires and releases the swfw semaphore individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`), and both clients compose their protocols from separate ops with userspace sleeps between them (`bcm_ixgbe.py` `mdio_read`: `w`, 3 ms sleep, `r`; `eeprom()`: pointer-write then read; same shape in `rollball_ixgbe.py`). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns: - **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`, so no mid-transaction bus corruption. But that traffic is event-driven — SFP identify after module/link events, link setup, `ethtool -m` — i.e. it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but *correlated with the interesting moments*. - **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the real risk is the other master *moving* it. An interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence (`eeprom()`, RollBall mailbox reads). - **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe — the driver never addresses 0x56 and I2C devices are address-filtered — but that's the one interleaving case with no proof. @@ -57,7 +75,12 @@ The `ixgbe` driver polls the SFP EEPROM (DOM/qualification) on its own; our `sff ## 5. Pre-FEC error visibility — design the experiment No module exposes a pre-FEC counter in reachable space; the Aquantia `3.e820` clear-on-read counter is the best candidate but is unclassified (pre- vs post-FEC) because it won't move on a healthy short link. cabletest's own noise pair can create a marginal channel — design the correlation (module counter vs. host CRC/loss under graded noise) that would classify it and turn it into a real leading indicator. -**Answered from documentation (2026-08): `3.E820` is post-FEC, and the pre-FEC counters are documented next door.** The Aquantia AQR105/107/109 register reference is publicly mirrored (assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf — the December 2017 "AQR107-AQR109" datasheet) and names `3.E820`/`3.E821` as the **PCS Receive Vendor Uncorrected Frame Counter** 1/2 — uncorrectable LDPC frames, i.e. it moves *with* host CRC/loss, not before. The same register listings name corrected-frame counters in the same block (search results cite `3.E816–E817` corrected-error-frame counters and a `3.E840–E843` corrected-frames group, apparently broken out by LDPC iteration count — confirm exact names from the PDF). A corrected-by-iteration histogram is the best possible leading indicator: frames needing more LDPC iterations = margin eroding while everything is still corrected and the host sees nothing. First action: pull the PDF, extract the full 3.E8xx map, add the corrected counters to the oracle client. +**Answered from documentation: `3.E820` is post-FEC, and the pre-FEC counters are documented next door.** The AQR107-AQR109 register reference (December 2017, rev 1.2.4) is vendored in this folder as `aquantia-aqr107-aqr109-datasheet.pdf` (source mirror: assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf). Verified against it: +- `3.E820`/`3.E821` = **PCS Receive Vendor Uncorrected Frame Counter** 1/2 (§6.7.225–226) — uncorrectable LDPC frames, i.e. it moves *with* host CRC/loss, not before. +- `3.E840–3.E84E` = **PCS Receive Vendor Corrected Frame N Iteration Counter** for N = 1…10 LDPC iterations (§6.7.228–242; N=1–5 are 32-bit register pairs, N=6–10 single 16-bit) — a full corrected-by-iteration histogram, the best possible leading indicator: mass shifting toward higher iteration counts = margin eroding while everything is still corrected and the host sees nothing. +- `3.E816` = PCS Receive Vendor RS Corrected Error Frame Counter (§6.7.223) — likely the system-side/RS path, secondary. + +First action: add the corrected-iteration counters to the oracle client. The correlation experiment becomes verification rather than classification. Design points that stand regardless: - Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic. @@ -70,4 +93,6 @@ The correlation experiment becomes verification rather than classification. Desi ## 6. Architecture sanity check Is putting *both* test traffic and module diagnostics on the X520 right, given its steering/timestamp weaknesses, or is there a cleaner split (traffic on a NIC that steers well, diagnostics on the X520)? The tension: the modules under test are where the traffic physically flows, so splitting is awkward — but worth pressure-testing the assumption. -**Resolved (2026-08): co-location is forced, not chosen.** The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md). Each X520 weakness has a within-card answer — steering → ETQF (item 1), timestamps → PTP-shaped probes on-card or the X710 fallback (item 2). The only scenario that reopens this question is the X520 failing zero-baseline-loss (item 2), and the response would be a different *single* card with host-drivable I2C (the txgbe WX1820 from transports.md), not a split. +**Resolved: co-location is forced, not chosen.** The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md). + +**But the single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (item 2: software stamping was tried, not precise enough) and raw-L2 steering is not (item 1: encap acceptable) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only, so it cannot carry the committed measurement path; and among the transports.md candidates, txgbe is also PTP-only (verified in upstream `wx_ptp.c`), leaving **ConnectX-4/5 (mlx5) as the only candidate that can meet the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (probe.go's assumption holds), native ETHER_FLOW ntuple steering (no driver patch, no encap needed), mature driver. Its open questions are exactly the diagnostics ones transports.md flagged: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only ever use 0x50/0x51) and whether the BCM SMI read's data phase (raw 2-byte read, no offset write) can be framed at all. RollBall (0x51-resident, offset-model) fits. If MCIA can't reach the BCM, the fork is: product diagnostics via Marvell/Aquantia modules only, with the X520 kept as an offline BCM diagnostics rig — its arbitrary bit-banged framing stays the only *certain* SMI transport, and it remains the bring-up card for proving the clients either way. diff --git a/docs/state.md b/docs/state.md index 47d6f27..4f487e0 100644 --- a/docs/state.md +++ b/docs/state.md @@ -1,12 +1,14 @@ # Current state ## Committed tree -AF_PACKET raw sockets everywhere (`sock.go`), flow-director steering, read-time-stamped rate buckets, framebuffer UI, harness. The zero-copy AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata) lives uncommitted in `stash@{0}`; it was built for the era when the test path ran on the E810, whose datapath delivers per-packet MAC timestamps. It is an experiment, not a requirement. +AF_PACKET raw sockets everywhere (`sock.go`), flow-director steering, per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — see open-questions.md §2 for what that demands of the NIC), read-time-stamped NIC-counter rates, hardware-timestamped length probes (`probe.go`), framebuffer UI, harness. The zero-copy AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata) lives uncommitted in `stash@{0}`; it was built for the era when the test path ran on the E810, whose datapath delivers per-packet MAC timestamps. It is an experiment, not a requirement. `stash@{1}` holds the phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, and a STATUS.md campaign log. Working copies extracted to `~/work/phydiag-work/` during the module work. The knowledge from that campaign is now in these docs; the code should eventually move into the repo properly. ## Hardware in flight - **Intel X520-DA2 ordered** — replaces the E810 in the box's single PCIe slot. Chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md. +- **ConnectX-5 ordered (dual SFP28, PCIe x8) — the product NIC candidate.** Rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement); the 82599 stamps PTP frames only, and with raw-L2 steering relaxed (encap acceptable), mlx5 is the one driver that meets the full set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: the MCIA diagnostics questions in open-questions.md §6 (BCM at I2C 0x56, SMI read data phase). On arrival: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`), SFP+ modules drop into SFP28 cages at 10G, and check whether `mlx5_ib` autoloading interferes with channel changes (the irdma lesson). +- **The X520 arrives first and carries two campaigns**: the diagnostics bring-up (certain arbitrary I2C framing for proving the BCM/RollBall clients), and the 82599 bucket-timestamping exploration in open-questions.md §2 — per-queue hardware counters as bucket content plus latch-anchor sampling. If that validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX. - **Replacement Wiitek (Marvell CUX3610) modules ordered.** Both original units were bricked by register exploration — see modules.md for the trap before ever touching one. - **In hand** (all copper RJ45 10GBASE-T modules with cloned/lying fiber EEPROMs — see hardware.md; none are actually fiber): - 2× FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented diagnostics). @@ -20,7 +22,7 @@ In `~/work/` alongside the ice-sff/phydiag-work artifacts, ready to fold into th - **RollBall client for the Wiiteks** (`~/work/phydiag-work/rollball_ixgbe.py`): the same `sff_i2c` transport, RollBall unlock/page/mailbox, per-pair IEEE SNR (PMA 1.133–1.136). The `*.0x??64` (high-byte ≥ 0x80) brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill. ## Bringup plan when the X520 arrives -1. Install card, move FS modules in, `./load-ixgbe`, confirm `sff_i2c` appears. +1. Install card, move FS modules in, `./load-ixgbe` (with `allow_unsupported_sfp=1` — ixgbe refuses SFPs without Intel qualification bytes, and the cloned EEPROMs aren't Intel-coded; `ixgbe_main.c:165`), confirm `sff_i2c` appears. 2. FS/BCM modules first: `bcm_ixgbe.py` sanity reads (EEPROM, PHY ID, STATUS) to prove the SMI transport, then the command handler (CMD_GET_SNR vs the IEEE registers) on the test cable set. 3. Replacement Wiiteks: IEEE-standard registers only (SNR, latches) unless/until a documented recipe exists for more; the VCT templates in modules.md are candidates for single-shot targeted probes on a sacrificial unit only. 4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs). @@ -34,10 +36,10 @@ But 82599 has hardware steering machinery the current code doesn't use, and **th - **MAC → VMDq pool steering.** 82599 assigns unicast MAC (RAR) entries to VMDq pools and pools to RX queues (`hw_set_rar` VMDq pool/queue selection). Distinct dest MACs per stream → distinct queues, fully raw Ethernet, no IP at all. Fallback if ETQF disappoints. - **RSS on a flexible field** — 82599 RSS hashes IP tuples, not raw L2; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline). -Fallbacks if raw-L2 steering genuinely can't be made to work: minimal bare-IPv4 framing (no UDP) steered by IP_USER_FLOW src/dst IP (lighter than the old UDP-encap plan), or single-queue RX (caps near the old ~1.6 Mpps single-NAPI ceiling, loses small-frame headroom and per-stream isolation). These are last resorts, not the plan. +Fallbacks if raw-L2 steering genuinely can't be made to work: minimal bare-IPv4 framing (no UDP) steered by IP_USER_FLOW src/dst IP (lighter than the old UDP-encap plan), or single-queue RX (caps near the old ~1.6 Mpps single-NAPI ceiling, loses small-frame headroom and per-stream isolation). Encap is acceptable as a fallback (see open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice. ## Open items - **ECD register chapter**: the one missing document for BCM cable length. FS has been responsive; the narrow ask is "which registers invoke ECD / report the DSP cable-length estimate, and does it run with link up?" - **Pre-FEC classification** on the Aquantia (3.e820): needs a marginal channel; cabletest's noise stress will provide one once diagnostics are integrated. -- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, 1 ft calibration cancels PHY latency); scoped but unbuilt. Superseded for the product if BCM ECD/DSP length pans out. +- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, 1 ft calibration cancels PHY latency); scoped but unbuilt — the committed `probe.go` is the *filter-all* variant of the same idea (raw-frame probes, needs all-packet rx stamping, so E810-only); the X710/X520 variant means reshaping probes as PTP frames. Superseded for the product if BCM ECD/DSP length pans out. - E810 leaves the box when the X520 arrives; the patched ice + `sff_i2c` remains useful only if an E810 returns for read-side work.