diff --git a/docs/nics/README.md b/docs/nics/README.md index 6f49b7a..f910f8e 100644 --- a/docs/nics/README.md +++ b/docs/nics/README.md @@ -12,7 +12,7 @@ Which card is where: [../state.md](../state.md). | NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict | |---|---|---|---| -| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only) | The certain diagnostics transport; product-capable only if the bucket exploration validates | +| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only, serial latch) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC (§2) | | [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set | | [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform | | [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes | diff --git a/docs/nics/connectx-5/README.md b/docs/nics/connectx-5/README.md index 620569e..ca1e540 100644 --- a/docs/nics/connectx-5/README.md +++ b/docs/nics/connectx-5/README.md @@ -12,7 +12,7 @@ Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set: - Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51. - Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model. - RollBall (0x51-resident, offset-model) fits MCIA — low risk. -- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig — or the X520 as fallback product NIC if the bucket exploration validates ([../../open-questions.md](../../open-questions.md) §2, §6). +- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../../open-questions.md](../../open-questions.md) §2, §6). ## Arrival notes diff --git a/docs/nics/x520/README.md b/docs/nics/x520/README.md index 33b3e42..6b4af47 100644 --- a/docs/nics/x520/README.md +++ b/docs/nics/x520/README.md @@ -1,6 +1,6 @@ # Intel X520-DA2 (82599, ixgbe) -The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse. Cannot run the committed measurement path — PTP-only RX timestamping ([../../open-questions.md](../../open-questions.md) §2) — unless the bucket-timestamping exploration validates an equivalent. +The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial PTP-only latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault ([../../open-questions.md](../../open-questions.md) §2). The ConnectX-5 is the product NIC. - PCIe Gen2 ×8 (5 GT/s, 32 Gb/s) — enough for 2×10G full duplex at the default mix. - **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it. @@ -10,16 +10,22 @@ The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmwa Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe//sff_i2c` (`w ` / `r `). - **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte. -- **Two-master windows**: each debugfs op takes the semaphore separately; serialization against the driver's own I2C holds per-op, but multi-op protocol sequences have unlocked gaps, and driver I2C is event-driven (SFP identify, link setup, `ethtool -m`) — firing exactly around cable swaps. Hardening item: a compound op under one semaphore hold ([../../open-questions.md](../../open-questions.md) §4). +- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave. The `x ` command (`ixgbe_i2c_raw_xfer`) runs write→delay→read under a single swfw hold, removing the window. Proven on the FS BCM bridge ([../../open-questions.md](../../open-questions.md) §4). + +## Register/timestamp bench transport (patched driver) + +For the §2 exploration the driver also carries `reg_ops` `read ` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm `, `control`, `latchrate `) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`. ## Timestamping facts -- RX latches **PTP frames only** (`TYPE_ALL`/TSIP are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`): one latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H), no RX-descriptor timestamp field. -- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset. -- Per-queue hardware counters QPRC/QBRC (16 slots via RQSMR, **read-clear**, `ixgbe_main.c:7777,7787`) count at queue write-back, upstream of ring residence — the basis of bucket-exploration route 1. -- Full analysis and bench pokes: [../../open-questions.md](../../open-questions.md) §2. +**No all-packet exact RX timestamping, and no path to it — this disqualifies the X520 as product NIC.** A cable tester needs each frame's exact arrival time; the 82599 cannot give it for the 0x88b5 test traffic. + +- RX latches **PTP frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood; `TSIP_UT_EN` (0x00800000) will not even hold (both are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field. +- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial and PTP-only, so it stamps a fraction of frames at best and none of the raw traffic. Its only surviving use is the low-volume length probes (§2/§3). +- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset (length-probe use only). +- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not. Measured facts (RQSMR maps 1:1; QPRC/QBRC exact, read-clear; the 2 s stats watchdog is a competing reader, `ixgbe_main.c:7780,7789,7822`) are kept in [../../open-questions.md](../../open-questions.md) §2, but the route is rejected. ## RX steering facts -- ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only) — the tool fails at startup as committed. -- **ETQF/ETQS L2 EtherType filters are the answer on paper**: dedicated ethertype→queue stage ahead of RSS/fdir, 8 slots, 7 free (index 3 reserved for 1588). Flow Director cannot match non-IP frames at all. Analysis and patch shape: [../../open-questions.md](../../open-questions.md) §1. +- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged. +- **ETQF/ETQS L2 EtherType steering is proven on hardware**: dedicated ethertype→queue stage ahead of RSS/fdir; 7 streams land on 7 queues, unfiltered ethertypes fall to queue 0. Flow Director cannot match non-IP frames at all (a flex-word rule on raw 0x88b5 steers nothing — confirmed). Details and the FCoE-slot fix: [../../open-questions.md](../../open-questions.md) §1. diff --git a/docs/open-questions.md b/docs/open-questions.md index 6045949..bd5d8c5 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -6,18 +6,18 @@ Framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attri ## 1. Raw-L2 RX steering on 82599 -**Status: answered on paper — use ETQF, bench-confirm.** The tool steers raw ethertype 0x88b5+ to per-queue NAPI contexts; ixgbe rejects ETHER_FLOW ntuple rules. +**Status: solved and proven on hardware — ETQF steers raw ethertypes to per-queue contexts.** The committed tool steers raw ethertype 0x88b5+ to per-queue NAPI contexts; stock ixgbe rejects ETHER_FLOW ntuple rules. The patched ixgbe maps exact-ethertype ETHER_FLOW inserts onto ETQF slots (`~/work/ixgbe-sff/ixgbe_ethtool.c`, `ixgbe_add_etqf_entry`), so `system.go` runs unchanged. -- **Flow Director is a dead end for raw L2.** 82599 fdir classifies IPv4/IPv6 only; the flex word (`FDIRCTRL_FLEX_SHIFT = 0x6` = byte offset 12 = the ethertype field) *narrows* an IP-typed match but cannot create a non-IP one — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7. - **ETQF/ETQS is a dedicated ethertype→RX-queue stage** checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses it — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`). -- **Slot budget fits**: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588, needed by §2) is reserved → 7 free = exactly the 7 streams. -- **Patch shape**: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots in the ixgbe ethtool path, so `system.go` works unchanged. Far smaller than fdir surgery; we already carry a driver patch. +- **Bench-confirmed** (`~/work/phydiag-work/etqfbench`): 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each on their queue; an 8th unfiltered ethertype falls to queue 0; delete restores the fall-through. `ETHTOOL_GRXCLSRULE`/`GRXCLSRLALL` round-trip the ETQF rules so the tool's stale-rule cleanup sees and clears them. +- **Slot budget fits**: 8 filters; the patch reserves index 3 (1588, needed by §2) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes were moved behind the FCoE-enabled guard (`ixgbe_fcoe.c`) so the default config leaves slots 0/1/2 free; without that, FCoE CRC pre-claimed slot 2. Restore-on-link-up (`ixgbe_etqf_filter_restore`) and clear-on-close (`ixgbe_etqf_filter_exit`) are hooked so a ring-resize reset re-applies the rules. +- **Flow Director is a dead end for raw L2 — confirmed on hardware.** 82599 fdir classifies IPv4/IPv6 only; a flex-word rule (`FDIRCTRL_FLEX_SHIFT = 0x6` = byte offset 12 = the ethertype field) targeting a raw 0x88b5 stream steers nothing — every frame falls to queue 0 (`etqfbench -mode fdirneg`). Non-IP frames never enter the fdir lookup; the flex word only *narrows* an IP-typed match. - Baseline behavior without filters: non-IP frames hash to RSS 0 → all land on queue 0. -- Fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred (no framing change, no header overhead in the size mix). +- Fallbacks (unneeded now): MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix). ## 2. Zero-baseline loss and timestamping on the 82599 -**Status: throughput yes; per-packet timestamps no — exploration routes mapped, plus one decisive experiment runnable on the X710 ahead of any build-out.** +**Status: throughput yes; all-packet exact RX timestamps no — and there is no substitute. A cable tester needs each frame's exact arrival time; aggregate counters/rates are honest only when nothing is wrong, which is useless for fault detection. The 82599 has one serial PTP-only latch and nothing else, so the X520 is disqualified as the product measurement NIC (diagnostics-only). The ConnectX-5 is the product path.** ### Throughput @@ -33,22 +33,25 @@ Framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attri **There is no software fallback for use 1** — software RX stamping was tried and cannot reach the needed precision. All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright. (measurement.md's read-time-stamping lesson covers the *NIC-counter* buckets, a separate system.) -### The full 82599 hardware space for use 1 — routes to honest buckets without per-packet stamps +### The full 82599 hardware space for use 1 — and why none of it substitutes for per-packet stamps -What X550 calls "timestamp all" is `TYPE_ALL` filtering **plus TSIP** (`TSYNCRXCTL_TSIP_UT_EN` — stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 documents neither: one RX latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`), attribute-capture registers (RXSATRL/H, unused by the driver), no timestamp field in the RX descriptor. Routes, best first: +**A cable tester measures the receive precisely when it is misbehaving — loss bursts, late frames, timing excursions to correlate with pre-FEC errors and retrains. That signal lives in each frame's exact arrival time. Any aggregate (a rate, a count per window) is only honest when the receive is well-behaved, i.e. when there is nothing to measure. Exact per-frame timestamps are the requirement, full stop; nothing on the 82599 provides them for all packets.** -1. **Per-queue hardware counters as bucket content — likely sufficient.** QPRC/QBRC (16 slots; queues map via RQSMR; 7 streams fit) increment at queue write-back — *upstream of ring residence*, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate (more than one 16 ms bucket), while the RX FIFO ahead of the counters bounds smear at 512 KB ≈ 400 µs worst-case, ≈ 0 in zero-loss steady state. Implementation: an hrtimer sampler in the patched driver (1–16 ms cadence, snapshot slots + ktime into a page the tool maps) — hardware-counted content, read-time-labeled boundaries, the same honesty discipline measurement.md proved. Constraint: QPRC/QBRC are read-clear (`ixgbe_main.c:7777,7787`) — exactly one reader; the sampler owns the reads and feeds the driver's accumulators. Nothing lost: `rateRun` only ever carried frames/bytes per bucket. -2. **The latch as an identifiable sampler.** Reading RXSTMPH re-arms it. Anchor frames shaped as L2 PTP V2 events carrying our sequence counter in the PTP sequenceID field get MAC-time stamps; RXSATRL/H capture the identifying bytes. Even ~1 k stamps/s continuously validates route 1's boundaries and measures FIFO+ring residence live. -3. **Wire-determinism reconstruction — the reserve.** At saturation the TX ring drains at exactly line rate; per-frame wire times are computable from sequence numbers + the size schedule, anchored by sparse latch stamps. Breaks off-saturation; only if 1–2 fall short. +What X550 calls "timestamp all" is `TYPE_ALL` filtering **plus TSIP** (`TSYNCRXCTL_TSIP_UT_EN` — stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 has neither — confirmed on hardware: writing `TYPE_ALL` sticks in the register but never arms the latch on raw frames, and the `TSIP_UT_EN` bit will not even hold. All it has is one RX latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`) with attribute-capture (RXSATRL/H), no timestamp field in the RX descriptor. -Bench pokes, first day on the card: +The routes that were mapped here are all **rejected** for a cable tester: -- Write `TYPE_ALL` (0x08) into TSYNCRXCTL and blast raw frames — defined silicon-wide in the header, documented only for X550; if the reserved encoding secretly arms the latch, route 2 sheds its PTP-shaping constraint (put the stream seq at the RXSATR capture offsets). Poke `TSIP_UT_EN` and look for a buffer prepend. Expected dead; minutes to know. -- RQSMR mapping, QPRC read-clear semantics, coexistence with the watchdog stats task. -- Latch service rate: sustainable identifiable stamps/s. -- Route-1 sampler prototype, validated against route-2 anchors and against software stamps at low load (where software is honest). +1. ~~Per-queue hardware counters as bucket content.~~ **Rejected — it's a rate, not timestamps.** QPRC/QBRC deltas per window give aggregate throughput, honest only in zero-loss steady state (the FIFO smear that would perturb a boundary window is ~0 only when nothing is going wrong). It measured cleanly on the bench (RQSMR maps 1:1, read-clear, exact counts) but it cannot place a single frame in time, so it cannot see the fault a cable tester is built to catch. Measured facts retained below; the conclusion is dead. +2. ~~The latch as an identifiable sampler.~~ **Not all-packet.** One serial latch, PTP-only, ~383 k stamps/s from a dedicated in-kernel spin loop against ~1.78 M pps/dir line rate. Useful only for the low-volume length probes (below), never for stamping the 0x88b5 test traffic. +3. ~~Wire-determinism reconstruction.~~ **Rejected — same failure as counters.** Reconstructing per-frame wire times from the size schedule assumes saturation regularity, which a loss/retrain event breaks; it is blind at exactly the interesting moments. -If route 1 validates, the X520 delivers honest buckets — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the MCIA fallback (§6). +**Verdict: the 82599 cannot do all-packet exact RX timestamping and has no path to it. The X520 is disqualified as the product measurement NIC — diagnostics / BCM-framing card only.** The ConnectX-5 (per-packet CQE stamps, `HWTSTAMP_FILTER_ALL`) is the product path (§6). + +Bench-poke facts (measured, retained for the record even though the counter conclusion is dead): + +- `TYPE_ALL` writable but inert for raw frames; `TSIP_UT_EN` will not hold — the latch is PTP-only silicon, not a hidden all-packet path. +- RQSMR maps queue→counter 1:1; QPRC/QBRC exact and read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`). +- Single PTP latch services ~383 k identifiable stamps/s; stream seq lands in `RXSATRH[31:16]` byte-swapped. ### Probes on the 82599 @@ -59,6 +62,8 @@ If route 1 validates, the X520 delivers honest buckets — its disqualification The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the timestamp length method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches — probes must be PTP-shaped there too): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. ≲ 10–20 ns → viable; much more → close §3's timestamp path permanently. +The bench pokes (measured facts above) were run through the patched driver's debugfs — `reg_ops` returns read values, and `ts_bench` runs the latch poll/re-arm loop in-kernel because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM` (drivers: `~/work/phydiag-work/x520poke`, `ts_bench` in `~/work/ixgbe-sff/ixgbe_debugfs.c`). What they establish for the 82599: no all-packet exact-timestamp path exists, so the only surviving use of the latch is the low-volume **length probes** (below), where sparse PTP-shaped anchors are exactly the model — not the line-rate 0x88b5 measurement. + ## 3. Cable-length strategy **Status: consolidated — PHY features are the product path, NIC timestamps the fallback. The BCM path works.** @@ -74,13 +79,13 @@ The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the timestamp le ## 4. Two-master I2C safety and the once-untested client assumptions -**Status: verified against source and hardware; one hardening item remains.** +**Status: verified against source and hardware; the hardening item is built and proven.** - **Per-op semaphore bracketing is real and is the weak point.** Each debugfs `w`/`r` acquires/releases swfw individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`); clients compose protocols from separate ops with userspace sleeps. Every multi-op sequence has windows where the driver can run its own I2C. - **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask` — no mid-transaction bus corruption. But driver traffic is event-driven (SFP identify after module/link events, link setup, `ethtool -m`) — it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but correlated with the interesting moments. - **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the risk is the other master *moving* the pointer — an interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence. -- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe (the driver never addresses 0x56; I2C is address-filtered) — the one interleaving case without proof. -- **Remaining hardening item**: a compound debugfs op (write bytes, STOP, optional delay µs, read n) under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds it longer during SFP identify. Same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the bus across multi-transfer sequences). +- **The BCM 0x56 bridge's pending read data** (0x56 7-bit = 0xAC/0xAD write/read) across the >1 ms window was the one interleaving case without proof; the compound `x` op removes the window rather than reasoning about it, so the case is now closed by construction. +- **The compound op is built and proven.** `sff_i2c` gains an `x ` command (`ixgbe_i2c_raw_xfer`, `ixgbe_phy.c`): write, STOP, in-kernel delay, read — all under a single swfw hold, closing the interleave window entirely. The write/read primitives were refactored into locked/unlocked halves so `x` reuses them under one acquire. Validated on the FS BCM (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004). A ~3 ms hold is nothing — the driver holds swfw longer during SFP identify. The delay sweep corroborates the not-ready hazard the op closes: at delay 0 the bridged read returns stale data (0x0004 left over from the prior read), at ≥200 µs it reads correctly — precisely the stale-pointer/stale-data corruption an interleaved driver access would cause. - **RollBall split transactions: field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules. - **The >1 ms BCM delay**: validated on hardware at 3 ms. Treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature. @@ -104,4 +109,4 @@ Design points: - **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation (§5), and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot. - **The single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (§2) and raw-L2 steering is not (§1) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only; txgbe is also PTP-only (verified in upstream `wx_ptp.c`); **ConnectX-4/5 (mlx5) is the only candidate meeting the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (`probe.go`'s assumption holds), native ETHER_FLOW ntuple steering, mature driver. - **Open on the ConnectX**: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only use 0x50/0x51) and whether the BCM SMI read data phase (raw 2-byte read, no offset write) can be framed. RollBall (0x51-resident, offset-model) fits. -- **If MCIA can't reach the BCM**: product diagnostics via the Marvell/Aquantia modules only, with the X520 kept as an offline BCM rig — its bit-banged framing stays the only *certain* SMI transport — or, if the §2 bucket exploration validates, the X520 itself as fallback product NIC. +- **If MCIA can't reach the BCM**: product diagnostics via the Marvell/Aquantia modules only, with the X520 kept as an offline BCM rig — its bit-banged framing stays the only *certain* SMI transport. The X520 is **not** a fallback product NIC: it has no all-packet exact-timestamp path (§2), and a cable tester's rate buckets require exactly that. diff --git a/docs/state.md b/docs/state.md index e615d64..7aae4c8 100644 --- a/docs/state.md +++ b/docs/state.md @@ -34,13 +34,14 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet | 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set | | E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work | -**The X520 cannot be the product NIC**: rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement) and the 82599 stamps PTP only. It carries two interim campaigns: the diagnostics bring-up (certain arbitrary I2C framing) and the 82599 bucket-timestamping exploration (open-questions.md §2). If that exploration validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX. +**The X520 is disqualified as the product NIC — measurement, not a narrow gap.** Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, open-questions.md §2). Per-queue counters were explored and **rejected** — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC. ## X520 diagnostics path — validated on hardware In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`: -- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w ` / `r `). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`. +- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w ` / `r ` / `x ` single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: **ETQF steering** (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in `ixgbe_ethtool.c`, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); **`reg_ops` read-returns-value** and a **`ts_bench`** debugfs command (in-kernel latch poll/re-arm for the §2 pokes). `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`. +- **Bench tools** (`~/work/phydiag-work/`): `etqfbench/` (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), `x520poke/` (Go; drives `reg_ops`/`ts_bench` for the timestamp/counter pokes), `compound_test.py` (proves the `x` compound op against the FS BCM). - **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via `CMD_GET_SNR = 0x8030` invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/. - **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/). - **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill. @@ -51,20 +52,15 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker 2. ~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source. 3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only. 4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs). -5. RX steering: program/verify the ETQF path (open-questions.md §1) before cabletest traffic runs. +5. ~~RX steering: program/verify the ETQF path (open-questions.md §1).~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (`~/work/phydiag-work/etqfbench`). `system.go` runs unchanged. ## RX flow-steering on ixgbe -cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW — so the insert is rejected and the tool fails at startup (no fallback, by design). This affects the datapath regardless of module type. +**Solved on hardware.** cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (`ixgbe_ethtool.c`), so `system.go` runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (`~/work/phydiag-work/etqfbench`). Full analysis in open-questions.md §1. -Leads, most promising first (full analysis in open-questions.md §1): - -- **ETQF/ETQS L2 EtherType queue filters — the answer on paper.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588); already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto ETQF slots so `system.go` is unchanged. Bench-confirm. -- **Flow Director flex-byte match — ruled out on paper.** fdir classifies IPv4/IPv6 only; the flex word narrows an IP match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding. -- **MAC → VMDq pool steering** — distinct dest MACs → pools → queues, fully raw. Fallback if ETQF disappoints. -- **RSS** — hashes IP tuples only; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline). - -Fallbacks if raw-L2 steering can't work: minimal bare-IPv4 framing steered by IP_USER_FLOW, or single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable as a fallback (open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice. +- **ETQF/ETQS L2 EtherType queue filters — proven.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free. +- **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match. +- **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice. ## Open items