60 lines
13 KiB
Markdown
60 lines
13 KiB
Markdown
# Intel X520-DA2 (82599, ixgbe)
|
||
|
||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate plan below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The plan, its prerequisite audit, and what it retires are in the section below.
|
||
|
||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
|
||
- Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
|
||
- **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
|
||
|
||
## The product plan: smoothed-bucket rate, no hardware stamps
|
||
|
||
The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. The plan replaces arrival-time bucketing with read-time bucketing plus a deconvolution pass, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.
|
||
|
||
- **Rate is a headline, never a verdict.** Lost/late/corrupt come from per-frame sequence numbers and CRC checks, independent of the rate; no cable-health decision keys off the rate number. This is the license for everything below.
|
||
- **Buckets are keyed by read time; excess above line rate moves backward.** Read jitter is a purely backward smear — a frame is read at or after its arrival, never before — so a bucket's excess above line rate is always frames that arrived earlier and were read late, and moving that excess backward to fill earlier deficits is the physically correct deconvolution, not an estimate. The dominant jitter shape (host stalls, then drains the backlog) is deficit-then-burst, which is exactly what the backward pass repairs. Line rate is known, so over/under is well-defined.
|
||
- **A forward pass is optional edge polish.** A stall/burst pair entirely inside the window balances under the backward pass alone; only pairs cut by the window boundary leave orphan excess or deficit. Pairing those up forward is cosmetic, is the pass most likely to paper over a genuine dip at the window edge, and is droppable if it ever lies.
|
||
- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge (measure the actual stall distribution before choosing). The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs.
|
||
- **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
|
||
- The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.
|
||
|
||
**Prerequisite audit before building**: confirm the rate buckets are the *only* consumer of hardware RX stamps in the committed path. The buckets are currently keyed by the MAC's clock (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`), and `probe.go` leaned on a shared-PHC assumption; anything else that needs true arrival times (one-way latency, cross-port stamp comparison) does not survive the change. "Late" must remain a sequence-number notion, not a timestamp one.
|
||
|
||
What this enables and retires:
|
||
|
||
- **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal.
|
||
- **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears.
|
||
- **Build on HEAD** — the committed X520/BCM/ECD implementation, not the CX-5 stash. The work: re-key the receive buckets from MAC-stamp time to read time, add the backward-fill pass, and remove the `rx_filter=ALL` hard host check (`ts.go`) whose requirement this dissolves. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
|
||
|
||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||
|
||
Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
|
||
|
||
- **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
|
||
- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave — traffic that fires exactly around cable swaps, when diagnostics also run, so collisions are rare but correlated with the interesting moments. Serialization itself holds (driver SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`), and AT24-style word pointers do persist across STOP — the hazard is the other master *moving* the pointer or consuming the BCM 0x56 bridge's pending read data. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→STOP→in-kernel delay→read under a single swfw hold, closing the window by construction; the write/read primitives are split into locked/unlocked halves so `x` reuses them under one acquire.
|
||
- **Compound op proven on the FS BCM** (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004); a ~3 ms hold is shorter than the driver's own SFP-identify holds. The delay sweep corroborates the hazard the op closes: at delay 0 the bridged read returns stale data (the prior read's 0x0004), at ≥200 µs it reads correctly. The >1 ms BCM delay is validated at 3 ms; treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.
|
||
- **RollBall split transactions are field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks; the kernel's combined form is not required by the modules.
|
||
|
||
## Register/timestamp bench transport (patched driver)
|
||
|
||
For the timestamp/counter bench pokes the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
|
||
|
||
## Timestamping facts
|
||
|
||
**No all-packet exact RX timestamping, and no path to it — measured, settled, and no longer disqualifying.** The 82599 cannot stamp the 0x88b5 test traffic per-frame; the smoothed-bucket plan above removes the need. The facts stand for the record:
|
||
|
||
- RX latches **1588-recognized frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08 = Type 100b) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood — and the datasheet marks 100b *reserved* on the 82599 (it's an X550 define); `TSIP_UT_EN` (0x00800000) will not even hold (X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
|
||
- RX latch eligibility is configurable, not hardwired to 0x88F7 (datasheet §7.9.3.2, §8.2.3.26.1, ETQF bit 30): a frame qualifies when its EtherType sits in an ETQF slot with the 1588 bit set and TSYNCRXCTL.Type admits its message class — valid encodings 000b (L2 sync/delay_req + msgID 2,3), 001b (V1/L4), 010b (V2 L2+L4), 101b (**every V2 event frame**: payload byte 0 low nibble with bit 3 clear). Declaring 0x88b5 as 1588 and shaping byte 0 would latch our own raw frames — datasheet-derived, untested on hardware, and irrelevant to the verdict: the serial ceiling is unchanged.
|
||
- TX latch has no parser (§8.2.3.26.7): the advanced TX data descriptor's 1588 bit (MAC field, bit 1) stamps **any** marked frame into TXSTMPL/H — serial, TXTT cleared by TXSTMPH read. Both directions therefore stamp arbitrary frames one-at-a-time: a ping-pong probe gets PHY-adjacent hardware stamps on both ends, paced by the MMIO read loop.
|
||
- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial, so it stamps a fraction of frames at best and none of the line-rate traffic. NIC-timestamp length measurement is discarded (length strategy: [../../modules/README.md](../../modules/README.md)), so the latch has no committed use.
|
||
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal; the two SYSTIMEs start at driver load and can sit seconds apart.
|
||
- SDP auxiliary elements (§7.9.4): an edge on SDP2/3 latches SYSTIME into AUXSTMP0/1; target-time registers assert an SDP pin when SYSTIME reaches a programmed value; SDP6/7 carry configurable clock outputs. No product use.
|
||
- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not — it cannot place a single frame in time, so it cannot see the fault. Measured facts retained for the record: RQSMR maps queue→counter 1:1; QPRC/QBRC exact, read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`). Also rejected: wire-determinism reconstruction (per-frame times from the size schedule assume saturation regularity — blind exactly at loss/retrain events).
|
||
|
||
## RX steering facts
|
||
|
||
- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged; `ETHTOOL_GRXCLSRULE`/`GRXCLSRLALL` round-trip the rules so stale-rule cleanup sees and clears them.
|
||
- **ETQF/ETQS L2 EtherType steering is proven on hardware** (`~/work/phydiag-work/etqfbench`): a dedicated ethertype→queue stage checked at L2, ahead of RSS/fdir — ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE + QUEUE_EN, and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`). 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each; unfiltered ethertypes fall to queue 0 (non-IP frames hash to RSS 0); delete restores the fall-through.
|
||
- **Slot budget fits**: 8 filters; the patch reserves index 3 (1588) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes moved behind the FCoE-enabled guard (without that, FCoE CRC pre-claims slot 2). Restore-on-link-up (`ixgbe_etqf_filter_restore`) and clear-on-close (`ixgbe_etqf_filter_exit`) are hooked so a ring-resize reset re-applies the rules.
|
||
- **Flow Director is a dead end for raw L2 — confirmed on hardware**: 82599 fdir classifies IPv4/IPv6 only; a flex-word rule at the ethertype offset (`FDIRCTRL_FLEX_SHIFT = 0x6`) on a raw 0x88b5 stream steers nothing (`etqfbench -mode fdirneg`) — the flex word only *narrows* an IP-typed match.
|
||
- Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap acceptable — raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix).
|