X520 live as the product NIC: rate buckets re-keyed to read time (one shared clock, one commit per drained batch) with the backward smear as a settled window — each completed bucket enters once, donates excess to the nearest earlier deficits once, locks for display when nothing later can refill it (per-sample recompute double-counted excess and read >20G on a 20G wire); timestamp machinery deleted (ts.go, SO_TIMESTAMPING, rx_filter=ALL check) after the audit confirmed the buckets were its sole consumer; testDriver ixgbe; RollBall mailbox hardened against orphaned-command completions (never sample status before a poll gap, value from the same block read, devad/reg echo required — a killed session's 7.60 read answered a PHY ID as 0x0006); full hardware pass: 20.00G flat both directions, zero errors, ECD 42m, SNR +7.4dB
This commit is contained in:
@@ -1,29 +1,31 @@
|
||||
# Intel X520-DA2 (82599, ixgbe)
|
||||
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate plan below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The plan, its prerequisite audit, and what it retires are in the section below.
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate design below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The committed tree implements this design.
|
||||
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
|
||||
- Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
|
||||
- The RX error counter set is the 82599's slimmer one (`counters.go`): no jabber, fragment, illegal-byte or MAC-fault counters exist — crc/missed/length errors are what this link reports; the ice set remains the richer reference.
|
||||
- ixgbe's mixed rx/tx interrupt vectors reject a tx-specific coalesce value; `checkCoalesce` falls back to rx-shared-with-tx on EINVAL, which is the normal path here. The full rx=8160/tx=4096 ring ask is taken as-is.
|
||||
- **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
|
||||
|
||||
## The product plan: smoothed-bucket rate, no hardware stamps
|
||||
## The smoothed-bucket rate: read-time buckets, backward smear, no hardware stamps
|
||||
|
||||
The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. The plan replaces arrival-time bucketing with read-time bucketing plus a deconvolution pass, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.
|
||||
The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. Read-time bucketing plus a deconvolution pass replaces arrival-time bucketing, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.
|
||||
|
||||
- **Rate is a headline, never a verdict.** Lost/late/corrupt come from per-frame sequence numbers and CRC checks, independent of the rate; no cable-health decision keys off the rate number. This is the license for everything below.
|
||||
- **Buckets are keyed by read time; excess above line rate moves backward.** Read jitter is a purely backward smear — a frame is read at or after its arrival, never before — so a bucket's excess above line rate is always frames that arrived earlier and were read late, and moving that excess backward to fill earlier deficits is the physically correct deconvolution, not an estimate. The dominant jitter shape (host stalls, then drains the backlog) is deficit-then-burst, which is exactly what the backward pass repairs. Line rate is known, so over/under is well-defined.
|
||||
- **A forward pass is optional edge polish.** A stall/burst pair entirely inside the window balances under the backward pass alone; only pairs cut by the window boundary leave orphan excess or deficit. Pairing those up forward is cosmetic, is the pass most likely to paper over a genuine dip at the window edge, and is droppable if it ever lies.
|
||||
- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge (measure the actual stall distribution before choosing). The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs.
|
||||
- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge. The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs. Currently `smearWindow` = 4 buckets = 64 ms at the 16 ms bucket; resize after measuring the actual stall distribution.
|
||||
- **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
|
||||
- The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.
|
||||
|
||||
**Prerequisite audit before building**: confirm the rate buckets are the *only* consumer of hardware RX stamps in the committed path. The buckets are currently keyed by the MAC's clock (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`), and `probe.go` leaned on a shared-PHC assumption; anything else that needs true arrival times (one-way latency, cross-port stamp comparison) does not survive the change. "Late" must remain a sequence-number notion, not a timestamp one.
|
||||
**The audit cleared**: the rate buckets were the *only* consumer of hardware RX stamps in the committed path (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`, nothing else); "late" is and stays a sequence-number notion. The implementation: buckets keyed by one shared host clock read once per drained batch (`rateEpoch`, `rxStats.commit`), and a **settled window** — each completed bucket enters once, donates its excess above line rate backward once (`fillBack`, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window: a recompute shows every excess twice (as the donation, then again unspent when its bucket reaches the display slot) and the headline reads above line rate — observed live as 20.0–20.2 G on a 20 G wire. The whole hardware-timestamp machinery is gone: the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`). The forward pass is not built.
|
||||
|
||||
What this enables and retires:
|
||||
|
||||
- **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal.
|
||||
- **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears.
|
||||
- **Build on HEAD** — the committed X520/BCM/ECD implementation, not the CX-5 stash. The work: re-key the receive buckets from MAC-stamp time to read time, add the backward-fill pass, and remove the `rx_filter=ALL` hard host check (`ts.go`) whose requirement this dissolves. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
|
||||
- **Built on the committed X520/BCM/ECD implementation**, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
|
||||
|
||||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user