X520 live as the product NIC: rate buckets re-keyed to read time (one shared clock, one commit per drained batch) with the backward smear as a settled window — each completed bucket enters once, donates excess to the nearest earlier deficits once, locks for display when nothing later can refill it (per-sample recompute double-counted excess and read >20G on a 20G wire); timestamp machinery deleted (ts.go, SO_TIMESTAMPING, rx_filter=ALL check) after the audit confirmed the buckets were its sole consumer; testDriver ixgbe; RollBall mailbox hardened against orphaned-command completions (never sample status before a poll gap, value from the same block read, devad/reg echo required — a killed session's 7.60 read answered a PHY ID as 0x0006); full hardware pass: 20.00G flat both directions, zero errors, ECD 42m, SNR +7.4dB

This commit is contained in:
flamingcow
2026-08-17 09:35:58 -07:00
parent 06de095d1c
commit 3d7105073e
15 changed files with 313 additions and 404 deletions
+8 -2
View File
@@ -10,7 +10,13 @@ Hard-won rules about measuring correctly. Violating these produces numbers that
- A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it.
- Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew.
This rule covers the **NIC-counter** buckets. The per-stream application buckets are a separate system stamped by per-packet MAC RX timestamps (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check); read-time stamping is not a substitute for those — software stamping was tried and cannot reach the needed precision. NICs without all-packet RX timestamping (X710, 82599) fail the host check; see nics/README.md.
This rule covers the **NIC-counter** buckets and the per-stream receive buckets alike: the receive buckets are keyed by the clock read immediately after each `recvmmsg` drain, never by the ticker. Their read jitter is then repaired by the backward excess-fill smear (nics/x520/) — bare read-time stamping alone was tried in the E810 era and could not reach the needed precision, which is why per-packet MAC RX stamps were once a hard host requirement; the smear is what dissolved it (nics/README.md). Per-frame hardware stamps are no longer used anywhere.
## A sliding smear must settle each bucket once, never recompute
- Any pass that moves quantity between buckets of a sliding window must run once per bucket with its result persisted (enter → donate → display-and-retire). Recomputing the pass from raw buckets at every sample shows moved quantity twice: once in the bucket it moved to (displayed early, then slid out) and again in its source (whose donation is forgotten by the time it reaches the display slot).
- Measured: the recompute form read 20.020.2 Gb/s on a 20 Gb/s wire — impossible throughput manufactured from double-counted burst excess.
- The settled form also caps the display at line rate whenever the matching deficit is in-window; residual above-rate readings then only appear when a stall outlives the window, which is the honest signal to resize it.
## NIC counters are not monotonic
@@ -34,7 +40,7 @@ This rule covers the **NIC-counter** buckets. The per-stream application buckets
| AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts.
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path gets the same stamps via the `SO_TIMESTAMPING` cmsg). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path no longer uses hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
## Dead ends — measured, do not re-attempt without new hardware