cabletest/docs/measurement.md

Measurement lessons

Hard-won rules about measuring correctly. Violating these produces numbers that look plausible and are wrong.

Rate buckets: stamp at read time, never at ticker-fire time

• direction.capture() stamps each bucket with time.Now() taken immediately after reading the counters — never a timestamp passed in from the sampler ticker.
• Why: a rate is Δcounters/Δt. The sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator — and because E[1/x] > 1/E[x], the error biases the rate upward; it doesn't cancel.
• Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high. Read-time stamping read within 2%.
• A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it.
• Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew.

This rule covers the NIC-counter buckets and the per-stream receive buckets alike: the receive buckets are keyed by the clock read immediately after each recvmmsg drain, never by the ticker. Their read jitter is then repaired by the backward excess-fill smear (nics/x520/) — bare read-time stamping alone was tried in the E810 era and could not reach the needed precision, which is why per-packet MAC RX stamps were once a hard host requirement; the smear is what dissolved it (nics/README.md). Per-frame hardware stamps are no longer used anywhere.

A sliding smear must settle each bucket once, never recompute

• Any pass that moves quantity between buckets of a sliding window must run once per bucket with its result persisted (enter → donate → display-and-retire). Recomputing the pass from raw buckets at every sample shows moved quantity twice: once in the bucket it moved to (displayed early, then slid out) and again in its source (whose donation is forgotten by the time it reaches the display slot).
• Measured: the recompute form read 20.0–20.2 Gb/s on a 20 Gb/s wire — impossible throughput manufactured from double-counted burst excess.
• The settled form also caps the display at line rate whenever the matching deficit is in-window; residual above-rate readings then only appear when a stall outlives the window, which is the honest signal to resize it.

NIC counters are not monotonic

• /sys/class/net/*/statistics/* run from boot, not process start, and reset to zero on driver stats resets.
• In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run; a backward step underflows now - base to ~2^64.
• Rule: accumulate only forward motion into a process-local total, then take every baseline from that total — never from a raw reading. Establish all baselines in one place before traffic starts.
• Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug.

Only zero-baseline-loss runs count; small frames measure the host

• A run counts only when tx frames == rx frames exactly with a clean cable — any nonzero baseline loss masks real cable faults.
• Small-frame runs are CPU/host-bound, not cable-bound; cable conclusions drawn from them are false.
• Keep to the default config (hardware.md) and judge from steady state — the first ~5 s is settling transient, not residual error.

Measured performance

The committed X520 path at the default mix: steady 20.00–20.04 Gb/s (line rate both directions, ~1.78 Mpps/dir), zero lost/corrupt/link/internal over a 90 s run, including startup.

Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not NIC-specific:

Config	Result
64 B only	pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s
AF_XDP experiment (stashed)	64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX

• RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810).
• The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.

Dead ends — measured, do not re-attempt without new hardware

Measured in the E810 era; the binding constraint was total CPU across ~28 goroutines on 20 threads.

• Splitting TX senders from RX streams — raises TX but collapses RX (RX scales with queue count, capped at 7 on that NIC).
• CPU pinning — the Go scheduler beats manual placement; E-cores are poor at TX.
• Batch sizes above 64 — no gain, worse loss.
• Chasing the TX sendmmsg busy-spin on ENOBUFS — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).

rxnfc flow-rule programming — three traps (found on ice; 1 and 2 are kernel-ABI-generic)

1. Mask polarity is inverted vs ethtool -n display. In raw m_u bytes a set bit means "must match": a working "match ethertype, ignore MACs" rule has m_u.ether_spec = dst 00×6, src 00×6, proto ff ff. ethtool -n prints the complement — trusting its display gives an inverted rule that silently matches nothing.
2. rule_locs sits at offset 188, not sizeof(struct ethtool_rxnfc) (192), on amd64 — rule_locs[] follows rule_cnt at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
3. ice rejects RX_CLS_LOC_ANY with ENOSPC — allocate a free location yourself (capacity from ETHTOOL_GRXCLSRLCNT's data; the CLI allocates downward from the top). When a rule inserts but steers nothing, diff the raw bytes of a CLI-made known-good rule against yours.