cabletest/docs/measurement.md

# Measurement lessons

Hard-won rules about measuring correctly. Violating these produces numbers that look plausible and are wrong.

## Rate buckets: stamp at read time, never at ticker-fire time

- `direction.sample()` reads the clock immediately after reading the counters — never a timestamp passed in from the sampler ticker.
- Why: a rate is Δcounters/Δt. The sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator — and because E[1/x] > 1/E[x], the error biases the rate **upward**; it doesn't cancel.
- Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high. Read-time stamping read within 2%.
- A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it.
- Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew.

This rule covers the **NIC-counter** buckets and the per-stream receive buckets alike: the receive buckets are keyed by the clock read immediately after each `recvmmsg` drain, never by the ticker. Their read jitter is then repaired by the backward excess-fill smear (nics/x520/) — bare read-time stamping alone was tried in the E810 era and could not reach the needed precision, which is why per-packet MAC RX stamps were once a hard host requirement; the smear is what dissolved it (nics/README.md). Per-frame hardware stamps are no longer used anywhere.

## A sliding smear must settle each bucket once, never recompute

- Any pass that moves quantity between buckets of a sliding window must run once per bucket with its result persisted (enter → donate → display-and-retire). Recomputing the pass from raw buckets at every sample shows moved quantity twice: once in the bucket it moved to (displayed early, then slid out) and again in its source (whose donation is forgotten by the time it reaches the display slot).
- Measured: the recompute form read 20.0–20.2 Gb/s on a 20 Gb/s wire — impossible throughput manufactured from double-counted burst excess.
- The settled form also caps the display at line rate whenever the matching deficit is in-window; residual above-rate readings then only appear when a stall outlives the window, which is the honest signal to resize it.

## NIC counters are not monotonic

- `/sys/class/net/*/statistics/*` run from boot, not process start, and reset to zero on driver stats resets.
- In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run; a backward step underflows `now - base` to ~2^64.
- Rule: only forward motion counts — the delta layer clamps a backward step to zero and re-bases, never handing a raw difference upward. Establish all baselines in one place before traffic starts.
- Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug.

## Only zero-baseline-loss runs count; small frames measure the host

- A run counts only when `tx frames == rx frames` exactly with a clean cable — any nonzero baseline loss masks real cable faults.
- Small-frame runs are CPU/host-bound, not cable-bound; cable conclusions drawn from them are false.
- Keep to the default config (hardware.md) and judge from steady state — the first ~5 s is settling transient, not residual error.

## Measured performance

The committed X520 path at the default config (2 streams, 2.0 GHz pinned clock): steady 20.00–20.04 Gb/s (line rate both directions, ~0.75 Mpps/dir at the jumbo mix), zero lost/corrupt/link/internal over a 96 s run, including startup, at ~30 W CPU package on the bench host. The pre-jumbo mix ran ~1.78 Mpps/dir at the same rate.

Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not NIC-specific:

| Config | Result |
|---|---|
| 64 B only | pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s |
| AF_XDP experiment ([xdp.md](xdp.md)) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |

- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810).
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). Its code is dropped; the design record and its constraints live in [xdp.md](xdp.md) — its novelty was delivery mechanism and throughput, not the use of hardware stamps.

## Power: the busy cores' voltage is the cost, not the idle cores

Measured on the bench host (i9-13900H, 6P+8E, 20 threads), RAPL package power at line rate both directions:

| Config | Package | Wire |
|---|---|---|
| 7 streams, unbounded clock | 80.0 W | clean |
| 2 streams, unbounded clock (4.2–4.6 GHz) | 70.7 W | clean |
| 2 streams @ 2.4 GHz | 35.0 W | clean |
| **2 streams @ 2.0 GHz (shipped)** | **29.7 W** | clean, 43% busy |
| 2 streams @ 1.6 GHz | 25.1 W | clean, 49% busy |
| 2 streams @ 1.2 GHz | 20.4 W | **fails — wire sags to 17.5–19.8 G, zero errors: TX starvation, the honest failure mode** |
| 7 streams @ 1.6 GHz | 29.8 W | line rate but 94% busy and the rate reading wobbles |
| idle, no tool | 10.9 W | — |

- The frequency pin is the lever: core power 56 → 16 W is V² savings on the busy cores. ~13.5 W of uncore (PCIe/memory fabric) is a floor no CPU knob touches.
- Stream count buys scheduling headroom, not watts: 7→1 streams unbounded saves only ~13 W, because freed threads park in C1 (the 10 µs cpu_dma_latency cap) while the surviving TX spinners boost harder. At a fixed clock the cut is what makes the low clock safe — 49% busy at 2 streams where 7 needs 94%.
- Every TX worker is a full-time spinner (the ENOBUFS spin is the flow control; there is no blocking send under PACKET_QDISC_BYPASS), so TX cost scales with stream count regardless of traffic; RX blocks in recvmmsg and scales with traffic.

## Dead ends — measured, do not re-attempt without new hardware

Measured in the E810 era; the binding constraint was total CPU across ~28 goroutines on 20 threads.

- **Splitting TX senders from RX streams** — raises TX but collapses RX (RX scales with queue count, capped at 7 on that NIC).
- **CPU pinning for performance** — the Go scheduler beats manual placement; E-cores are poor at TX. (The TX feeders are nevertheless each locked to a dedicated cpu, spaced off shared hyperthreads: XPS keys the hardware ring on the sending cpu, and a stable per-stream ring is an ordering requirement, not a speed one.)
- **Batch sizes above 64** — no gain, worse loss.
- **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
- **Letting idle cores sleep by relaxing cpu_dma_latency** (200 µs admits the 127 µs C2_ACPI) — steady-state deep-idle residency ~2% and power unchanged: the Go scheduler spreads wakeups across every thread, so no core idles long enough for the idle governor to pick a deep state.
- **Offlining cores for power** (all 8 E-cores) — 1.7 W. The offline event mid-run is invisible on the wire (no blip at line rate), so parking cores is safe if ever wanted for another reason; it is not a power lever. Note the SETTINGS governor check fails loudly with offline cpus present — it reads every `cpu*/cpufreq/scaling_governor` and an offline cpu returns EBUSY.
- **Alien-crosstalk injection from an intertwined noise cable** — bench-measured: two unshielded cables tightly wrapped moved the victim's per-pair SNR margin only 0.2–0.3 dB, far too little to stress even a marginal cable; 10GBASE-T's alien-noise budget absorbs it. The noise pair, its link-cycling, and the panel lane are removed from the tool.

## rxnfc flow-rule programming — three traps (found on ice; 1 and 2 are kernel-ABI-generic)

1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement — trusting its display gives an inverted rule that silently matches nothing.
2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192), on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
3. **ice rejects `RX_CLS_LOC_ANY`** with ENOSPC — allocate a free location yourself (capacity from `ETHTOOL_GRXCLSRLCNT`'s `data`; the CLI allocates downward from the top). When a rule inserts but steers nothing, diff the raw bytes of a CLI-made known-good rule against yours.