Power: 2 streams and a 2.0 GHz clock pinned by a new checkMaxFreq SETTINGS check (performance governor + scaling_max_freq = fixed clock) take line rate from 80 W to 29.7 W package, qualified 96 s zero-loss; measured on the bench: streams buy scheduling headroom not watts (C1-parked threads, lonelier spinners boost harder), relaxed cpu_dma_latency never reaches deep idle (scheduler spreads wakeups, ~2% residency), offlining E-cores saves 1.7 W and is wire-invisible mid-run, the cliff is TX starvation below 1.6 GHz (1.2 GHz sags to ~18 G, zero errors)
This commit is contained in:
+22
-1
@@ -33,7 +33,7 @@ This rule covers the **NIC-counter** buckets and the per-stream receive buckets
|
||||
|
||||
## Measured performance
|
||||
|
||||
The committed X520 path at the default mix: steady 20.00–20.04 Gb/s (line rate both directions, ~1.78 Mpps/dir), zero lost/corrupt/link/internal over a 90 s run, including startup.
|
||||
The committed X520 path at the default config (2 streams, 2.0 GHz pinned clock): steady 20.00–20.04 Gb/s (line rate both directions, ~0.75 Mpps/dir at the jumbo mix), zero lost/corrupt/link/internal over a 96 s run, including startup, at ~30 W CPU package on the bench host. The pre-jumbo mix ran ~1.78 Mpps/dir at the same rate.
|
||||
|
||||
Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not NIC-specific:
|
||||
|
||||
@@ -45,6 +45,25 @@ Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not
|
||||
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810).
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
|
||||
## Power: the busy cores' voltage is the cost, not the idle cores
|
||||
|
||||
Measured on the bench host (i9-13900H, 6P+8E, 20 threads), RAPL package power at line rate both directions:
|
||||
|
||||
| Config | Package | Wire |
|
||||
|---|---|---|
|
||||
| 7 streams, unbounded clock | 80.0 W | clean |
|
||||
| 2 streams, unbounded clock (4.2–4.6 GHz) | 70.7 W | clean |
|
||||
| 2 streams @ 2.4 GHz | 35.0 W | clean |
|
||||
| **2 streams @ 2.0 GHz (shipped)** | **29.7 W** | clean, 43% busy |
|
||||
| 2 streams @ 1.6 GHz | 25.1 W | clean, 49% busy |
|
||||
| 2 streams @ 1.2 GHz | 20.4 W | **fails — wire sags to 17.5–19.8 G, zero errors: TX starvation, the honest failure mode** |
|
||||
| 7 streams @ 1.6 GHz | 29.8 W | line rate but 94% busy and the rate reading wobbles |
|
||||
| idle, no tool | 10.9 W | — |
|
||||
|
||||
- The frequency pin is the lever: core power 56 → 16 W is V² savings on the busy cores. ~13.5 W of uncore (PCIe/memory fabric) is a floor no CPU knob touches.
|
||||
- Stream count buys scheduling headroom, not watts: 7→1 streams unbounded saves only ~13 W, because freed threads park in C1 (the 10 µs cpu_dma_latency cap) while the surviving TX spinners boost harder. At a fixed clock the cut is what makes the low clock safe — 49% busy at 2 streams where 7 needs 94%.
|
||||
- Every TX worker is a full-time spinner (the ENOBUFS spin is the flow control; there is no blocking send under PACKET_QDISC_BYPASS), so TX cost scales with stream count regardless of traffic; RX blocks in recvmmsg and scales with traffic.
|
||||
|
||||
## Dead ends — measured, do not re-attempt without new hardware
|
||||
|
||||
Measured in the E810 era; the binding constraint was total CPU across ~28 goroutines on 20 threads.
|
||||
@@ -53,6 +72,8 @@ Measured in the E810 era; the binding constraint was total CPU across ~28 gorout
|
||||
- **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at TX.
|
||||
- **Batch sizes above 64** — no gain, worse loss.
|
||||
- **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
|
||||
- **Letting idle cores sleep by relaxing cpu_dma_latency** (200 µs admits the 127 µs C2_ACPI) — steady-state deep-idle residency ~2% and power unchanged: the Go scheduler spreads wakeups across every thread, so no core idles long enough for the idle governor to pick a deep state.
|
||||
- **Offlining cores for power** (all 8 E-cores) — 1.7 W. The offline event mid-run is invisible on the wire (no blip at line rate), so parking cores is safe if ever wanted for another reason; it is not a power lever. Note the SETTINGS governor check fails loudly with offline cpus present — it reads every `cpu*/cpufreq/scaling_governor` and an offline cpu returns EBUSY.
|
||||
- **Alien-crosstalk injection from an intertwined noise cable** — bench-measured: two unshielded cables tightly wrapped moved the victim's per-pair SNR margin only 0.2–0.3 dB, far too little to stress even a marginal cable; 10GBASE-T's alien-noise budget absorbs it. The noise pair, its link-cycling, and the panel lane are removed from the tool.
|
||||
|
||||
## rxnfc flow-rule programming — three traps (found on ice; 1 and 2 are kernel-ABI-generic)
|
||||
|
||||
Reference in New Issue
Block a user