From 4c6de3b618545b330bb529750e4f004cdaad986c Mon Sep 17 00:00:00 2001 From: flamingcow Date: Wed, 19 Aug 2026 14:11:19 -0700 Subject: [PATCH] Power: 2 streams and a 2.0 GHz clock pinned by a new checkMaxFreq SETTINGS check (performance governor + scaling_max_freq = fixed clock) take line rate from 80 W to 29.7 W package, qualified 96 s zero-loss; measured on the bench: streams buy scheduling headroom not watts (C1-parked threads, lonelier spinners boost harder), relaxed cpu_dma_latency never reaches deep idle (scheduler spreads wakeups, ~2% residency), offlining E-cores saves 1.7 W and is wire-invisible mid-run, the cliff is TX starvation below 1.6 GHz (1.2 GHz sags to ~18 G, zero errors) --- docs/hardware.md | 9 +++++---- docs/measurement.md | 23 +++++++++++++++++++++- main.go | 2 +- system.go | 48 ++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 75 insertions(+), 7 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index 1195b97..b10ff75 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -12,12 +12,13 @@ Module identities, EEPROM honesty (and why the X520 needs `allow_unsupported_sfp ## Host tuning (resets every reboot; re-apply before trusting results) -Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults. +Required for zero baseline loss (without them baseline loss is 0.02–0.3% and masks cable faults) and for the power budget. 1. **CPU governor → `performance`** on all cores. Biggest single contributor: under powersave, cores idle at 400 MHz, ramp too slowly, and drop frames at startup. -2. **Disable adaptive coalescing**: `ethtool -C adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC. -3. **Rings deep**, raised toward the max via `ethtool -G` (the tool asks RX 8160 / TX 4096; ixgbe takes it as-is). Effect was not isolated in testing; don't claim it as a proven win. -4. **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp. +2. **CPU max frequency → 2.0 GHz** (`scaling_max_freq`, all cores). Under the performance governor this is a pinned clock, not a ceiling, and it is the power budget: ~30 W package at line rate against ~80 W unbounded (measurement.md, "Power"). The value is host-silicon-bound — 1.6 GHz still holds line rate with thinner margin, 1.2 GHz cannot fill the wire — so re-derive it on new host silicon and re-qualify zero-loss. +3. **Disable adaptive coalescing**: `ethtool -C adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC. +4. **Rings deep**, raised toward the max via `ethtool -G` (the tool asks RX 8160 / TX 4096; ixgbe takes it as-is). Effect was not isolated in testing; don't claim it as a proven win. +5. **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp. Verify a clean run shows `tx frames == rx frames` exactly before a run counts. diff --git a/docs/measurement.md b/docs/measurement.md index a9b2aff..eb3bde8 100644 --- a/docs/measurement.md +++ b/docs/measurement.md @@ -33,7 +33,7 @@ This rule covers the **NIC-counter** buckets and the per-stream receive buckets ## Measured performance -The committed X520 path at the default mix: steady 20.00–20.04 Gb/s (line rate both directions, ~1.78 Mpps/dir), zero lost/corrupt/link/internal over a 90 s run, including startup. +The committed X520 path at the default config (2 streams, 2.0 GHz pinned clock): steady 20.00–20.04 Gb/s (line rate both directions, ~0.75 Mpps/dir at the jumbo mix), zero lost/corrupt/link/internal over a 96 s run, including startup, at ~30 W CPU package on the bench host. The pre-jumbo mix ran ~1.78 Mpps/dir at the same rate. Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not NIC-specific: @@ -45,6 +45,25 @@ Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not - RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810). - The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps. +## Power: the busy cores' voltage is the cost, not the idle cores + +Measured on the bench host (i9-13900H, 6P+8E, 20 threads), RAPL package power at line rate both directions: + +| Config | Package | Wire | +|---|---|---| +| 7 streams, unbounded clock | 80.0 W | clean | +| 2 streams, unbounded clock (4.2–4.6 GHz) | 70.7 W | clean | +| 2 streams @ 2.4 GHz | 35.0 W | clean | +| **2 streams @ 2.0 GHz (shipped)** | **29.7 W** | clean, 43% busy | +| 2 streams @ 1.6 GHz | 25.1 W | clean, 49% busy | +| 2 streams @ 1.2 GHz | 20.4 W | **fails — wire sags to 17.5–19.8 G, zero errors: TX starvation, the honest failure mode** | +| 7 streams @ 1.6 GHz | 29.8 W | line rate but 94% busy and the rate reading wobbles | +| idle, no tool | 10.9 W | — | + +- The frequency pin is the lever: core power 56 → 16 W is V² savings on the busy cores. ~13.5 W of uncore (PCIe/memory fabric) is a floor no CPU knob touches. +- Stream count buys scheduling headroom, not watts: 7→1 streams unbounded saves only ~13 W, because freed threads park in C1 (the 10 µs cpu_dma_latency cap) while the surviving TX spinners boost harder. At a fixed clock the cut is what makes the low clock safe — 49% busy at 2 streams where 7 needs 94%. +- Every TX worker is a full-time spinner (the ENOBUFS spin is the flow control; there is no blocking send under PACKET_QDISC_BYPASS), so TX cost scales with stream count regardless of traffic; RX blocks in recvmmsg and scales with traffic. + ## Dead ends — measured, do not re-attempt without new hardware Measured in the E810 era; the binding constraint was total CPU across ~28 goroutines on 20 threads. @@ -53,6 +72,8 @@ Measured in the E810 era; the binding constraint was total CPU across ~28 gorout - **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at TX. - **Batch sizes above 64** — no gain, worse loss. - **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable). +- **Letting idle cores sleep by relaxing cpu_dma_latency** (200 µs admits the 127 µs C2_ACPI) — steady-state deep-idle residency ~2% and power unchanged: the Go scheduler spreads wakeups across every thread, so no core idles long enough for the idle governor to pick a deep state. +- **Offlining cores for power** (all 8 E-cores) — 1.7 W. The offline event mid-run is invisible on the wire (no blip at line rate), so parking cores is safe if ever wanted for another reason; it is not a power lever. Note the SETTINGS governor check fails loudly with offline cpus present — it reads every `cpu*/cpufreq/scaling_governor` and an offline cpu returns EBUSY. - **Alien-crosstalk injection from an intertwined noise cable** — bench-measured: two unshielded cables tightly wrapped moved the victim's per-pair SNR margin only 0.2–0.3 dB, far too little to stress even a marginal cable; 10GBASE-T's alien-noise budget absorbs it. The noise pair, its link-cycling, and the panel lane are removed from the tool. ## rxnfc flow-rule programming — three traps (found on ice; 1 and 2 are kernel-ABI-generic) diff --git a/main.go b/main.go index 06331ff..ef2df76 100644 --- a/main.go +++ b/main.go @@ -105,7 +105,7 @@ func totalRow(elapsed time.Duration, v view, target float64, phy phyDisplay) []s } const ( - numStreams = 7 + numStreams = 2 batchSize = 64 testDriver = "ixgbe" diff --git a/system.go b/system.go index 8be22c2..df46544 100644 --- a/system.go +++ b/system.go @@ -312,6 +312,7 @@ type ethtoolCoalesce struct { const ( wantGovernor = "performance" + wantMaxFreqKHz = "2000000" wantCoalesceUsecs = 25 wantRxRing = 8160 wantTxRing = 4096 @@ -562,6 +563,50 @@ func checkGovernor(want string) checkResult { return res } +// Under the performance governor a max-frequency bound is a fixed clock, not a +// ceiling the cores rove beneath, so the line-rate margin it was qualified with +// is the margin every run gets. +func checkMaxFreq(wantKHz string) checkResult { + res := checkResult{item: "cpu max frequency"} + paths, err := filepath.Glob("/sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq") + if err != nil || len(paths) == 0 { + res.err = fmt.Errorf("no cpufreq max frequencies found") + return res + } + var wrong []string + counts := map[string]int{} + for _, p := range paths { + b, err := os.ReadFile(p) + if err != nil { + res.err = err + return res + } + got := strings.TrimSpace(string(b)) + counts[got]++ + if got != wantKHz { + wrong = append(wrong, p) + } + } + if len(wrong) == 0 { + res.state = fmt.Sprintf("%s kHz on all %d cpus", wantKHz, len(paths)) + return res + } + var found []string + for k, v := range counts { + found = append(found, fmt.Sprintf("%s:%d", k, v)) + } + for _, p := range wrong { + if err := os.WriteFile(p, []byte(wantKHz), 0o644); err != nil { + res.err = err + res.state = fmt.Sprintf("could not set %s", p) + return res + } + } + res.fixed = true + res.state = fmt.Sprintf("was %s, now %s kHz on all %d cpus", strings.Join(found, " "), wantKHz, len(paths)) + return res +} + // Bounds how far into idle any cpu may sink, so a worker or interrupt never // waits behind a deep state's exit. Ten microseconds admits C1, whose exit is // noise under the interrupt coalescing already in the path, and blocks the @@ -690,7 +735,8 @@ func withIoctlSocket(fn func(fd int) []checkResult) []checkResult { // under test, so with no cable there is never going to be one. func configureSystem(ifnames []string, ethertypes []uint16) []checkResult { return withIoctlSocket(func(fd int) []checkResult { - out := []checkResult{checkGovernor(wantGovernor), checkIdleLatency(wantIdleLatency)} + out := []checkResult{checkGovernor(wantGovernor), checkMaxFreq(wantMaxFreqKHz), + checkIdleLatency(wantIdleLatency)} for _, ifname := range ifnames { out = append(out, checkLinkUp(fd, ifname)) out = append(out, checkMTU(fd, ifname, wantMTU))