Docs cleanup: stale-claim conflicts fixed (e810 stamps/probe.go, CX-5 candidate title, committed-path perf table now the X520 20G result with E810-era rows marked historical, ice-pair ring note), irdma/ice notes moved to e810, state.md halved (done-checklist, RX-steering and standing-goals duplication out; ECD suppression fact into modules/fs), changelog framing flattened to present tense
This commit is contained in:
+9
-6
@@ -31,27 +31,30 @@ This rule covers the **NIC-counter** buckets and the per-stream receive buckets
|
||||
- Small-frame runs are CPU/host-bound, not cable-bound; cable conclusions drawn from them are false.
|
||||
- Keep to the default config (hardware.md) and judge from steady state — the first ~5 s is settling transient, not residual error.
|
||||
|
||||
## Measured performance (AF_PACKET committed path, ice/E810 era)
|
||||
## Measured performance
|
||||
|
||||
The committed X520 path at the default mix: steady 20.00–20.04 Gb/s (line rate both directions, ~1.78 Mpps/dir), zero lost/corrupt/link/internal over a 90 s run, including startup.
|
||||
|
||||
Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not NIC-specific:
|
||||
|
||||
| Config | Result |
|
||||
|---|---|
|
||||
| Full size mix, 7 flow-director streams | Line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup |
|
||||
| 64 B only | pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s |
|
||||
| AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
|
||||
|
||||
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts.
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path no longer uses hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810).
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
|
||||
## Dead ends — measured, do not re-attempt without new hardware
|
||||
|
||||
The binding constraint was total CPU across ~28 goroutines on 20 threads.
|
||||
Measured in the E810 era; the binding constraint was total CPU across ~28 goroutines on 20 threads.
|
||||
|
||||
- **Splitting TX senders from RX streams** — raises TX but collapses RX (RX scales with queue count, capped at 7 on that NIC).
|
||||
- **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at TX.
|
||||
- **Batch sizes above 64** — no gain, worse loss.
|
||||
- **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
|
||||
|
||||
## rxnfc / Flow Director programming (ice) — three traps
|
||||
## rxnfc flow-rule programming — three traps (found on ice; 1 and 2 are kernel-ABI-generic)
|
||||
|
||||
1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement — trusting its display gives an inverted rule that silently matches nothing.
|
||||
2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192), on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
|
||||
|
||||
Reference in New Issue
Block a user