theater/cabletest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

All three stashes distilled into docs and dropped: xdp.md carries the AF_XDP record (ixgbe's 3 KB single-buffer XDP cap excludes the locked jumbo mix; XSK need_wakeup TX as the spin-free flow control, zero-copy-or-die, the stray-RX fill-ring trap, BTF-resolved stamp kfunc, wire-weighted size schedule as an unadopted option); connectx-5 README gains the PCICONF/ICMD/MCIA gateway constants so a rebuild needs no mstflint re-mine; e810 keeps the read-side transport map (cage-cache, bus=port, bus-5 phantom, 16-bit-addr hazard, latency as wire-truth) and NVM rollback pointers; x710 the >=30 ms EEPROM write pacing; fibergaga the empirical vendor-space finds; fs the inert-I2C-surface map; state.md stash section now empty

Browse Source
This commit is contained in:
flamingcow
2026-08-19 14:42:13 -07:00
parent 4c6de3b618
commit 2d3369af85
10 changed files with 88 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/xdp.md
+36
View File
@@ -0,0 +1,36 @@
# AF_XDP datapath — record of the working experiment
A complete AF_XDP conversion of both directions ran on the E810-era datapath: hand-rolled zero-copy XSK sockets (no libbpf/libxdp), a hand-assembled per-port XDP program delivering per-frame MAC RX stamps through metadata, and the throughput numbers in measurement.md (64 B TX ~14.5 Mpps/dir vs ~5.4 on AF_PACKET; RX ~13.5 Mpps vs ~4). The code is dropped; this file carries what it proved and what rebuilding it needs. The committed AF_PACKET design needs none of it — no hardware stamps, and line rate at the jumbo mix costs ~0.75 Mpps/dir.
## Why it cannot run on the product NIC
**ixgbe caps XDP frames at one buffer — 3 KB (2 KB with legacy-rx or 8 KB pages), no multi-buffer support** — and refuses jumbo MTU while any program is attached (`ixgbe_max_xdp_frame_size` / `ixgbe_change_mtu`). The 9014-byte frame in the locked-in mix excludes AF_XDP on the X520 outright. The transport becomes interesting again only on a NIC with XDP multi-buffer (ice has it) *and* a reason to displace the X520.
## What it would buy, bounded
- **TX gains real flow control** — the one thing AF_PACKET cannot have: a full TX ring plus `XDP_USE_NEED_WAKEUP` means senders sleep instead of riding the ENOBUFS spin (under `PACKET_QDISC_BYPASS` the spin *is* the flow control; measurement.md). While the wakeup flag is down, the ring drains with zero syscalls.
- Zero-copy removes both memcpys (~5 GB/s of memory traffic at line rate) and the per-frame skb path.
- Bounded by what survives any transport: the ~13.5 W uncore floor and the payload CRC work (the product's job). Against the 29.7 W shipped operating point, realistically ~5–7 W — not worth a datapath rewrite on its own.
## Kernel and driver requirements
- Appliance kernel: `CONFIG_XDP_SOCKETS=y`, plus `CONFIG_DEBUG_INFO_BTF=y` (with DWARF debug info enabled to feed it) — the stamp kfunc is resolved by ID from `/sys/kernel/btf/vmlinux` at runtime, so BTF must ship in the image.
- ice will not allocate its XDP TX rings until *some* program is attached to the interface — XSK TX silently depends on the attach. Attaching bounces the link (prime counters afterward).
- ice refuses channel-count changes while flow rules point into the queues: clear rules first, then set channels, then rings, then rules (the committed check order already clears first).
- Channel count wants pinning (the experiment used 16) so the queue layout is identical every boot instead of inheriting the driver's CPU-count sizing: RX streams on queues 0..N−1 by flow rule, XSK TX sockets bound above them with one spare queue between to catch whatever RSS delivers.
## Design facts a rebuild should keep
- **Demand zero-copy, never request it**: bind with `XDP_ZEROCOPY` hard. Copy mode is a different tester with different limits; a silent fallback would misreport every number on the panel.
- **A zero-copy bind owns the whole queue pair**, so even TX-only sockets need a small stocked fill ring: stray receive traffic landing on a TX queue is decoded out of those chunks and passed to the stack by the program — left unstocked it counts as NIC drops and lights the link column for a fault the cable never committed.
- **TX umem chunks are prefilled with their pattern for the life of the run** (chunk count not a multiple of the pattern count, so the pattern↔size pairing rotates); a send writes only the 20-byte header in place. Completion-ring addresses recycle through a plain free list; producer/consumer words are shadowed so the shared words are crossed once per batch.
- **TX full-ring nap**: an empty send pass means every frame is in the ring or in flight — the wire is the ceiling — so a ~20 µs sleep steals nothing (the ring holds far more wire time than the nap) and frees the CPU the AF_PACKET spin burns.
- **RX chunk budget must out-provision the hardware ring**: enough chunks (16384 at ring 8160) to keep the ring fully posted with the fill ring stocked and frames still in a worker's hands — too few quietly shallows the ring the host tuning asked for. The driver returns frames offset by its headroom; recover the chunk base by alignment before refreeing.
- **Drops accounting**: `XDP_STATISTICS` (`rx_dropped + rx_ring_full`) replaces the packet socket's drop statistic; cumulative for the socket's life, so accumulate forward motion only.
- **The program is ~34 hand-assembled BPF instruction words** — no clang/LLVM in the build: match the stream ethertypes, `xdp_adjust_meta(-8)`, call the driver's `bpf_xdp_metadata_rx_timestamp` kfunc to put the MAC stamp in the metadata word (zero = no stamp, not an error), redirect into an XSKMAP keyed by `rx_queue_index`; everything else (probe, stack traffic) passes. Loaded with `BPF_F_XDP_HAS_FRAGS` (ice demands frags-capable programs while the MTU admits jumbo) **and** `BPF_F_XDP_DEV_BOUND_ONLY` — device-binding is what lets the verifier resolve the metadata kfunc to this port's driver. Attached with a bpf_link in DRV mode; map, program and link all die with the process.
- **Kfunc IDs are a property of the running kernel build**: a minimal BTF walker (12-byte headers plus kind-dependent tails, match `KIND_FUNC` by name) finds the ID at startup; a kernel without the function fails loudly there.
- The metadata-stamp delivery was E810-only (the kfunc binds to the driver's reader); the design is generic but each driver must implement the kfunc.
## Wire-weighted size schedule (transport-independent idea, lived only here)
The experiment replaced the uniform per-frame size rotation with a schedule weighting each size inversely to its wire cost, occurrences placed by virtual time within the cycle: every size gets an equal share of *wire time* instead of the largest frames dominating it, small frames carry the packet rate, and sizes interleave rather than burst. Orthogonal to the transport, and in tension with the committed mix's role as the locked-in comparable config — recorded as a design option, not a plan.