cabletest/docs/xdp.md

AF_XDP datapath — record of the working experiment

A complete AF_XDP conversion of both directions ran on the E810-era datapath: hand-rolled zero-copy XSK sockets (no libbpf/libxdp), a hand-assembled per-port XDP program delivering per-frame MAC RX stamps through metadata, and the throughput numbers in measurement.md (64 B TX ~14.5 Mpps/dir vs ~5.4 on AF_PACKET; RX ~13.5 Mpps vs ~4). The code is dropped; this file carries what it proved and what rebuilding it needs. The committed AF_PACKET design needs none of it — no hardware stamps, and line rate at the jumbo mix costs ~0.75 Mpps/dir.

Why it cannot run on the product NIC

ixgbe caps XDP frames at one buffer — 3 KB (2 KB with legacy-rx or 8 KB pages), no multi-buffer support — and refuses jumbo MTU while any program is attached (ixgbe_max_xdp_frame_size / ixgbe_change_mtu). The 9014-byte frame in the locked-in mix excludes AF_XDP on the X520 outright. The transport becomes interesting again only on a NIC with XDP multi-buffer (ice has it) and a reason to displace the X520.

What it would buy, bounded

• TX gains real flow control — the one thing AF_PACKET cannot have: a full TX ring plus XDP_USE_NEED_WAKEUP means senders sleep instead of riding the ENOBUFS spin (under PACKET_QDISC_BYPASS the spin is the flow control; measurement.md). While the wakeup flag is down, the ring drains with zero syscalls.
• Zero-copy removes both memcpys (~5 GB/s of memory traffic at line rate) and the per-frame skb path.
• Bounded by what survives any transport: the ~13.5 W uncore floor and the payload CRC work (the product's job). Against the 29.7 W shipped operating point, realistically ~5–7 W — not worth a datapath rewrite on its own.

Kernel and driver requirements

• Appliance kernel: CONFIG_XDP_SOCKETS=y, plus CONFIG_DEBUG_INFO_BTF=y (with DWARF debug info enabled to feed it) — the stamp kfunc is resolved by ID from /sys/kernel/btf/vmlinux at runtime, so BTF must ship in the image.
• ice will not allocate its XDP TX rings until some program is attached to the interface — XSK TX silently depends on the attach. Attaching bounces the link (prime counters afterward).
• ice refuses channel-count changes while flow rules point into the queues: clear rules first, then set channels, then rings, then rules (the committed check order already clears first).
• Channel count wants pinning (the experiment used 16) so the queue layout is identical every boot instead of inheriting the driver's CPU-count sizing: RX streams on queues 0..N−1 by flow rule, XSK TX sockets bound above them with one spare queue between to catch whatever RSS delivers.

Design facts a rebuild should keep

• Demand zero-copy, never request it: bind with XDP_ZEROCOPY hard. Copy mode is a different tester with different limits; a silent fallback would misreport every number on the panel.
• A zero-copy bind owns the whole queue pair, so even TX-only sockets need a small stocked fill ring: stray receive traffic landing on a TX queue is decoded out of those chunks and passed to the stack by the program — left unstocked it counts as NIC drops and lights the link column for a fault the cable never committed.
• TX umem chunks are prefilled with their pattern for the life of the run (chunk count not a multiple of the pattern count, so the pattern↔size pairing rotates); a send writes only the 20-byte header in place. Completion-ring addresses recycle through a plain free list; producer/consumer words are shadowed so the shared words are crossed once per batch.
• TX full-ring nap: an empty send pass means every frame is in the ring or in flight — the wire is the ceiling — so a ~20 µs sleep steals nothing (the ring holds far more wire time than the nap) and frees the CPU the AF_PACKET spin burns.
• RX chunk budget must out-provision the hardware ring: enough chunks (16384 at ring 8160) to keep the ring fully posted with the fill ring stocked and frames still in a worker's hands — too few quietly shallows the ring the host tuning asked for. The driver returns frames offset by its headroom; recover the chunk base by alignment before refreeing.
• Drops accounting: XDP_STATISTICS (rx_dropped + rx_ring_full) replaces the packet socket's drop statistic; cumulative for the socket's life, so accumulate forward motion only.
• The program is ~34 hand-assembled BPF instruction words — no clang/LLVM in the build: match the stream ethertypes, xdp_adjust_meta(-8), call the driver's bpf_xdp_metadata_rx_timestamp kfunc to put the MAC stamp in the metadata word (zero = no stamp, not an error), redirect into an XSKMAP keyed by rx_queue_index; everything else (probe, stack traffic) passes. Loaded with BPF_F_XDP_HAS_FRAGS (ice demands frags-capable programs while the MTU admits jumbo) and BPF_F_XDP_DEV_BOUND_ONLY — device-binding is what lets the verifier resolve the metadata kfunc to this port's driver. Attached with a bpf_link in DRV mode; map, program and link all die with the process.
• Kfunc IDs are a property of the running kernel build: a minimal BTF walker (12-byte headers plus kind-dependent tails, match KIND_FUNC by name) finds the ID at startup; a kernel without the function fails loudly there.
• The metadata-stamp delivery was E810-only (the kfunc binds to the driver's reader); the design is generic but each driver must implement the kfunc.

Wire-weighted size schedule (transport-independent idea, lived only here)

The experiment replaced the uniform per-frame size rotation with a schedule weighting each size inversely to its wire cost, occurrences placed by virtual time within the cycle: every size gets an equal share of wire time instead of the largest frames dominating it, small frames carry the packet rate, and sizes interleave rather than burst. Orthogonal to the transport, and in tension with the committed mix's role as the locked-in comparable config — recorded as a design option, not a plan.