All three stashes distilled into docs and dropped: xdp.md carries the AF_XDP record (ixgbe's 3 KB single-buffer XDP cap excludes the locked jumbo mix; XSK need_wakeup TX as the spin-free flow control, zero-copy-or-die, the stray-RX fill-ring trap, BTF-resolved stamp kfunc, wire-weighted size schedule as an unadopted option); connectx-5 README gains the PCICONF/ICMD/MCIA gateway constants so a rebuild needs no mstflint re-mine; e810 keeps the read-side transport map (cage-cache, bus=port, bus-5 phantom, 16-bit-addr hazard, latency as wire-truth) and NVM rollback pointers; x710 the >=30 ms EEPROM write pacing; fibergaga the empirical vendor-space finds; fs the inert-I2C-surface map; state.md stash section now empty
This commit is contained in:
+2
-1
@@ -5,9 +5,10 @@ Project knowledge that moves and versions with the repo.
|
||||
General:
|
||||
|
||||
- [goals.md](goals.md) — what cabletest is for and what it must output.
|
||||
- [state.md](state.md) — where things are now: committed vs stashed, the box and what's in it, out-of-repo artifacts, every-boot steps. **All current-placement state lives here and only here.**
|
||||
- [state.md](state.md) — where things are now: the box and what's in it, out-of-repo artifacts, every-boot steps. **All current-placement state lives here and only here.**
|
||||
- [hardware.md](hardware.md) — host rules: interface conventions, tuning, how to run.
|
||||
- [measurement.md](measurement.md) — how to measure correctly; performance results and dead ends.
|
||||
- [xdp.md](xdp.md) — the AF_XDP datapath experiment: what it proved, what rebuilding it needs, why it can't run on the X520.
|
||||
- [open-questions.md](open-questions.md) — the genuinely open problems, as pointers into the docs that hold their detail.
|
||||
|
||||
Per-device (each directory: device doc + its datasheets):
|
||||
|
||||
+2
-2
@@ -40,10 +40,10 @@ Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not
|
||||
| Config | Result |
|
||||
|---|---|
|
||||
| 64 B only | pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s |
|
||||
| AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
|
||||
| AF_XDP experiment ([xdp.md](xdp.md)) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
|
||||
|
||||
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810).
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). Its code is dropped; the design record and its constraints live in [xdp.md](xdp.md) — its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
|
||||
## Power: the busy cores' voltage is the cost, not the idle cores
|
||||
|
||||
|
||||
@@ -15,6 +15,22 @@ The documented oracle — and only that: the part's 30 m reach limit makes it un
|
||||
| 1E.C880–C883 | Reflection impedance/magnitude |
|
||||
| PMA 1.133–1.140, PCS 3.32/3.33 | Standard set ([../README.md](../README.md)), proven live |
|
||||
|
||||
## Vendor space found empirically (beyond the documented set)
|
||||
|
||||
From a 544-register sweep of the vendor regions plus snapshot-diffing to separate live counters from static state (staleness ruled out: 20/20 interleaved known-value reads correct). The method transfers to any undocumented map — sweep for nonzero, then diff snapshots across time and across link events:
|
||||
|
||||
| Register | Behaviour |
|
||||
|---|---|
|
||||
| 1E.C886 | Link uptime, monotonic +1 per second |
|
||||
| 1E.C869 | ~452 Hz free-running clear-on-read tick — **not errors**: returned an identical count on consecutive fixed-length samples, deterministic where errors would be Poisson |
|
||||
| 1E.C860–C868 | MDIO write shadow log — echoes devad/register/value of the last write |
|
||||
| 1E.C820/C821 | Die temperature (~97 °C on bench) and its valid bit |
|
||||
| 3.C820, 3.E810, 3.E812 | Spike during link training, quiet on a settled link |
|
||||
|
||||
Live behaviour of the documented 3.E820 (post-FEC uncorrected frames): reads 0 on a settled link, ~1852 within 0.7 s of a retrain — LDPC coding gain keeps it at exactly zero anywhere short of the margin cliff, so it moves during training and near failure, nothing in between.
|
||||
|
||||
Two negative results worth keeping: short-reach mode (PMA 1.131 bit 0) lowers *our* launch power, which degrades the far end's receive, not our own SNR — useless for self-degradation; and per-pair margins are retrain-dependent on this module too (the same cable trained to 66/45/56/48 and 52/36/45/46 in consecutive sessions) — compare within one training session only.
|
||||
|
||||
## Pre-FEC correlation run (verification, not classification)
|
||||
|
||||
`3.E820` is post-FEC; the pre-FEC counters are the corrected-by-iteration histogram (3.E840–E84E). The correlation run verifies the expected ordering; this module is its instrument. Design points:
|
||||
|
||||
@@ -16,6 +16,12 @@ PHY at I2C 8-bit **0xAC write / 0xAD read**.
|
||||
- Single-byte reads see the bridge as inert (returns 0) — the DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead. Treat 0x0000 reads as retry-with-longer-delay: 0 is also the bridge's not-ready signature.
|
||||
- Recorded for a future multi-byte firmware transport (mlx5 MCIA shape): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`.
|
||||
|
||||
## The rest of the I2C surface (probed exhaustively over a real write path)
|
||||
|
||||
- **Undeclared live DOM**: A0h byte 92 (0x5C) claims no DDM, but A2h 0x60+ serves real temperature/Vcc/bias/power (~79 °C, 3.25 V observed on bench). Usable read-only.
|
||||
- **A2h 0x00–0x7A takes writes and is plain memory, not a mailbox**: every non-volatile byte holds any value, tagged block writes cause zero side effects anywhere in A2h, and sliding the command-handler STATUS signature (1E.4037, idle 0x0004) across every consecutive 3-byte window in both byte orders never surfaced it. 0x7B–0xFF does not take writes (0x7B–0x7E RollBall password area, write-only by convention; 0x7F page register pinned 0xFF; upper half reads zeros).
|
||||
- The 7-bit 0x53 window ACKs every read and write and always serves zeros — inert, unlike 0x56, whose zeros were the unframed bridge.
|
||||
|
||||
## Host interface (XFI) bring-up gating
|
||||
|
||||
**The module never enables its XFI TX toward the host until it receives valid host signal** — observed live on the ConnectX-5, where it produces a mutual-wait deadlock ([../../nics/connectx-5/README.md](../../nics/connectx-5/README.md)):
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Status: retired — the X520 smoothed-bucket plan removes the need for this card
|
||||
|
||||
The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is **parked in a git stash** ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved.
|
||||
The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR with the headline as worst pair across both ends, BCM+length dropped) ran on hardware — line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules — and its code is dropped; this file is the complete record (including the gateway constants below), with mstflint as the reimplementation reference. Everything below is what was proven and what was never resolved.
|
||||
|
||||
**Known:**
|
||||
|
||||
@@ -72,6 +72,15 @@ Two things that are not optional and one that is not needed:
|
||||
|
||||
There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.
|
||||
|
||||
### PCICONF gateway constants (verified live; rebuildable without re-mining mstflint)
|
||||
|
||||
- Vendor-specific capability ID **0x09**; offsets from the cap base: **ctrl 0x4** (low 16 bits select the address space; a readback with bits 31:29 zero means the space is unsupported), **counter 0x8**, **semaphore 0xC**, **addr 0x10**, **data 0x14**.
|
||||
- The semaphore is a ticket lock: free at 0; taken by writing the counter register's current value and reading it back unchanged. Address spaces: **CR 0x2**, **ICMD 0x3**.
|
||||
- One dword op: for a write, data register then addr with bit 31 set; for a read, addr with bit 31 clear, then read data. The flag bit toggles inside the device's own access (clear = write done, set = read done) — always settled before the PCI round-trip that reads it back, so an unsettled flag is a fault, not a wait.
|
||||
- ICMD space: **ctrl at 0x0** (opcode bits 31:16, status bits 15:8, go bit 0 — set to fire, cleared by firmware on completion, observed < 5 ms), **mailbox size at 0x1000**, **mailbox at 0x100000**. CR-space **0xB5E04 bit 31** = firmware static config not done; check before any command.
|
||||
- Command **FLASH_REG_ACCESS 0x9001**: operation TLV (type 1, 4 dwords — register ID at bytes 4–5, method byte 6 with GET=1/SET=2, class byte 7 = 1 for register access; completion status returns in byte 2 & 0x7F) + reg TLV (type 3, length in dwords) + register payload, all big-endian dwords in the mailbox.
|
||||
- MCIA register ID **0x9014**, 16-byte header + 48 data bytes: module at byte 1, per-access status at byte 3, 8-bit I2C device address at byte 4, device offset at byte 7, size at bytes 10–11, data from byte 16.
|
||||
|
||||
## The FS cannot link on the CX-5 — host-interface deadlock
|
||||
|
||||
The FS's XFI never comes up against this firmware; **diagnostics to the FS work regardless** (all of the above ran with the link down). The mechanism, each side observed directly:
|
||||
@@ -97,7 +106,7 @@ Paths forward if this card ever returns, in escalation order:
|
||||
- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
|
||||
- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
|
||||
|
||||
## Host-setup deltas from the ice/ixgbe path (the stashed implementation's `system.go`/`counters.go`)
|
||||
## Host-setup deltas from the ice/ixgbe path
|
||||
|
||||
Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
|
||||
|
||||
|
||||
@@ -17,6 +17,17 @@ The richest per-frame RX error breakdown of the NICs used — `illegal_bytes` an
|
||||
- Every write returns aq 1 (EPERM) — tested identically against EEPROM 0x50 and BCM 0x56, so the block is write-vs-read, not address-scoped. Intel policy, survives NVM update; NVM 5.01 hardened the 3.10 silent-drop into an explicit EPERM.
|
||||
- Dead for anything needing writes: RollBall unlock needs a write; BCM SMI reads *start* with a write.
|
||||
|
||||
Read-side transport details (the patched ice exists for read-side work, so these stay load-bearing):
|
||||
|
||||
- CTX_OVERRIDE (ntc 0x50, handle 0) reaches the real wire; `index` is the raw I2C bus number and equals the port number (verified by serial), so one PF0 debugfs file reaches both cages. NACK reads report `err -5 aq 1`.
|
||||
- Cage-node addressing (ntc 0x26/0x46, handle from the netlist) returns a **firmware cache** of the module EEPROM, not the wire: unknown regions read zeros, and the cache serves stale bytes after the wire changes. Don't use it.
|
||||
- Override bus 5 ACKs every address with zeros — a firmware phantom.
|
||||
- 16-bit-address mode (params bit4) turns the offset phase into a 2-byte write an SFF EEPROM can latch as data — never use it on 0x50/0x51.
|
||||
- No offset-less read mode exists, so the plain mdio-i2c clause-45 read phase (bcm84881-style at 0x56) cannot be formed. RollBall fits fully (≤4 B native multi-byte writes).
|
||||
- Latency is the wire-truth judge on this transport: on 3.10 firmware, silently-dropped 1-byte writes completed in ~63 µs while real wire traffic ran ~103 µs/byte (reads) and multi-ms ack-polling (writes), and `ok` returns even for writes at an empty address — protocol-level readback is the only delivery confirmation.
|
||||
|
||||
Firmware on the card: NVM 5.01 (release 31.2.2, EETRACK 0x80021FA2). The update package and logs live in `~/work/e810nvm/`; `nvmupdate64e` needs `-AllowMissingCert` on this platform (harmless — nothing boots from the NIC), writes no file backup, and rollback lives in the card's inactive flash bank (`nvmupdate64e -r -m <MAC>`).
|
||||
|
||||
## The bit-bang-below-the-AQ idea — retired with the timestamp requirement
|
||||
|
||||
The write block is at the **Admin Queue** (firmware) level — the `0x06E2/E3` topo-I2C commands are refused with EPERM. That is not the same as the module's SDA/SCL being unreachable: the 82599 reaches the module I2C **without firmware** by toggling SDA/SCL as GPIO bits in `I2CCTL` (the `~/work/ixgbe-sff/` patch bit-bangs arbitrary START…STOP from those primitives), and whether the E810 exposes an equivalent register-level / SDP-GPIO path below the AQ policy was never tested. The idea existed to marry all-packet stamps to raw I2C in one card; the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves the stamps requirement, so the X520 covers everything and this investigation is not needed. It stays recorded as the E810's one open door if per-frame arrival times are ever wanted again — the investigation would start at the datasheet's SDP/GPIO and low-level I2C controller registers, asking whether the cage's SDA/SCL are host-addressable.
|
||||
|
||||
@@ -24,7 +24,7 @@ The per-frame hardware RX timestamp requirement existed for exactly one consumer
|
||||
What this design carries in practice:
|
||||
|
||||
- **The FS links and runs full diagnostics here** (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative.
|
||||
- **Built on the committed X520/BCM/ECD implementation**, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
|
||||
- **Built on the committed X520/BCM/ECD implementation.** The CX-5 implementation is dropped; its record, including the MCIA gateway constants, is [../connectx-5/](../connectx-5/README.md).
|
||||
|
||||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
- Opcode 0x0629 get / 0x0628 set PHY register. param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value.
|
||||
- Result comes back in the dmesg "AQ desc WB" line, 9th field.
|
||||
- Proven on the Fibergaga, link up: per-pair SNR (PMA 1.133–1.136), PCS 3.32/3.33, the 3.E820 counter ([../../modules/fibergaga/](../../modules/fibergaga/README.md)).
|
||||
- **Write pacing**: single-byte writes issued ~7 ms apart can land inside the EEPROM's internal write cycle and be silently dropped — no AQ error, stale readback. Space writes ≥30 ms when framing a protocol; a client whose per-op cost is ≥11 ms (page-select + poll sleeps) never trips it, which is why the RollBall path did not.
|
||||
|
||||
## Timestamping
|
||||
|
||||
|
||||
+2
-6
@@ -6,11 +6,7 @@
|
||||
|
||||
## Stashes
|
||||
|
||||
Indices shift as stashes are pushed/popped — match by message, not number.
|
||||
|
||||
- **"CX-5 Wiitek: MCIA transport…"** (currently `stash@{0}`) — the whole Wiitek-pair rewrite for the CX-5, kept as a parts bin: `phy_mcia.go` (VSC/ICMD MCIA transport), transport-interface `phy.go` (the same RollBall logic over either patched-ixgbe `sff_i2c` or MCIA), dual-end per-pair SNR from both modules (headline: worst pair across both ends), mlx5 ntuple steering + PHY RMON counters, test interfaces at MTU 9000 with a 9018-byte jumbo in the size mix; BCM/ECD/length dropped. Technical record: nics/connectx-5/. Carries live debug instrumentation and `rbCmdPoll` lowered to 2 ms from the mailbox-latency investigation — **not shippable as-is**, clean it before committing — plus the pre-existing uncommitted `ui.go`/`ui_test.go` edits that were entangled in the same files.
|
||||
- **"AF_XDP both directions…"** (currently `stash@{1}`) — AF_XDP conversion (hand-rolled XSK + XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
|
||||
- **"phydiag campaign…"** (currently `stash@{2}`) — python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
|
||||
None. Every parked experiment is distilled into docs and its code dropped: the AF_XDP conversion in xdp.md, the CX-5 Wiitek-pair implementation in nics/connectx-5/ (including the MCIA gateway constants), the phydiag campaign across nics/e810/, nics/x710/, modules/fs/ and modules/fibergaga/. Remaining phydiag working copies: `~/work/phydiag-work/` (below).
|
||||
|
||||
## The box
|
||||
|
||||
@@ -30,7 +26,7 @@ Indices shift as stashes are pushed/popped — match by message, not number.
|
||||
| Item | Status | Notes |
|
||||
|---|---|---|
|
||||
| X520-DA2 | **Installed** in the single PCIe slot — **the product NIC** (nics/x520/) | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven; the smoothed-bucket rate design removes the need for hardware RX stamps |
|
||||
| ConnectX-5 | Out of the box — retired | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
|
||||
| ConnectX-5 | Out of the box — retired | Ran the two-Wiitek pair at near line rate with dual-end SNR (MCIA transport, recorded with its gateway constants in nics/connectx-5/), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
|
||||
| FS SFP-10G-T-100 ×2 | One in the X520 test pair, one on the shelf | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration |
|
||||
| Replacement Wiiteks | One in the X520 test pair, two seated in the unused X710, rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first. IEEE-standard registers only (modules/README.md safe set); VCT templates are single-shot candidates on a sacrificial unit only |
|
||||
| Fibergaga SFP-10G-T-30M | On the shelf | Aquantia, RollBall, the documented oracle; the alternate length path if the pair mix ever changes |
|
||||
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
# AF_XDP datapath — record of the working experiment
|
||||
|
||||
A complete AF_XDP conversion of both directions ran on the E810-era datapath: hand-rolled zero-copy XSK sockets (no libbpf/libxdp), a hand-assembled per-port XDP program delivering per-frame MAC RX stamps through metadata, and the throughput numbers in measurement.md (64 B TX ~14.5 Mpps/dir vs ~5.4 on AF_PACKET; RX ~13.5 Mpps vs ~4). The code is dropped; this file carries what it proved and what rebuilding it needs. The committed AF_PACKET design needs none of it — no hardware stamps, and line rate at the jumbo mix costs ~0.75 Mpps/dir.
|
||||
|
||||
## Why it cannot run on the product NIC
|
||||
|
||||
**ixgbe caps XDP frames at one buffer — 3 KB (2 KB with legacy-rx or 8 KB pages), no multi-buffer support** — and refuses jumbo MTU while any program is attached (`ixgbe_max_xdp_frame_size` / `ixgbe_change_mtu`). The 9014-byte frame in the locked-in mix excludes AF_XDP on the X520 outright. The transport becomes interesting again only on a NIC with XDP multi-buffer (ice has it) *and* a reason to displace the X520.
|
||||
|
||||
## What it would buy, bounded
|
||||
|
||||
- **TX gains real flow control** — the one thing AF_PACKET cannot have: a full TX ring plus `XDP_USE_NEED_WAKEUP` means senders sleep instead of riding the ENOBUFS spin (under `PACKET_QDISC_BYPASS` the spin *is* the flow control; measurement.md). While the wakeup flag is down, the ring drains with zero syscalls.
|
||||
- Zero-copy removes both memcpys (~5 GB/s of memory traffic at line rate) and the per-frame skb path.
|
||||
- Bounded by what survives any transport: the ~13.5 W uncore floor and the payload CRC work (the product's job). Against the 29.7 W shipped operating point, realistically ~5–7 W — not worth a datapath rewrite on its own.
|
||||
|
||||
## Kernel and driver requirements
|
||||
|
||||
- Appliance kernel: `CONFIG_XDP_SOCKETS=y`, plus `CONFIG_DEBUG_INFO_BTF=y` (with DWARF debug info enabled to feed it) — the stamp kfunc is resolved by ID from `/sys/kernel/btf/vmlinux` at runtime, so BTF must ship in the image.
|
||||
- ice will not allocate its XDP TX rings until *some* program is attached to the interface — XSK TX silently depends on the attach. Attaching bounces the link (prime counters afterward).
|
||||
- ice refuses channel-count changes while flow rules point into the queues: clear rules first, then set channels, then rings, then rules (the committed check order already clears first).
|
||||
- Channel count wants pinning (the experiment used 16) so the queue layout is identical every boot instead of inheriting the driver's CPU-count sizing: RX streams on queues 0..N−1 by flow rule, XSK TX sockets bound above them with one spare queue between to catch whatever RSS delivers.
|
||||
|
||||
## Design facts a rebuild should keep
|
||||
|
||||
- **Demand zero-copy, never request it**: bind with `XDP_ZEROCOPY` hard. Copy mode is a different tester with different limits; a silent fallback would misreport every number on the panel.
|
||||
- **A zero-copy bind owns the whole queue pair**, so even TX-only sockets need a small stocked fill ring: stray receive traffic landing on a TX queue is decoded out of those chunks and passed to the stack by the program — left unstocked it counts as NIC drops and lights the link column for a fault the cable never committed.
|
||||
- **TX umem chunks are prefilled with their pattern for the life of the run** (chunk count not a multiple of the pattern count, so the pattern↔size pairing rotates); a send writes only the 20-byte header in place. Completion-ring addresses recycle through a plain free list; producer/consumer words are shadowed so the shared words are crossed once per batch.
|
||||
- **TX full-ring nap**: an empty send pass means every frame is in the ring or in flight — the wire is the ceiling — so a ~20 µs sleep steals nothing (the ring holds far more wire time than the nap) and frees the CPU the AF_PACKET spin burns.
|
||||
- **RX chunk budget must out-provision the hardware ring**: enough chunks (16384 at ring 8160) to keep the ring fully posted with the fill ring stocked and frames still in a worker's hands — too few quietly shallows the ring the host tuning asked for. The driver returns frames offset by its headroom; recover the chunk base by alignment before refreeing.
|
||||
- **Drops accounting**: `XDP_STATISTICS` (`rx_dropped + rx_ring_full`) replaces the packet socket's drop statistic; cumulative for the socket's life, so accumulate forward motion only.
|
||||
- **The program is ~34 hand-assembled BPF instruction words** — no clang/LLVM in the build: match the stream ethertypes, `xdp_adjust_meta(-8)`, call the driver's `bpf_xdp_metadata_rx_timestamp` kfunc to put the MAC stamp in the metadata word (zero = no stamp, not an error), redirect into an XSKMAP keyed by `rx_queue_index`; everything else (probe, stack traffic) passes. Loaded with `BPF_F_XDP_HAS_FRAGS` (ice demands frags-capable programs while the MTU admits jumbo) **and** `BPF_F_XDP_DEV_BOUND_ONLY` — device-binding is what lets the verifier resolve the metadata kfunc to this port's driver. Attached with a bpf_link in DRV mode; map, program and link all die with the process.
|
||||
- **Kfunc IDs are a property of the running kernel build**: a minimal BTF walker (12-byte headers plus kind-dependent tails, match `KIND_FUNC` by name) finds the ID at startup; a kernel without the function fails loudly there.
|
||||
- The metadata-stamp delivery was E810-only (the kfunc binds to the driver's reader); the design is generic but each driver must implement the kfunc.
|
||||
|
||||
## Wire-weighted size schedule (transport-independent idea, lived only here)
|
||||
|
||||
The experiment replaced the uniform per-frame size rotation with a schedule weighting each size inversely to its wire cost, occurrences placed by virtual time within the cycle: every size gets an equal share of *wire time* instead of the largest frames dominating it, small frames carry the packet rate, and sizes interleave rather than burst. Orthogonal to the transport, and in tension with the committed mix's role as the locked-in comparable config — recorded as a design option, not a plan.
|
||||
Reference in New Issue
Block a user