Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables

docs/measurement.md

@@ -3,27 +3,50 @@
 Hard-won rules about measuring correctly. Violating these produces numbers that look plausible and are wrong.
 
 ## Rate buckets: stamp at read time, never at ticker-fire time
-`direction.capture()` stamps each bucket with `time.Now()` taken immediately after reading the counters — never a timestamp passed in from the sampler ticker. A rate is Δcounters/Δt; the sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator, and because E[1/x] > 1/E[x] the error biases the rate **upward**, it doesn't cancel. Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high; read-time stamping read within 2%. A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it. Never share one timestamp across directions "so buckets share an instant" — nothing needs it and it reintroduces the skew.
+
+- `direction.capture()` stamps each bucket with `time.Now()` taken immediately after reading the counters — never a timestamp passed in from the sampler ticker.
+- Why: a rate is Δcounters/Δt. The sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator — and because E[1/x] > 1/E[x], the error biases the rate **upward**; it doesn't cancel.
+- Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high. Read-time stamping read within 2%.
+- A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it.
+- Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew.
+
+This rule covers the **NIC-counter** buckets. The per-stream application buckets are a separate system stamped by per-packet MAC RX timestamps (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check); read-time stamping is not a substitute for those — software stamping was tried and cannot reach the needed precision. NICs without all-packet RX timestamping (X710, 82599) fail the host check; see open-questions.md §2.
 
 ## NIC counters are not monotonic
-`/sys/class/net/*/statistics/*` run from boot, not process start, and reset to zero on driver stats resets. In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run, and a backward step underflows `now - base` to ~2^64. Accumulate only forward motion into a process-local total, then take every baseline from that total — never from a raw reading. Establish all baselines in one place before traffic starts. Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug.
+
+- `/sys/class/net/*/statistics/*` run from boot, not process start, and reset to zero on driver stats resets.
+- In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run; a backward step underflows `now - base` to ~2^64.
+- Rule: accumulate only forward motion into a process-local total, then take every baseline from that total — never from a raw reading. Establish all baselines in one place before traffic starts.
+- Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug.
 
 ## Only zero-baseline-loss runs count; small frames measure the host
-Any nonzero baseline loss masks real cable faults, so a run counts only when `tx frames == rx frames` exactly with a clean cable. Small-frame runs are CPU/host-bound, not cable-bound, so cable conclusions drawn from them are false. Keep to the default config (see hardware.md) and judge from steady state — the first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) that can read far below line rate and is not residual error.
+
+- A run counts only when `tx frames == rx frames` exactly with a clean cable — any nonzero baseline loss masks real cable faults.
+- Small-frame runs are CPU/host-bound, not cable-bound; cable conclusions drawn from them are false.
+- Keep to the default config (hardware.md) and judge from steady state — the first ~5 s is settling transient, not residual error.
 
 ## Measured performance (AF_PACKET committed path, ice/E810 era)
-- Full size mix at 7 flow-director streams: line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup.
-- 64 B only: pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops), so only ~3.8 Gb/s.
-- RX cannot be parallelized by RSS (hardware RSS on ice can't hash raw ethertypes) — Flow Director steering by ethertype to distinct queues is what gives multiple NAPI contexts. rxnfc/fdir programming has sharp edges; see the rxnfc notes below.
-- Dead ends, measured and not to be re-attempted without new hardware (the binding constraint is total CPU across ~28 goroutines on 20 threads):
-  - **Splitting tx senders from rx streams** — raises tx but collapses rx, since rx scales with queue count, capped at 7 on that NIC.
-  - **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at tx.
-  - **Batch sizes above 64** — no gain, worse loss.
-- The tx `sendmmsg` busy-spin on ENOBUFS is not worth chasing: it only exists when the tx ring is full, which means the wire is the ceiling, so recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
 
-The AF_XDP experiment (stashed) removed the tx frame-generation ceiling (64 B tx ~14.5 Mpps/dir) and moved the bottleneck to RX (~4 Mpps/dir on AF_PACKET, ~13.5 with AF_XDP RX). It required per-packet MAC rx timestamps for honest buckets, delivered via an XDP-metadata kfunc — available on the E810 datapath, not the X710. With the test path off the E810 this whole path is parked. Note the committed AF_PACKET path *also* buckets by per-packet MAC rx stamps — via the `SO_TIMESTAMPING` cmsg, with `rx_filter=ALL` enforced as a hard host check — so the stash's novelty was the delivery mechanism and throughput, not the use of hardware stamps. Read-time stamping is how the *NIC-counter* buckets are labeled (the lesson above), not a substitute for the per-frame stamps; NICs without all-packet rx timestamping (X710, 82599) fail the host check and need a bucketing fallback — see open-questions.md §2.
+| Config | Result |
+|---|---|
+| Full size mix, 7 flow-director streams | Line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup |
+| 64 B only | pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s |
+| AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
 
-## rxnfc / Flow Director programming (ice), three traps that each cost a debugging round
-1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement, so trusting its display gives an inverted rule that silently matches nothing.
-2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192) on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
+- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts.
+- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path gets the same stamps via the `SO_TIMESTAMPING` cmsg). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
+
+## Dead ends — measured, do not re-attempt without new hardware
+
+The binding constraint was total CPU across ~28 goroutines on 20 threads.
+
+- **Splitting TX senders from RX streams** — raises TX but collapses RX (RX scales with queue count, capped at 7 on that NIC).
+- **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at TX.
+- **Batch sizes above 64** — no gain, worse loss.
+- **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
+
+## rxnfc / Flow Director programming (ice) — three traps
+
+1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement — trusting its display gives an inverted rule that silently matches nothing.
+2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192), on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
 3. **ice rejects `RX_CLS_LOC_ANY`** with ENOSPC — allocate a free location yourself (capacity from `ETHTOOL_GRXCLSRLCNT`'s `data`; the CLI allocates downward from the top). When a rule inserts but steers nothing, diff the raw bytes of a CLI-made known-good rule against yours.