X520 live as the product NIC: rate buckets re-keyed to read time (one shared clock, one commit per drained batch) with the backward smear as a settled window — each completed bucket enters once, donates excess to the nearest earlier deficits once, locks for display when nothing later can refill it (per-sample recompute double-counted excess and read >20G on a 20G wire); timestamp machinery deleted (ts.go, SO_TIMESTAMPING, rx_filter=ALL check) after the audit confirmed the buckets were its sole consumer; testDriver ixgbe; RollBall mailbox hardened against orphaned-command completions (never sample status before a poll gap, value from the same block read, devad/reg echo required — a killed session's 7.60 read answered a PHY ID as 0x0006); full hardware pass: 20.00G flat both directions, zero errors, ECD 42m, SNR +7.4dB

counters.go

@@ -47,7 +47,7 @@ var nicRxStatsByDriver = map[string][]string{
 		"mac_remote_faults.nic",
 	},
 	// The 82599 exposes no jabber, fragment, illegal-byte or fault counters
-	// (divergence: docs/open-questions.md).
+	// (docs/nics/x520/).
 	"ixgbe": {
 		"rx_crc_errors",
 		"rx_missed_errors",

docs/measurement.md

@@ -10,7 +10,13 @@ Hard-won rules about measuring correctly. Violating these produces numbers that
 - A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it.
 - Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew.
 
-This rule covers the **NIC-counter** buckets. The per-stream application buckets are a separate system stamped by per-packet MAC RX timestamps (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check); read-time stamping is not a substitute for those — software stamping was tried and cannot reach the needed precision. NICs without all-packet RX timestamping (X710, 82599) fail the host check; see nics/README.md.
+This rule covers the **NIC-counter** buckets and the per-stream receive buckets alike: the receive buckets are keyed by the clock read immediately after each `recvmmsg` drain, never by the ticker. Their read jitter is then repaired by the backward excess-fill smear (nics/x520/) — bare read-time stamping alone was tried in the E810 era and could not reach the needed precision, which is why per-packet MAC RX stamps were once a hard host requirement; the smear is what dissolved it (nics/README.md). Per-frame hardware stamps are no longer used anywhere.
+
+## A sliding smear must settle each bucket once, never recompute
+
+- Any pass that moves quantity between buckets of a sliding window must run once per bucket with its result persisted (enter → donate → display-and-retire). Recomputing the pass from raw buckets at every sample shows moved quantity twice: once in the bucket it moved to (displayed early, then slid out) and again in its source (whose donation is forgotten by the time it reaches the display slot).
+- Measured: the recompute form read 20.0–20.2 Gb/s on a 20 Gb/s wire — impossible throughput manufactured from double-counted burst excess.
+- The settled form also caps the display at line rate whenever the matching deficit is in-window; residual above-rate readings then only appear when a stall outlives the window, which is the honest signal to resize it.
 
 ## NIC counters are not monotonic
 
@@ -34,7 +40,7 @@ This rule covers the **NIC-counter** buckets. The per-stream application buckets
 | AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
 
 - RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts.
-- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path gets the same stamps via the `SO_TIMESTAMPING` cmsg). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
+- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path no longer uses hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
 
 ## Dead ends — measured, do not re-attempt without new hardware
 

docs/modules/wiitek/README.md

@@ -40,6 +40,8 @@ cabletest drives this module through its own Go RollBall client (`phy.go`): modu
 
 Not available in safe space: pre-FEC counter, cable length — those live in the µC vendor space that killed the modules.
 
+**Mailbox trap: an orphaned command's completion answers as yours.** If a session dies mid-mailbox-op (killed process), the µC finishes the orphaned command on its own time, leaving CMD=DONE and that command's value in the block. A next session that issues a command and samples status immediately reads the orphan's DONE and value as its own — seen live: a PHY ID 1.2 read answered 0x0006, the orphaned EEE-advert (7.60) value of the killed session, while the very next read was correct. The client therefore never samples status before a full poll gap, reads the value from the same 6-byte block read as the status, and accepts a completion only when the block echoes the issued devad/reg (`phy.go` mbox). The split-transaction python clients always had the pre-sample gap, which is why the field campaign never hit this.
+
 ## VCT/cable length: no confirmed-safe path
 
 This PHY offers no length capability the product needs (the length strategy lives in [../README.md](../README.md)); every known VCT candidate lands in the exact danger zone, and the safe-looking option is a documentation error corrected here:

docs/nics/README.md

@@ -8,7 +8,7 @@ A NIC is usable for module diagnostics only if the **host** can master the modul
 
 ## The timestamp requirement is dissolved
 
-All-packet hardware RX timestamps existed for one consumer — bucketing received frames by arrival time so the displayed rate stayed honest under host read-jitter. The smoothed-bucket plan ([x520](x520/README.md)) computes the rate from read-time buckets with a backward excess-fill instead, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. With that, the capability that disqualified the X520 — and the only thing the firmware-managed NICs offered over it — is no longer required (pending the audit that the rate buckets were the stamps' sole consumer). Raw-L2 steering was never a hard requirement either (encap acceptable).
+All-packet hardware RX timestamps existed for one consumer — bucketing received frames by arrival time so the displayed rate stayed honest under host read-jitter. The smoothed-bucket plan ([x520](x520/README.md)) computes the rate from read-time buckets with a backward excess-fill instead, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. The audit confirmed the rate buckets were the stamps' sole consumer, and the committed tree implements the plan; the capability that disqualified the X520 — and the only thing the firmware-managed NICs offered over it — is not required. Raw-L2 steering was never a hard requirement either (encap acceptable).
 
 ## Comparison
 

docs/nics/x520/README.md

@@ -1,29 +1,31 @@
 # Intel X520-DA2 (82599, ixgbe)
 
-The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate plan below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The plan, its prerequisite audit, and what it retires are in the section below.
+The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate design below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The committed tree implements this design.
 
 - PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
 - Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
+- The RX error counter set is the 82599's slimmer one (`counters.go`): no jabber, fragment, illegal-byte or MAC-fault counters exist — crc/missed/length errors are what this link reports; the ice set remains the richer reference.
+- ixgbe's mixed rx/tx interrupt vectors reject a tx-specific coalesce value; `checkCoalesce` falls back to rx-shared-with-tx on EINVAL, which is the normal path here. The full rx=8160/tx=4096 ring ask is taken as-is.
 - **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
 
-## The product plan: smoothed-bucket rate, no hardware stamps
+## The smoothed-bucket rate: read-time buckets, backward smear, no hardware stamps
 
-The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. The plan replaces arrival-time bucketing with read-time bucketing plus a deconvolution pass, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.
+The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. Read-time bucketing plus a deconvolution pass replaces arrival-time bucketing, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.
 
 - **Rate is a headline, never a verdict.** Lost/late/corrupt come from per-frame sequence numbers and CRC checks, independent of the rate; no cable-health decision keys off the rate number. This is the license for everything below.
 - **Buckets are keyed by read time; excess above line rate moves backward.** Read jitter is a purely backward smear — a frame is read at or after its arrival, never before — so a bucket's excess above line rate is always frames that arrived earlier and were read late, and moving that excess backward to fill earlier deficits is the physically correct deconvolution, not an estimate. The dominant jitter shape (host stalls, then drains the backlog) is deficit-then-burst, which is exactly what the backward pass repairs. Line rate is known, so over/under is well-defined.
 - **A forward pass is optional edge polish.** A stall/burst pair entirely inside the window balances under the backward pass alone; only pairs cut by the window boundary leave orphan excess or deficit. Pairing those up forward is cosmetic, is the pass most likely to paper over a genuine dip at the window edge, and is droppable if it ever lies.
-- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge (measure the actual stall distribution before choosing). The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs.
+- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge. The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs. Currently `smearWindow` = 4 buckets = 64 ms at the 16 ms bucket; resize after measuring the actual stall distribution.
 - **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
 - The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.
 
-**Prerequisite audit before building**: confirm the rate buckets are the *only* consumer of hardware RX stamps in the committed path. The buckets are currently keyed by the MAC's clock (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`), and `probe.go` leaned on a shared-PHC assumption; anything else that needs true arrival times (one-way latency, cross-port stamp comparison) does not survive the change. "Late" must remain a sequence-number notion, not a timestamp one.
+**The audit cleared**: the rate buckets were the *only* consumer of hardware RX stamps in the committed path (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`, nothing else); "late" is and stays a sequence-number notion. The implementation: buckets keyed by one shared host clock read once per drained batch (`rateEpoch`, `rxStats.commit`), and a **settled window** — each completed bucket enters once, donates its excess above line rate backward once (`fillBack`, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window: a recompute shows every excess twice (as the donation, then again unspent when its bucket reaches the display slot) and the headline reads above line rate — observed live as 20.0–20.2 G on a 20 G wire. The whole hardware-timestamp machinery is gone: the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`). The forward pass is not built.
 
 What this enables and retires:
 
 - **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal.
 - **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears.
-- **Build on HEAD** — the committed X520/BCM/ECD implementation, not the CX-5 stash. The work: re-key the receive buckets from MAC-stamp time to read time, add the backward-fill pass, and remove the `rx_filter=ALL` hard host check (`ts.go`) whose requirement this dissolves. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
+- **Built on the committed X520/BCM/ECD implementation**, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
 
 ## `sff_i2c` diagnostics transport (patched driver, validated)
 

docs/open-questions.md

@@ -2,18 +2,14 @@
 
 The genuinely open, thinking-worthy problems — nothing here is resolved. Analyses and settled facts live in the device docs; each entry points at the doc that holds its detail. Read `state.md` first for placement context.
 
-## 1. Can ConnectX-5 MCIA reach the BCM? (decide on arrival)
-
-The decisive unknown for product diagnostics coverage ([nics/connectx-5/](nics/connectx-5/README.md)):
-
-- Does firmware MCIA accept **arbitrary I2C device addresses**? The register format allows it (8-bit field, per-access status — verified in `mlx5_ifc.h`); firmware acceptance is the unknown. The module side is settled: the BCM bridge tolerates MCIA's offset-write-then-read framing, proven on hardware ([nics/connectx-5/](nics/connectx-5/README.md)).
-- Does MCIA emit **one offset byte per write** (which would let SMI command frames be synthesized exactly), and STOP-separated or repeated-START reads?
-- If firmware says no: product diagnostics via the Marvell/Aquantia modules only.
-
-## 2. ECD run disturbance
+## 1. ECD run disturbance
 
 The BCM ECD run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear). Until the disturbance is characterized, length measurement is a between-runs operation, not a during-run one ([modules/fs/](modules/fs/README.md)).
 
+## 2. Smear window sizing
+
+`smearWindow` (4 buckets = 64 ms) is a chosen default, not a measured one. Measure the host read-stall distribution under load on the X520 datapath, then size the window comfortably past the worst stall — it is also the display latency, so no larger than needed ([nics/x520/](nics/x520/README.md)).
+
 ## 3. Pre-FEC correlation run
 
 The register question is answered (post-FEC vs corrected-by-iteration histogram located); the graded-noise correlation run that verifies the expected ordering has not happened. Design and instrument: [modules/fibergaga/](modules/fibergaga/README.md).
@@ -21,13 +17,3 @@ The register question is answered (post-FEC vs corrected-by-iteration histogram
 ## 4. Wiitek VCT — pursue or leave dead?
 
 No confirmed-safe path exists (every candidate lands in the µC danger window). The open decision is whether the capability is worth the NDA route or a sacrificial unit — the product doesn't need it for length ([modules/wiitek/](modules/wiitek/README.md), [modules/README.md](modules/README.md)).
-
-## 5. X520 bench divergences — features to restore on the product NIC
-
-Running on the X520 (BCM development) required parking product-NIC capabilities the 82599 lacks. Each stays parked only until the ConnectX-5 is in; none is a settled design change:
-
-- **Hardware timestamp hard check bypassed** (`ts.go`): `rx_filter=ALL` failure now reports yellow and continues instead of stopping the run. On the X520 that means the per-frame-stamp rate buckets never fill and the panel/console rates read zero — the measurement doctrine (exact per-frame RX stamps as a hard host requirement) is intact in the docs and must return to fatal on the product NIC.
-- **RX error counter set is per-driver** (`counters.go`, `nicRxStatsByDriver`): the ice set is the reference — jabber, fragments, `illegal_bytes` (64b/66b decode errors) and MAC local/remote faults have **no ixgbe ethtool equivalent**, so those signals are simply absent on the bench. The mlx5 name set needs deriving on CX-5 arrival; the "as close to BER as the link reports" counters (illegal bytes, faults) are the ones to insist on finding there.
-- **TX interrupt moderation** (`system.go`): ixgbe's mixed rx/tx vectors reject a tx-specific value, so `checkCoalesce` falls back to rx-shared-with-tx on EINVAL. Generic and self-reporting, but verify the product NIC takes the full rx+tx pair (the fallback must never fire there).
-- **`testDriver` still names "ice"** (`main.go`): the default pair discovery has no working target — bench runs pass `-a`/`-b` explicitly. Point it at the product driver (mlx5_core) when the CX-5 lands.
-- **Module I2C transport is ixgbe-only** (`phy.go`, `openBCM`): the sff_i2c debugfs path. The CX-5 needs the MCIA answer (question 1) and a second transport arm.

docs/state.md

@@ -1,14 +1,14 @@
 # Current state
 
-## Where the code is right now, and the plan
+## Where the code is right now
 
-**The committed tree (HEAD) is the X520/BCM/ECD implementation** — AF_PACKET datapath, the mixed FS+Wiitek diagnostics, the ECD length path — and **the plan is to build on exactly that**: the smoothed-bucket rate design (nics/x520/, "The product plan") dissolves the per-frame hardware-timestamp requirement that exiled the X520, so it returns as the product NIC with the FS+Wiitek pair (length + SNR) intact. The work, once the timestamp-consumer audit clears: re-key the receive buckets from MAC-stamp time to read time, add the backward excess-fill pass, and drop the `rx_filter=ALL` hard check in `ts.go`. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
+**The committed tree is the X520 product implementation**: AF_PACKET datapath with read-time rate buckets and the backward excess-fill smear (nics/x520/, "The smoothed-bucket rate" — rate is a headline, never a verdict; `smearWindow` 4 buckets = 64 ms display latency, resize after measuring the stall distribution: open-questions), the mixed FS+Wiitek diagnostics with the BCM ECD length path, `testDriver` = ixgbe. The hardware-timestamp machinery is gone — the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`) — after the audit confirmed the rate buckets were its sole consumer. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
 
 ## The stashed Wiitek/CX-5 implementation (parts bin)
 
 All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.
 
-AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively — the `ts.go` bypass is now a no-op since the check passes); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
+AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
 
 Module diagnostics (`phy.go` + `phy_mcia.go`): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe `sff_i2c` debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because `/dev/fwctl` is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).
 
@@ -24,32 +24,29 @@ Indices shift as stashes are pushed/popped — match by message, not number.
 
 ## The box
 
-- Single usable PCIe slot (Gen4 x8): holds the ConnectX-5 (MCX512A-ACU, dual SFP28), trained Gen3 ×8 (8 GT/s, 63 Gb/s). The X520 is out of the box.
+- Single usable PCIe slot (Gen4 x8): holds the **X520-DA2** (dual SFP+). Verify 5 GT/s ×8 trained at bringup (`lspci -vv`). The ConnectX-5 is out of the box.
 - X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning).
 - Many CPU cores; goroutine-heavy designs welcome.
-- mstflint (AUR — not in the Arch repos) provides `mstlink`/`mstreg`/`mstconfig` for the CX-5.
+- mstflint (AUR — not in the Arch repos) provides `mstlink`/`mstreg`/`mstconfig` for the CX-5, if it ever returns.
 
 | Interface | Device | Role (rules: hardware.md) |
 |---|---|---|
-| `enp1s0f0np0` | ConnectX-5 port 0 (mlx5) | Test pair — Wiitek module (SN …X256U). **Linked at 10G** to its partner over the long (~45 m) cable |
-| `enp1s0f1np1` | ConnectX-5 port 1 (mlx5) | Test pair — Wiitek module (SN …X039U). **Linked at 10G** |
-| `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots). **The FS cannot live here**: X710 firmware disables Rx/Tx on the honest FS EEPROM ("unsupported SFP module type") and i40e has no `allow_unsupported_sfp` — the noise pair needs lying modules. Seated: Fibergaga (f0) + Wiitek SN …X170U (f1), both qualified, noise cycle live |
+| `enp1s0f0` + second port | X520 (ixgbe) | Test pair — FS + Wiitek, dispatched by EEPROM PN so either port may hold either module. **Stock ixgbe refuses the FS's honest EEPROM and its port has no netdev until `./load-ixgbe`** (allow_unsupported_sfp=1 + the sff_i2c/ETQF patches); the tool needs exactly 2 ixgbe interfaces |
+| `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots) — two Wiiteks, both qualified by the X710. The FS cannot live here (firmware rejects the honest EEPROM, no `allow_unsupported_sfp` in i40e) |
 | `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark |
 
 ## Hardware
 
 | Item | Status | Notes |
 |---|---|---|
-| ConnectX-5 | **Installed** in the single PCIe slot — retired as product NIC, to swap out for the X520 | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
-| X520-DA2 | Out of the box — **the product NIC under the smoothed-bucket rate plan** (nics/x520/); goes back into the slot with FS+Wiitek | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven — the HEAD implementation targets exactly this |
-| Replacement Wiiteks | Two in the CX-5 test pair (…X256U / …X039U), one in the X710 noise pair (…X170U), rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
-| FS SFP-10G-T-100 ×2 | Both on the shelf — one returns to the test pair with the X520 | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration under the X520 plan, and is what HEAD implements |
-| Fibergaga SFP-10G-T-30M | In the X710 noise pair (f0) | Aquantia, RollBall, the documented oracle |
+| X520-DA2 | **Installed** in the single PCIe slot — **the product NIC** (nics/x520/) | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven; the smoothed-bucket rate design removes the need for hardware RX stamps |
+| ConnectX-5 | Out of the box — retired | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
+| FS SFP-10G-T-100 ×2 | One in the X520 test pair, one on the shelf | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration |
+| Replacement Wiiteks | One in the X520 test pair, two in the X710 noise pair, rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
+| Fibergaga SFP-10G-T-30M | On the shelf | Aquantia, RollBall, the documented oracle; the alternate length path if the pair mix ever changes |
 | 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
 | E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
 
-**The X520 is disqualified as the product NIC — measurement, not a narrow gap.** Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, nics/x520/). Per-queue counters were explored and **rejected** — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.
-
 ## X520 diagnostics path — validated on hardware
 
 In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`:
@@ -60,13 +57,12 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
 - **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
 - **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill.
 
-## Bringup plan
+## Bringup on the swapped box
 
-1. ~~Card installed, modules seated, `./load-ixgbe`, `sff_i2c` present.~~ Done.
-2. ~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
-3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
-4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
-5. ~~RX steering: program/verify the ETQF path.~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (`~/work/phydiag-work/etqfbench`). `system.go` runs unchanged.
+1. `./load-ixgbe` — until then the FS port has no netdev (stock driver, error -95) and `driverPair("ixgbe")` cannot find 2 interfaces. Verify `sff_i2c` present for both ports and 5 GT/s ×8 trained.
+2. ~~Run.~~ Done — full pass on hardware: both modules identified and dispatched, ECD at startup (all pairs ok, ~42 m headline), steady 20.00–20.04 G both directions with zero lost/corrupt/link/internal over a 90 s run, SNR +7.4–7.5 dB, noise cycle live. SETTINGS self-reports the ixgbe shape (rx=8160/tx=4096 taken as-is; coalesce tx-shares-rx).
+3. Measure the host read-stall distribution under load and resize `smearWindow` if 64 ms is mis-sized (open-questions).
+4. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
 
 ## RX flow-steering on ixgbe
 
@@ -76,7 +72,8 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
 - **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
 - **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.
 
-- **The NIC decision is made: the X520, via the smoothed-bucket rate plan** (nics/x520/, "The product plan"). Next steps, in order: (1) the audit — confirm the rate buckets are the only consumer of hardware RX stamps (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`; `probe.go`'s shared-PHC lean; "late" must be sequence-number-based); (2) measure the host read-stall distribution to size the smoothing window; (3) re-key buckets to read time, add the backward excess-fill, drop the `rx_filter=ALL` hard check (`ts.go`). Hardware: put the X520 back in the slot, CX-5 out, FS+Wiitek seated as at HEAD. The CX-5 mailbox-quantum mystery and the E810 bit-bang idea stay recorded in their device pages but are retired.
-- **Length returns as a goal** — the X520 plan restores the FS (ECD length) alongside the Wiitek (SNR), the configuration HEAD already implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
-- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
+## Standing goals
+
+- **Length is a product goal again** — the FS (ECD length) rides alongside the Wiitek (SNR) in the test pair, which the tree implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
+- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset, re-baselining counters after the relink so the blip is never charged (`phy.go`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then — open-questions).
 - **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).

main.go

@@ -47,14 +47,24 @@ type direction struct {
 	drops uint64
 	base  counterSet
 
-	// The completed receive bucket the display draws its rate from, refreshed by
-	// the sampler because the buckets are keyed by the mac's clock and staleness
-	// has to be judged against the wall.
+	// The smeared receive bucket the display draws its rate from, refreshed by
+	// the sampler because the buckets are keyed by the shared read clock and
+	// staleness has to be judged against the wall.
 	rateFrames uint64
 	rateBytes  uint64
 	epoch      int64
 	epochAt    time.Time
 
+	// The settled window: each completed bucket enters once, donates its
+	// excess backward once, and leaves for display once no later bucket can
+	// still refill it. Recomputing the smear from the raw ring every sample
+	// instead would show each excess twice — once as the donation and again,
+	// unspent, when its bucket reaches the display slot of a later window.
+	smFrames [smearWindow]uint64
+	smBytes  [smearWindow]uint64
+	smLen    int
+	smNext   int64
+
 	nic    atomic.Uint64
 	poller *nicPoller
 }
@@ -136,11 +146,13 @@ func (w *rateWindow) at(i int) counterSet {
 // frontier means a stream has stopped delivering rather than an unchanged rate.
 const rateStale = 100 * time.Millisecond
 
-// A stamp is only knowable once a worker drains the frame carrying it, so each
-// stream's newest epoch is a frontier: everything the wire delivered to that
-// queue before it has been counted. Reading one bucket across the board takes
-// the one behind the lowest frontier, which every stream has delivered past.
-// The leader's frontier would claim buckets the stragglers are still filling.
+// An epoch is only reachable once a worker drains frames into it, so each
+// stream's newest epoch is a frontier: everything that queue has been read
+// through. Epochs behind the lowest frontier — the leader's would claim
+// buckets the stragglers are still filling — are closed to further commits,
+// so each is settled into the window exactly once. The display takes the
+// window's oldest bucket, the one no later bucket can still refill, so the
+// headline runs one window behind real time.
 func (d *direction) readRateBucket(now time.Time) {
 	newest := int64(math.MaxInt64)
 	for _, r := range d.rxStats {
@@ -151,15 +163,35 @@ func (d *direction) readRateBucket(now time.Time) {
 	if newest > d.epoch {
 		d.epoch, d.epochAt = newest, now
 	}
+	if d.smNext == 0 && d.epoch > 1 {
+		d.smNext = d.epoch - 1
+	}
+	for e := d.smNext; e > 0 && e < d.epoch; e++ {
+		d.settle(e)
+		d.smNext = e + 1
+	}
 	d.rateFrames, d.rateBytes = 0, 0
-	if d.epoch == 0 || now.Sub(d.epochAt) > rateStale {
+	if d.smLen == 0 || now.Sub(d.epochAt) > rateStale {
 		return
 	}
-	for _, r := range d.rxStats {
-		f, b := r.bucket(d.epoch - 1)
-		d.rateFrames += f
-		d.rateBytes += b
+	d.rateFrames, d.rateBytes = d.smFrames[0], d.smBytes[0]
+}
+
+func (d *direction) settle(e int64) {
+	if d.smLen == smearWindow {
+		copy(d.smFrames[:], d.smFrames[1:])
+		copy(d.smBytes[:], d.smBytes[1:])
+		d.smLen--
 	}
+	var f, b uint64
+	for _, r := range d.rxStats {
+		rf, rb := r.bucket(e)
+		f += rf
+		b += rb
+	}
+	d.smFrames[d.smLen], d.smBytes[d.smLen] = f, b
+	d.smLen++
+	fillBack(d.smFrames[:d.smLen], d.smBytes[:d.smLen])
 }
 
 type sample struct {
@@ -424,11 +456,6 @@ func buildDirection(label string, tx, rx endpoint) (*direction, error) {
 		if err != nil {
 			return nil, fmt.Errorf("%s rx socket for 0x%04x: %w", label, et, err)
 		}
-		// The mac already stamps every frame; this only asks for the stamp to be
-		// delivered.
-		if err := enableRxTimestamps(fd); err != nil {
-			return nil, fmt.Errorf("%s rx timestamps for 0x%04x: %w", label, et, err)
-		}
 		d.rxFDs = append(d.rxFDs, fd)
 		d.rxStats = append(d.rxStats, &rxStats{})
 	}
@@ -493,7 +520,7 @@ const (
 	numStreams = 7
 	batchSize  = 64
 
-	testDriver = "ice"
+	testDriver = "ixgbe"
 
 	// A constant rather than the negotiated speed, since this has to come up
 	// with no cable in the port and nothing to negotiate.
@@ -508,7 +535,7 @@ func main() {
 	// Left empty, the test pair is found by driver name instead: as PID 1 there
 	// is no udev to pin names and no command line to pass, and which port gets
 	// which ethN shifts with every driver built into the kernel.
-	aName := flag.String("a", "", "first interface (default: the ice pair)")
+	aName := flag.String("a", "", "first interface (default: the ixgbe pair)")
 	bName := flag.String("b", "", "second interface")
 	flag.Parse()
 
@@ -531,7 +558,7 @@ const (
 	// the sampling clock would stretch the window it reports.
 	sampleInterval = 16 * time.Millisecond
 	// How far back the shown errors reach. The rate is not taken from this ring
-	// but from the receive buckets, which are keyed by the mac's clock.
+	// but from the smeared receive buckets, which are keyed by the read clock.
 	rateWindowSpan = time.Second
 )
 

main_test.go

@@ -5,15 +5,13 @@ import (
 	"time"
 )
 
-// A frame lands in the bucket its receive stamp falls in, whenever the worker
-// got round to draining it.
-func TestRxObserveBucketsByStamp(t *testing.T) {
+// Batches land in the bucket of the epoch they were read in, summed, and
+// newest follows the highest epoch committed.
+func TestRxCommitBucketsByEpoch(t *testing.T) {
 	var s rxStats
-	run := rateRun{stats: &s}
-	run.add(3*int64(time.Millisecond), 100)
-	run.add(5*int64(time.Millisecond), 200)
-	run.add(rateBucketNs+int64(time.Millisecond), 300)
-	run.flush()
+	s.commit(0, 1, 100)
+	s.commit(0, 1, 200)
+	s.commit(1, 1, 300)
 
 	if f, b := s.bucket(0); f != 2 || b != 300 {
 		t.Errorf("epoch 0 = %d frames, %d bytes; want 2, 300", f, b)
@@ -30,10 +28,8 @@ func TestRxObserveBucketsByStamp(t *testing.T) {
 // reports nothing rather than the stale counts.
 func TestRxBucketWraps(t *testing.T) {
 	var s rxStats
-	run := rateRun{stats: &s}
-	run.add(1, 100)
-	run.add(rateBuckets*rateBucketNs+1, 200)
-	run.flush()
+	s.commit(0, 1, 100)
+	s.commit(rateBuckets, 1, 200)
 
 	if f, b := s.bucket(0); f != 0 || b != 0 {
 		t.Errorf("evicted epoch = %d frames, %d bytes; want 0, 0", f, b)
@@ -44,18 +40,14 @@ func TestRxBucketWraps(t *testing.T) {
 }
 
 // Every stream's frontier is past the epoch being read, so the bucket is
-// complete across the board and the rate comes from the one before the lowest
-// frontier.
+// complete across the board and the rate comes from the window behind the
+// lowest frontier.
 func TestDirectionRateReadsOneBucketBack(t *testing.T) {
 	d := &direction{rxStats: []*rxStats{{}, {}}}
-	r0 := rateRun{stats: d.rxStats[0]}
-	r1 := rateRun{stats: d.rxStats[1]}
-	r0.add(rateBucketNs+1, 500)
-	r1.add(rateBucketNs+2, 700)
-	r0.add(2*rateBucketNs+1, 900)
-	r1.add(2*rateBucketNs+2, 1100)
-	r0.flush()
-	r1.flush()
+	d.rxStats[0].commit(1, 1, 500)
+	d.rxStats[1].commit(1, 1, 700)
+	d.rxStats[0].commit(2, 1, 900)
+	d.rxStats[1].commit(2, 1, 1100)
 
 	d.readRateBucket(time.Now())
 	if d.rateFrames != 2 || d.rateBytes != 1200 {
@@ -69,11 +61,8 @@ func TestDirectionRateReadsOneBucketBack(t *testing.T) {
 // half-filled bucket summed.
 func TestDirectionRateWaitsForSlowestStream(t *testing.T) {
 	d := &direction{rxStats: []*rxStats{{}, {}}}
-	r0 := rateRun{stats: d.rxStats[0]}
-	r1 := rateRun{stats: d.rxStats[1]}
-	r0.add(rateBucketNs+1, 500)
-	r0.add(2*rateBucketNs+1, 900)
-	r0.flush()
+	d.rxStats[0].commit(1, 1, 500)
+	d.rxStats[0].commit(2, 1, 900)
 
 	// One stream has never delivered at all, so there is no epoch every stream
 	// has reached and nothing to read.
@@ -85,8 +74,7 @@ func TestDirectionRateWaitsForSlowestStream(t *testing.T) {
 
 	// The straggler is still filling the epoch the leader finished, so its
 	// bucket must not be read yet.
-	r1.add(rateBucketNs+2, 700)
-	r1.flush()
+	d.rxStats[1].commit(1, 1, 700)
 	d.readRateBucket(time.Now())
 	if d.rateFrames != 0 || d.rateBytes != 0 {
 		t.Errorf("rate = %d frames, %d bytes; want nothing while a stream is still filling the epoch",
@@ -95,8 +83,7 @@ func TestDirectionRateWaitsForSlowestStream(t *testing.T) {
 
 	// Once it moves past, the bucket is complete for both streams and is read
 	// whole.
-	r1.add(2*rateBucketNs+2, 1100)
-	r1.flush()
+	d.rxStats[1].commit(2, 1, 1100)
 	d.readRateBucket(time.Now())
 	if d.rateFrames != 2 || d.rateBytes != 1200 {
 		t.Errorf("rate = %d frames, %d bytes; want the whole completed epoch, 2 and 1200",
@@ -104,14 +91,12 @@ func TestDirectionRateWaitsForSlowestStream(t *testing.T) {
 	}
 }
 
-// A stamp only advances when a frame arrives, so an epoch that stops moving is
-// a quiet wire and must not keep reporting the last bucket.
+// An epoch only advances when a frame is read, so a frontier that stops moving
+// is a quiet wire and must not keep reporting the last bucket.
 func TestDirectionRateGoesStale(t *testing.T) {
 	d := &direction{rxStats: []*rxStats{{}}}
-	run := rateRun{stats: d.rxStats[0]}
-	run.add(rateBucketNs+1, 500)
-	run.add(2*rateBucketNs+1, 900)
-	run.flush()
+	d.rxStats[0].commit(1, 1, 500)
+	d.rxStats[0].commit(2, 1, 900)
 
 	now := time.Now()
 	d.readRateBucket(now)
@@ -124,6 +109,119 @@ func TestDirectionRateGoesStale(t *testing.T) {
 	}
 }
 
+// Each bucket entering the settled window donates once, as it arrives.
+func settleAll(frames, bytes []uint64) {
+	for i := 1; i <= len(frames); i++ {
+		fillBack(frames[:i], bytes[:i])
+	}
+}
+
+// A donated byte must never display twice: once a burst's excess has refilled
+// an earlier deficit, the burst bucket itself reaches the display slot
+// trimmed. The pre-settled-window implementation recomputed the smear from
+// the raw ring per sample and displayed the excess again — 2C here, and >20G
+// on the wire.
+func TestDirectionRateNeverRedisplaysDonatedExcess(t *testing.T) {
+	d := &direction{rxStats: []*rxStats{{}}}
+	c := bucketWireCap
+	seq := []uint64{c, 0, 2 * c, c, c, c, c, c}
+	now := time.Now()
+	for i, b := range seq {
+		d.rxStats[0].commit(int64(i+1), 0, b)
+		d.readRateBucket(now)
+		if d.rateBytes > c {
+			t.Errorf("after epoch %d: displayed %d bytes, over line rate %d", i+1, d.rateBytes, c)
+		}
+	}
+	// Jitter settled: the display holds flat line rate, not just under it.
+	if d.rateBytes != c {
+		t.Errorf("settled display = %d bytes, want exactly %d", d.rateBytes, c)
+	}
+}
+
+// A stall's deficit is refilled exactly by the burst that drains its backlog,
+// so host jitter flattens to line rate.
+func TestSmearRepairsStallBurst(t *testing.T) {
+	c := bucketWireCap
+	bytes := []uint64{c, 0, 0, 3 * c}
+	frames := make([]uint64, 4)
+	settleAll(frames, bytes)
+	for i, want := range []uint64{c, c, c, c} {
+		if bytes[i] != want {
+			t.Errorf("bucket %d = %d bytes, want %d", i, bytes[i], want)
+		}
+	}
+}
+
+// Excess refills the nearest earlier deficit, so a genuine wire dip — a
+// deficit with no matching excess — keeps its full size in its own bucket.
+func TestSmearLeavesRealLossInPlace(t *testing.T) {
+	c := bucketWireCap
+	bytes := []uint64{0, c, 0, 2 * c}
+	frames := make([]uint64, 4)
+	settleAll(frames, bytes)
+	for i, want := range []uint64{0, c, c, c} {
+		if bytes[i] != want {
+			t.Errorf("bucket %d = %d bytes, want %d", i, bytes[i], want)
+		}
+	}
+}
+
+// A deficit after the excess is the host not having read those frames yet, not
+// jitter to repair: nothing ever moves forward.
+func TestSmearNeverMovesForward(t *testing.T) {
+	c := bucketWireCap
+	bytes := []uint64{c, 2 * c, 0}
+	frames := make([]uint64, 3)
+	settleAll(frames, bytes)
+	for i, want := range []uint64{c, 2 * c, 0} {
+		if bytes[i] != want {
+			t.Errorf("bucket %d = %d bytes, want %d", i, bytes[i], want)
+		}
+	}
+}
+
+// Truly below line rate has nothing to move and displays as-is.
+func TestSmearLeavesBelowRateAlone(t *testing.T) {
+	c := bucketWireCap
+	bytes := []uint64{c / 2, c / 4, c / 2}
+	frames := []uint64{100, 50, 100}
+	want := []uint64{c / 2, c / 4, c / 2}
+	settleAll(frames, bytes)
+	for i := range bytes {
+		if bytes[i] != want[i] {
+			t.Errorf("bucket %d = %d bytes, want untouched %d", i, bytes[i], want[i])
+		}
+	}
+}
+
+// Frames ride along with the bytes in the donor's own proportion, and nothing
+// is invented or lost in the move.
+func TestSmearConservesFramesAndBytes(t *testing.T) {
+	c := bucketWireCap
+	donorFrames := uint64(1000)
+	donorBytes := 2*c - donorFrames*wireOverhead
+	frames := []uint64{0, donorFrames}
+	bytes := []uint64{0, donorBytes}
+	settleAll(frames, bytes)
+
+	if tf := frames[0] + frames[1]; tf != donorFrames {
+		t.Errorf("total frames = %d, want %d conserved", tf, donorFrames)
+	}
+	if tb := bytes[0] + bytes[1]; tb != donorBytes {
+		t.Errorf("total bytes = %d, want %d conserved", tb, donorBytes)
+	}
+	if got := bytes[0] + frames[0]*wireOverhead; got != c {
+		t.Errorf("refilled bucket = %d wire bytes, want exactly %d", got, c)
+	}
+	if frames[0] != donorFrames/2 {
+		t.Errorf("moved frames = %d, want the donor's proportion %d", frames[0], donorFrames/2)
+	}
+	if got := bytes[1] + frames[1]*wireOverhead; got != c {
+		t.Errorf("donor left = %d wire bytes, want trimmed to %d", got, c)
+	}
+}
+
 // Indexed oldest first, so the error window still spans the whole ring once it
 // has wrapped.
 func TestRateWindowIndexesOldestFirst(t *testing.T) {

phy.go

@@ -535,34 +535,44 @@ func (r *rollball) unlock() error {
 	return r.i2cWrite(rbOffPassword, 0xFF, 0xFF, 0xFF, 0xFF)
 }
 
-func (r *rollball) mbox(cmd byte, devad, reg, val uint16) error {
-	if err := r.unlock(); err != nil {
-		return err
+// The µC can be mid-service of an earlier session's command when this one is
+// issued: it completes the old one, leaving DONE and a stale value in the
+// block, and a status sample taken before the new command commits reads them
+// as this command's (seen live: PHY ID high word answered by an orphaned 7.60
+// read). So status is never sampled before a full poll gap, the value rides
+// in the same block read as the status, and a completion only counts when the
+// block echoes this command's devad/reg.
+func (r *rollball) mbox(cmd byte, devad, reg, val uint16) (out [2]byte, err error) {
+	if err = r.unlock(); err != nil {
+		return
 	}
-	if err := r.i2cWrite(rbOffDevad, byte(devad), byte(reg>>8), byte(reg)); err != nil {
-		return err
+	if err = r.i2cWrite(rbOffDevad, byte(devad), byte(reg>>8), byte(reg)); err != nil {
+		return
 	}
 	if cmd == rbCmdWrite {
-		if err := r.i2cWrite(rbOffValHi, byte(val>>8), byte(val)); err != nil {
-			return err
+		if err = r.i2cWrite(rbOffValHi, byte(val>>8), byte(val)); err != nil {
+			return
 		}
 	}
-	if err := r.i2cWrite(rbOffCmd, cmd); err != nil {
-		return err
+	if err = r.i2cWrite(rbOffCmd, cmd); err != nil {
+		return
 	}
 	deadline := time.Now().Add(rbCmdTimeout)
 	for {
-		d, err := r.i2cRead(rbOffCmd, 1)
-		if err != nil {
-			return err
+		time.Sleep(rbCmdPoll)
+		var d []byte
+		if d, err = r.i2cRead(rbOffCmd, 6); err != nil {
+			return
 		}
-		if d[0] == rbCmdDone {
-			return nil
+		if d[0] == rbCmdDone && d[1] == byte(devad) &&
+			d[2] == byte(reg>>8) && d[3] == byte(reg) {
+			out[0], out[1] = d[4], d[5]
+			return
 		}
 		if time.Now().After(deadline) {
-			return fmt.Errorf("%s: mailbox %d.%#04x stuck at %#02x", r.ifname, devad, reg, d[0])
+			err = fmt.Errorf("%s: mailbox %d.%#04x stuck at %#02x", r.ifname, devad, reg, d[0])
+			return
 		}
-		time.Sleep(rbCmdPoll)
 	}
 }
 
@@ -575,10 +585,7 @@ func rbGuard(safe map[uint16]map[uint16]bool, ifname, what string, devad, reg ui
 
 func (r *rollball) mdioRead(devad, reg uint16) (uint16, error) {
 	rbGuard(rbReadSafe, r.ifname, "read", devad, reg)
-	if err := r.mbox(rbCmdRead, devad, reg, 0); err != nil {
-		return 0, err
-	}
-	d, err := r.i2cRead(rbOffValHi, 2)
+	d, err := r.mbox(rbCmdRead, devad, reg, 0)
 	if err != nil {
 		return 0, err
 	}
@@ -587,7 +594,8 @@ func (r *rollball) mdioRead(devad, reg uint16) (uint16, error) {
 
 func (r *rollball) mdioWrite(devad, reg, val uint16) error {
 	rbGuard(rbWriteSafe, r.ifname, "write", devad, reg)
-	return r.mbox(rbCmdWrite, devad, reg, val)
+	_, err := r.mbox(rbCmdWrite, devad, reg, val)
+	return err
 }
 
 func (r *rollball) identify() (ident string, err error) {

rx.go

@@ -4,6 +4,7 @@ import (
 	"hash/crc32"
 	"sync"
 	"sync/atomic"
+	"time"
 
 	"golang.org/x/sys/unix"
 )
@@ -17,20 +18,35 @@ type rxStats struct {
 	crcErr   atomic.Uint64
 	rxErrs   atomic.Uint64
 
-	// Frames counted into the interval their mac receive stamp falls in, rather
-	// than the interval a worker got round to draining them in.
+	// Frames counted into the interval the worker read them in, on one shared
+	// host clock; read jitter is repaired at display time by the backward smear.
 	newest  atomic.Int64
 	buckets [rateBuckets]rxBucket
 }
 
-// One sample interval of arrivals, keyed by the mac's clock, with enough of them
-// kept that a bucket is read long before its slot comes round again.
+// One sample interval of reads per bucket, with enough of them kept that the
+// smear window fits and a bucket is read long before its slot comes round
+// again. smearWindow bounds how far back excess may travel and is how far
+// behind real time the displayed rate runs; it must comfortably exceed the
+// worst host read stall.
 const (
 	rateBucketNs   = int64(sampleInterval)
 	rateBuckets    = 64
 	rateBucketSecs = float64(rateBucketNs) / 1e9
+
+	smearWindow = 4
+
+	// Full wire occupancy of one bucket at line rate, in the measure gbps()
+	// reports: counted bytes plus wireOverhead per frame.
+	bucketWireCap = uint64(linkSpeed*1e9/8) * uint64(rateBucketNs) / 1_000_000_000
 )
 
+var rateEpochStart = time.Now()
+
+func rateEpoch() int64 {
+	return int64(time.Since(rateEpochStart)) / rateBucketNs
+}
+
 type rxBucket struct {
 	epoch  atomic.Int64
 	frames atomic.Uint64
@@ -53,32 +69,6 @@ func (s *rxStats) commit(e int64, frames, bytes uint64) {
 	}
 }
 
-// Frames drained together that fell in the same epoch, so the buckets take one
-// pair of adds per epoch a batch spans rather than one per frame.
-type rateRun struct {
-	stats  *rxStats
-	epoch  int64
-	frames uint64
-	bytes  uint64
-}
-
-func (r *rateRun) add(stamp int64, n uint64) {
-	if e := stamp / rateBucketNs; e != r.epoch {
-		r.flush()
-		r.epoch = e
-	}
-	r.frames++
-	r.bytes += n
-}
-
-func (r *rateRun) flush() {
-	if r.frames == 0 {
-		return
-	}
-	r.stats.commit(r.epoch, r.frames, r.bytes)
-	r.frames, r.bytes = 0, 0
-}
-
 // What this worker counted into one epoch, or nothing if that epoch has already
 // fallen out of the ring.
 func (s *rxStats) bucket(e int64) (frames, bytes uint64) {
@@ -89,6 +79,36 @@ func (s *rxStats) bucket(e int64) (frames, bytes uint64) {
 	return b.frames.Load(), b.bytes.Load()
 }
 
+// Donate the newest bucket's excess above line rate backward into the nearest
+// earlier deficits. Read jitter is purely backward — a frame is read at or
+// after its arrival — so excess is frames that arrived earlier and were read
+// late, and the burst drains the backlog of the stall immediately before it;
+// a deficit with no later excess (a genuine wire dip) keeps its full size,
+// and excess never moves forward. Frames travel with the bytes they carried,
+// in the donor's proportion. Each bucket donates exactly once, on entry to
+// the settled window — the donation mutates the stored values, so a donated
+// byte can never display again in its donor.
+func fillBack(frames, bytes []uint64) {
+	i := len(frames) - 1
+	for j := i - 1; j >= 0; j-- {
+		wire := bytes[i] + frames[i]*wireOverhead
+		if wire <= bucketWireCap {
+			return
+		}
+		have := bytes[j] + frames[j]*wireOverhead
+		if have >= bucketWireCap {
+			continue
+		}
+		take := min(wire-bucketWireCap, bucketWireCap-have)
+		mf := frames[i] * take / wire
+		mb := take - mf*wireOverhead
+		frames[i] -= mf
+		bytes[i] -= mb
+		frames[j] += mf
+		bytes[j] += mb
+	}
+}
+
 type rxWorker struct {
 	fd     int
 	batch  int
@@ -107,17 +127,11 @@ func (w *rxWorker) run(done *atomic.Bool) {
 			bufs[i][j] = 0
 		}
 	}
-	hdrs, oob := newRxMmsghdrs(bufs)
-	run := rateRun{stats: w.stats}
+	hdrs, _ := newMmsghdrs(bufs)
 
 	w.ready.Done()
 
 	for !done.Load() {
-		// The kernel overwrites each Controllen with what it wrote, so they are
-		// reset before every call.
-		for i := range hdrs {
-			hdrs[i].hdr.Controllen = cmsgLen
-		}
 		n, err := recvmmsg(w.fd, hdrs, unix.MSG_WAITFORONE)
 		if n <= 0 {
 			if err != nil && err != unix.EAGAIN && err != unix.EINTR {
@@ -125,6 +139,7 @@ func (w *rxWorker) run(done *atomic.Bool) {
 			}
 			continue
 		}
+		epoch := rateEpoch()
 		var frames, bytes uint64
 		for i := 0; i < n; i++ {
 			buf := bufs[i][:int(hdrs[i].len)]
@@ -139,9 +154,6 @@ func (w *rxWorker) run(done *atomic.Bool) {
 			}
 			frames++
 			bytes += uint64(len(buf))
-			if ts, ok := hwTimestamp(oob[i][:hdrs[i].hdr.Controllen]); ok {
-				run.add(ts, uint64(len(buf)))
-			}
 
 			// The ethertype this socket is bound to already says which stream the
 			// frame belongs to, so a header naming another one is damaged, as is a
@@ -165,6 +177,8 @@ func (w *rxWorker) run(done *atomic.Bool) {
 		// batch commits once rather than locking the line for every frame.
 		w.stats.frames.Add(frames)
 		w.stats.bytes.Add(bytes)
-		run.flush()
+		if frames > 0 {
+			w.stats.commit(epoch, frames, bytes)
+		}
 	}
 }

sock.go

@@ -101,22 +101,6 @@ func newMmsghdrs(bufs [][]byte) ([]mmsghdr, []unix.Iovec) {
 	return hdrs, iovs
 }
 
-// Room for one SCM_TIMESTAMPING and its three timespecs.
-const cmsgLen = 128
-
-// Receive headers carry a control buffer each, so the mac's receive stamp comes
-// back alongside every frame.
-func newRxMmsghdrs(bufs [][]byte) ([]mmsghdr, [][]byte) {
-	hdrs, _ := newMmsghdrs(bufs)
-	oob := make([][]byte, len(bufs))
-	for i := range bufs {
-		oob[i] = make([]byte, cmsgLen)
-		hdrs[i].hdr.Control = &oob[i][0]
-		hdrs[i].hdr.Controllen = cmsgLen
-	}
-	return hdrs, oob
-}
-
 func packetDrops(fd int) uint64 {
 	st, err := unix.GetsockoptTpacketStats(fd, unix.SOL_PACKET, unix.PACKET_STATISTICS)
 	if err != nil {

system.go

@@ -754,7 +754,6 @@ func configureSystem(ifnames []string, ethertypes []uint16) []checkResult {
 			// queues have to be installed afterwards.
 			out = append(out, checkRings(fd, ifname, wantRxRing, wantTxRing))
 			out = append(out, checkCoalesce(fd, ifname, wantCoalesceUsecs, wantCoalesceUsecs))
-			out = append(out, checkTimestamping(fd, ifname))
 			out = append(out, checkFlowRules(fd, ifname, ethertypes))
 		}
 		return out

ts.go

@@ -1,115 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"unsafe"
-
-	"golang.org/x/sys/unix"
-)
-
-const (
-	hwtstampTxOff     = 0
-	hwtstampFilterAll = 1
-)
-
-type hwtstampConfig struct {
-	flags    int32
-	txType   int32
-	rxFilter int32
-}
-
-// Temporarily bypassed so BCM work can run on the X520, which cannot stamp;
-// restore to fatal for the product NIC (docs/open-questions.md).
-func checkTimestamping(fd int, ifname string) checkResult {
-	res := configureTimestamping(fd, ifname)
-	if res.err != nil {
-		res.state = "bypassed: " + res.detail()
-		res.err = nil
-		res.fixed = true
-	}
-	return res
-}
-
-// Receive stamping is filtered by protocol and ours is not PTP, so nothing
-// narrower than "all" will see our frames.
-func configureTimestamping(fd int, ifname string) checkResult {
-	res := checkResult{item: ifname + " hw timestamps"}
-	desc := func(c hwtstampConfig) string {
-		return fmt.Sprintf("tx_type=%d rx_filter=%d", c.txType, c.rxFilter)
-	}
-
-	hwtstampCall := func(req uintptr, cfg *hwtstampConfig) error {
-		var ifr dataIfreq
-		copy(ifr.name[:], ifname)
-		ifr.data = unsafe.Pointer(cfg)
-		if _, _, errno := unix.Syscall(unix.SYS_IOCTL, uintptr(fd), req,
-			uintptr(unsafe.Pointer(&ifr))); errno != 0 {
-			return errno
-		}
-		return nil
-	}
-
-	var have hwtstampConfig
-	if err := hwtstampCall(unix.SIOCGHWTSTAMP, &have); err != nil {
-		res.err = err
-		return res
-	}
-	if have.txType == hwtstampTxOff && have.rxFilter == hwtstampFilterAll {
-		res.state = desc(have)
-		return res
-	}
-
-	// The ioctl reports back what the driver actually applied, which can be
-	// narrower than what was asked for.
-	want := hwtstampConfig{txType: hwtstampTxOff, rxFilter: hwtstampFilterAll}
-	if err := hwtstampCall(unix.SIOCSHWTSTAMP, &want); err != nil {
-		res.err = err
-		res.state = "could not set"
-		return res
-	}
-	if want.txType != hwtstampTxOff || want.rxFilter != hwtstampFilterAll {
-		res.err = fmt.Errorf("driver applied %s instead", desc(want))
-		return res
-	}
-	res.fixed = true
-	res.state = fmt.Sprintf("was %s, now %s", desc(have), desc(want))
-	return res
-}
-
-func enableRxTimestamps(fd int) error {
-	return unix.SetsockoptInt(fd, unix.SOL_SOCKET, unix.SO_TIMESTAMPING,
-		unix.SOF_TIMESTAMPING_RX_HARDWARE|unix.SOF_TIMESTAMPING_RAW_HARDWARE)
-}
-
-// Three timespecs, of which the third is the raw hardware clock. A zero there
-// means the mac did not produce a stamp.
-const scmTimestampingLen = 3 * int(unsafe.Sizeof(unix.Timespec{}))
-
-// A control message's payload starts at its header rounded up to a pointer, and
-// the next starts at its length rounded up the same way.
-const (
-	cmsgAlignMask = unix.SizeofPtr - 1
-	cmsgDataOff   = (unix.SizeofCmsghdr + cmsgAlignMask) &^ cmsgAlignMask
-)
-
-// Walked by hand rather than through ParseSocketControlMessage, which allocates
-// per call on a path that runs for every frame received.
-func hwTimestamp(oob []byte) (int64, bool) {
-	for off := 0; off+unix.SizeofCmsghdr <= len(oob); {
-		h := (*unix.Cmsghdr)(unsafe.Pointer(&oob[off]))
-		n := int(h.Len)
-		if n < unix.SizeofCmsghdr || off+n > len(oob) {
-			return 0, false
-		}
-		if h.Level == unix.SOL_SOCKET && h.Type == unix.SCM_TIMESTAMPING {
-			if cmsgDataOff+scmTimestampingLen > n {
-				return 0, false
-			}
-			ts := (*[3]unix.Timespec)(unsafe.Pointer(&oob[off+cmsgDataOff]))
-			ns := ts[2].Nano()
-			return ns, ns != 0
-		}
-		off += (n + cmsgAlignMask) &^ cmsgAlignMask
-	}
-	return 0, false
-}

ts_test.go

@@ -1,99 +0,0 @@
-package main
-
-import (
-	"testing"
-	"unsafe"
-
-	"golang.org/x/sys/unix"
-)
-
-func putCmsg(buf []byte, off int, level, typ int32, dataLen int) int {
-	h := (*unix.Cmsghdr)(unsafe.Pointer(&buf[off]))
-	h.SetLen(unix.CmsgLen(dataLen))
-	h.Level = level
-	h.Type = typ
-	return off + unix.CmsgSpace(dataLen)
-}
-
-func putStamp(buf []byte, off int, raw unix.Timespec) int {
-	next := putCmsg(buf, off, unix.SOL_SOCKET, unix.SCM_TIMESTAMPING, scmTimestampingLen)
-	ts := (*[3]unix.Timespec)(unsafe.Pointer(&buf[off+cmsgDataOff]))
-	ts[2] = raw
-	return next
-}
-
-func TestHwTimestampReadsRawHardwareClock(t *testing.T) {
-	buf := make([]byte, cmsgLen)
-	n := putStamp(buf, 0, unix.Timespec{Sec: 5, Nsec: 7})
-
-	got, ok := hwTimestamp(buf[:n])
-	if !ok || got != 5e9+7 {
-		t.Errorf("hwTimestamp = %d, %v; want 5000000007, true", got, ok)
-	}
-}
-
-func TestHwTimestampSkipsOtherMessages(t *testing.T) {
-	buf := make([]byte, cmsgLen)
-	off := putCmsg(buf, 0, unix.SOL_SOCKET, unix.SCM_RIGHTS, 4)
-	n := putStamp(buf, off, unix.Timespec{Sec: 1, Nsec: 2})
-
-	got, ok := hwTimestamp(buf[:n])
-	if !ok || got != 1e9+2 {
-		t.Errorf("hwTimestamp = %d, %v; want 1000000002, true", got, ok)
-	}
-}
-
-func TestHwTimestampRejectsZeroStamp(t *testing.T) {
-	buf := make([]byte, cmsgLen)
-	n := putStamp(buf, 0, unix.Timespec{})
-
-	if got, ok := hwTimestamp(buf[:n]); ok {
-		t.Errorf("hwTimestamp = %d, %v; want a zero stamp refused", got, ok)
-	}
-}
-
-func TestHwTimestampWithoutAStamp(t *testing.T) {
-	buf := make([]byte, cmsgLen)
-	n := putCmsg(buf, 0, unix.SOL_SOCKET, unix.SCM_RIGHTS, 4)
-
-	if _, ok := hwTimestamp(buf[:n]); ok {
-		t.Error("a buffer carrying no timestamp yielded one")
-	}
-	if _, ok := hwTimestamp(nil); ok {
-		t.Error("an empty buffer yielded a timestamp")
-	}
-}
-
-func TestHwTimestampRefusesBadLengths(t *testing.T) {
-	buf := make([]byte, cmsgLen)
-	n := putStamp(buf, 0, unix.Timespec{Sec: 1})
-	if _, ok := hwTimestamp(buf[:n-1]); ok {
-		t.Error("a message running past the buffer yielded a timestamp")
-	}
-
-	short := make([]byte, cmsgLen)
-	putCmsg(short, 0, unix.SOL_SOCKET, unix.SCM_TIMESTAMPING, scmTimestampingLen)
-	(*unix.Cmsghdr)(unsafe.Pointer(&short[0])).SetLen(unix.SizeofCmsghdr - 1)
-	if _, ok := hwTimestamp(short); ok {
-		t.Error("a message shorter than its own header yielded a timestamp")
-	}
-
-	trunc := make([]byte, cmsgLen)
-	tn := putCmsg(trunc, 0, unix.SOL_SOCKET, unix.SCM_TIMESTAMPING, scmTimestampingLen/3)
-	if _, ok := hwTimestamp(trunc[:tn]); ok {
-		t.Error("a truncated stamp yielded a timestamp")
-	}
-}
-
-func BenchmarkHwTimestamp(b *testing.B) {
-	buf := make([]byte, cmsgLen)
-	n := putStamp(buf, 0, unix.Timespec{Sec: 5, Nsec: 7})
-	oob := buf[:n]
-
-	b.ReportAllocs()
-	for b.Loop() {
-		if _, ok := hwTimestamp(oob); !ok {
-			b.Fatal("no timestamp")
-		}
-	}
-}