Dissolve resolved analyses into standalone device docs; open-questions.md keeps only the four live problems; discard NIC-timestamp length path (BCM ECD is proven) and Aquantia length trivia (30m part, oracle-only)

docs/nics/README.md

@@ -12,9 +12,14 @@ Which card is where: [../state.md](../state.md).
 
 | NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict |
 |---|---|---|---|
-| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only, serial latch) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC (§2) |
+| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (single serial latch; 1588-event frames only, EtherType configurable) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC |
 | [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set |
 | [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform |
 | [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes |
 
-Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable) — [../open-questions.md](../open-questions.md) §1–2, §6.
+Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable). Full analyses live in each device directory.
+
+## Architecture
+
+- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
+- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules and the X520 stays an offline BCM rig ([connectx-5](connectx-5/README.md)).

docs/nics/connectx-5/README.md

@@ -12,7 +12,7 @@ Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
 - Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
 - Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model.
 - RollBall (0x51-resident, offset-model) fits MCIA — low risk.
-- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../../open-questions.md](../../open-questions.md) §2, §6).
+- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
 
 ## Arrival notes
 

docs/nics/x520/README.md

@@ -1,8 +1,9 @@
 # Intel X520-DA2 (82599, ixgbe)
 
-The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial PTP-only latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault ([../../open-questions.md](../../open-questions.md) §2). The ConnectX-5 is the product NIC.
+The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial 1588-event latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault. The ConnectX-5 is the product NIC.
 
-- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s) — enough for 2×10G full duplex at the default mix.
+- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
+- Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
 - **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
 
 ## `sff_i2c` diagnostics transport (patched driver, validated)
@@ -10,22 +11,30 @@ The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmwa
 Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
 
 - **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
-- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→delay→read under a single swfw hold, removing the window. Proven on the FS BCM bridge ([../../open-questions.md](../../open-questions.md) §4).
+- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave — traffic that fires exactly around cable swaps, when diagnostics also run, so collisions are rare but correlated with the interesting moments. Serialization itself holds (driver SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`), and AT24-style word pointers do persist across STOP — the hazard is the other master *moving* the pointer or consuming the BCM 0x56 bridge's pending read data. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→STOP→in-kernel delay→read under a single swfw hold, closing the window by construction; the write/read primitives are split into locked/unlocked halves so `x` reuses them under one acquire.
+- **Compound op proven on the FS BCM** (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004); a ~3 ms hold is shorter than the driver's own SFP-identify holds. The delay sweep corroborates the hazard the op closes: at delay 0 the bridged read returns stale data (the prior read's 0x0004), at ≥200 µs it reads correctly. The >1 ms BCM delay is validated at 3 ms; treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.
+- **RollBall split transactions are field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks; the kernel's combined form is not required by the modules.
 
 ## Register/timestamp bench transport (patched driver)
 
-For the §2 exploration the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
+For the timestamp/counter bench pokes the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
 
 ## Timestamping facts
 
 **No all-packet exact RX timestamping, and no path to it — this disqualifies the X520 as product NIC.** A cable tester needs each frame's exact arrival time; the 82599 cannot give it for the 0x88b5 test traffic.
 
-- RX latches **PTP frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood; `TSIP_UT_EN` (0x00800000) will not even hold (both are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
-- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial and PTP-only, so it stamps a fraction of frames at best and none of the raw traffic. Its only surviving use is the low-volume length probes (§2/§3).
-- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset (length-probe use only).
-- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not. Measured facts (RQSMR maps 1:1; QPRC/QBRC exact, read-clear; the 2 s stats watchdog is a competing reader, `ixgbe_main.c:7780,7789,7822`) are kept in [../../open-questions.md](../../open-questions.md) §2, but the route is rejected.
+- RX latches **1588-recognized frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08 = Type 100b) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood — and the datasheet marks 100b *reserved* on the 82599 (it's an X550 define); `TSIP_UT_EN` (0x00800000) will not even hold (X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
+- RX latch eligibility is configurable, not hardwired to 0x88F7 (datasheet §7.9.3.2, §8.2.3.26.1, ETQF bit 30): a frame qualifies when its EtherType sits in an ETQF slot with the 1588 bit set and TSYNCRXCTL.Type admits its message class — valid encodings 000b (L2 sync/delay_req + msgID 2,3), 001b (V1/L4), 010b (V2 L2+L4), 101b (**every V2 event frame**: payload byte 0 low nibble with bit 3 clear). Declaring 0x88b5 as 1588 and shaping byte 0 would latch our own raw frames — datasheet-derived, untested on hardware, and irrelevant to the verdict: the serial ceiling is unchanged.
+- TX latch has no parser (§8.2.3.26.7): the advanced TX data descriptor's 1588 bit (MAC field, bit 1) stamps **any** marked frame into TXSTMPL/H — serial, TXTT cleared by TXSTMPH read. Both directions therefore stamp arbitrary frames one-at-a-time: a ping-pong probe gets PHY-adjacent hardware stamps on both ends, paced by the MMIO read loop.
+- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial, so it stamps a fraction of frames at best and none of the line-rate traffic. NIC-timestamp length measurement is discarded (length strategy: [../../modules/README.md](../../modules/README.md)), so the latch has no committed use.
+- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal; the two SYSTIMEs start at driver load and can sit seconds apart.
+- SDP auxiliary elements (§7.9.4): an edge on SDP2/3 latches SYSTIME into AUXSTMP0/1 (two external-event timestamp channels); target-time registers assert an SDP pin when SYSTIME reaches a programmed value; SDP6/7 carry configurable clock outputs — hooks for external instrumentation on the same clock that stamps packets.
+- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not — it cannot place a single frame in time, so it cannot see the fault. Measured facts retained for the record: RQSMR maps queue→counter 1:1; QPRC/QBRC exact, read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`). Also rejected: wire-determinism reconstruction (per-frame times from the size schedule assume saturation regularity — blind exactly at loss/retrain events).
 
 ## RX steering facts
 
-- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged.
-- **ETQF/ETQS L2 EtherType steering is proven on hardware**: dedicated ethertype→queue stage ahead of RSS/fdir; 7 streams land on 7 queues, unfiltered ethertypes fall to queue 0. Flow Director cannot match non-IP frames at all (a flex-word rule on raw 0x88b5 steers nothing — confirmed). Details and the FCoE-slot fix: [../../open-questions.md](../../open-questions.md) §1.
+- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged; `ETHTOOL_GRXCLSRULE`/`GRXCLSRLALL` round-trip the rules so stale-rule cleanup sees and clears them.
+- **ETQF/ETQS L2 EtherType steering is proven on hardware** (`~/work/phydiag-work/etqfbench`): a dedicated ethertype→queue stage checked at L2, ahead of RSS/fdir — ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE + QUEUE_EN, and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`). 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each; unfiltered ethertypes fall to queue 0 (non-IP frames hash to RSS 0); delete restores the fall-through.
+- **Slot budget fits**: 8 filters; the patch reserves index 3 (1588) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes moved behind the FCoE-enabled guard (without that, FCoE CRC pre-claims slot 2). Restore-on-link-up (`ixgbe_etqf_filter_restore`) and clear-on-close (`ixgbe_etqf_filter_exit`) are hooked so a ring-resize reset re-applies the rules.
+- **Flow Director is a dead end for raw L2 — confirmed on hardware**: 82599 fdir classifies IPv4/IPv6 only; a flex-word rule at the ethertype offset (`FDIRCTRL_FLEX_SHIFT = 0x6`) on a raw 0x88b5 stream steers nothing (`etqfbench -mode fdirneg`) — the flex word only *narrows* an IP-typed match.
+- Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap acceptable — raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix).

docs/nics/x710/README.md

@@ -17,5 +17,5 @@
 ## Timestamping
 
 - Rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`) — PTP-only, so no committed-path bucketing here.
-- Genuinely shared PHC across ports (one oscillator) + PTP latches: the platform for the retrain-stability experiment and the scoped-but-unbuilt PTP path-delay length fallback ([../../open-questions.md](../../open-questions.md) §2) — probes must be PTP-shaped here too.
+- Genuinely shared PHC across ports (one oscillator) + PTP latches — no committed use: NIC-timestamp length measurement is discarded (length strategy: [../../modules/README.md](../../modules/README.md)).
 - No EEE on i40e/X710.