Dissolve resolved analyses into standalone device docs; open-questions.md keeps only the four live problems; discard NIC-timestamp length path (BCM ECD is proven) and Aquantia length trivia (30m part, oracle-only)
This commit is contained in:
flamingcow
+1
-1
@@ -8,7 +8,7 @@ General:
|
||||
- [state.md](state.md) — where things are now: committed vs stashed, the box and what's in it, bringup progress, open items. **All current-placement state lives here and only here.**
|
||||
- [hardware.md](hardware.md) — host rules: interface conventions, tuning, how to run.
|
||||
- [measurement.md](measurement.md) — how to measure correctly; performance results and dead ends.
|
||||
- [open-questions.md](open-questions.md) — the open problems, with their current answers.
|
||||
- [open-questions.md](open-questions.md) — the genuinely open problems, as pointers into the docs that hold their detail.
|
||||
|
||||
Per-device (each directory: device doc + its datasheets):
|
||||
|
||||
|
||||
+2
-5
@@ -9,11 +9,8 @@ The target of measurement is the cable, not throughput. Line rate is a means to
|
||||
1. **Loss and error attribution** — reception gaps, link errors, NIC/driver counters as first-class output alongside application loss. Baseline loss must be exactly zero before a run counts; any host-side loss masks real cable faults.
|
||||
2. **Noise tolerance** — a deliberately-bad "noise" cable intertwined with the test cable, driven by link up/down cycling, stresses the cable under test with alien crosstalk.
|
||||
3. **Per-pair SNR** from the module PHYs — the leading indicator of a marginal cable before it drops frames. IEEE 802.3an standard registers on the Marvells; the BCM leaves those unpopulated and reports through its vendor command handler instead (modules/fs/).
|
||||
4. **Cable length** — sanity check and fault localization. Sources, in preference order:
|
||||
- Module PHY DSP estimate — works on a linked cable.
|
||||
- PHY TDR — localizes opens/shorts both-ended; healthy-cable length only single-ended.
|
||||
- NIC timestamp path-delay — works linked, module-independent; needs hardware timestamps and a short-cable calibration.
|
||||
5. **Pre-FEC error visibility** — corrected-error counters that move before post-FEC loss appears. Vendor-specific; located on the Aquantia (open-questions.md §5), unlocated elsewhere. Standard latched PCS counters (errored blocks, BER, block-lock loss) are the working proxy under noise stress.
|
||||
4. **Cable length** — sanity check and fault localization. The BCM ECD is the product path: per-pair lengths in meters, healthy pairs included, proven meter-accurate on the bench (modules/fs/).
|
||||
5. **Pre-FEC error visibility** — corrected-error counters that move before post-FEC loss appears. Vendor-specific; located on the Aquantia (modules/fibergaga/), unlocated elsewhere. Standard latched PCS counters (errored blocks, BER, block-lock loss) are the working proxy under noise stress.
|
||||
|
||||
## Design preferences that shaped the tool
|
||||
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ Hard-won rules about measuring correctly. Violating these produces numbers that
|
||||
- A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it.
|
||||
- Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew.
|
||||
|
||||
This rule covers the **NIC-counter** buckets. The per-stream application buckets are a separate system stamped by per-packet MAC RX timestamps (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check); read-time stamping is not a substitute for those — software stamping was tried and cannot reach the needed precision. NICs without all-packet RX timestamping (X710, 82599) fail the host check; see open-questions.md §2.
|
||||
This rule covers the **NIC-counter** buckets. The per-stream application buckets are a separate system stamped by per-packet MAC RX timestamps (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check); read-time stamping is not a substitute for those — software stamping was tried and cannot reach the needed precision. NICs without all-packet RX timestamping (X710, 82599) fail the host check; see nics/README.md.
|
||||
|
||||
## NIC counters are not monotonic
|
||||
|
||||
|
||||
@@ -53,6 +53,5 @@ Split transactions (separate STOP per byte) are field-proven on both the Fiberga
|
||||
|
||||
## Cable-length physics (applies to all vendors)
|
||||
|
||||
- **TDR** (classic VCT/CDT/ECD): locates opens/shorts both-ended. But *healthy-cable length* needs a far-end reflection, and a plugged-in terminated far end absorbs the pulse — "OK" pairs report no distance. Effectively single-ended for length; every kernel implementation only reports distance for *fault* results.
|
||||
- **TDR** (classic VCT/CDT/ECD): locates opens/shorts both-ended. The generic concern — *healthy-cable length* needs a far-end reflection, and a plugged-in terminated far end absorbs the pulse, so "OK" pairs report no distance (every kernel implementation only reports distance for *fault* results) — is disproven for the BCM: its ECD reports per-pair length for healthy pairs on a plugged, terminated, linked cable, meter-accurate ([fs/](fs/README.md)). **The BCM ECD is the product length path.**
|
||||
- **DSP estimate**: both vendors' PHYs compute linked-cable length continuously from insertion loss during training (Marvell "proactively determines cable length"; the BCM's limited-reach power mode switches "based on cable length" while linked). Works in the product's both-ends-plugged topology — the question is only whether the estimate is exposed in a readable register (Aquantia: yes, `1E.C884`; BCM/Marvell: not in documented space).
|
||||
- **NIC timestamp path-delay** — the module-independent fallback: works linked, needs hardware timestamps and a short-cable calibration to cancel fixed PHY latency (~4.8 ns/m one way; 50 m ≈ 240 ns). Viability gated on PHY-latency stability across retrains ([../open-questions.md](../open-questions.md) §2).
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# Fibergaga SFP-10G-T-30M (Aquantia AQR)
|
||||
|
||||
The documented oracle. RollBall module, PHY ID 0x31c31c13, proven over the i40e oracle path ([../../nics/x710/](../../nics/x710/README.md)). Its PHY family register reference is vendored in this directory (`aquantia-aqr107-aqr109-datasheet.pdf` — the AQR107-AQR109 register reference).
|
||||
The documented oracle — and only that: the part's 30 m reach limit makes it unusable in the test set, so its registers matter as reference/instrument, never as product capability. RollBall module, PHY ID 0x31c31c13, proven over the i40e oracle path ([../../nics/x710/](../../nics/x710/README.md)). Its PHY family register reference is vendored in this directory (`aquantia-aqr107-aqr109-datasheet.pdf` — the AQR107-AQR109 register reference).
|
||||
|
||||
## Documented registers
|
||||
|
||||
| Registers | What |
|
||||
|---|---|
|
||||
| 3.E820/3.E821 | PCS Receive Vendor **Uncorrected** Frame Counter (post-FEC) |
|
||||
| 3.E840–3.E84E | Corrected Frame N Iteration Counters, N = 1…10 LDPC iterations — the pre-FEC leading indicator ([../../open-questions.md](../../open-questions.md) §5) |
|
||||
| 3.E840–3.E84E | Corrected Frame N Iteration Counters, N = 1…10 LDPC iterations — the pre-FEC leading indicator |
|
||||
| 3.E816 | RS Corrected Error Frame Counter (system-side, secondary) |
|
||||
| 1E.C884 bits 7:0 | **Estimated cable length in meters, ±1 m**, RO, from the cable diagnostic engine |
|
||||
| 1E.C800 | Per-pair TDR fault verdicts (open/short/mismatch/cross-pair) |
|
||||
@@ -15,7 +15,13 @@ The documented oracle. RollBall module, PHY ID 0x31c31c13, proven over the i40e
|
||||
| 1E.C880–C883 | Reflection impedance/magnitude |
|
||||
| PMA 1.133–1.140, PCS 3.32/3.33 | Standard set ([../README.md](../README.md)), proven live |
|
||||
|
||||
## Open bench questions
|
||||
## Pre-FEC correlation run (verification, not classification)
|
||||
|
||||
- Whether `1E.C884` is continuously valid on a linked cable or only after an explicit diag run.
|
||||
- The pre-FEC correlation experiment ([../../open-questions.md](../../open-questions.md) §5) — this module is its instrument.
|
||||
`3.E820` is post-FEC; the pre-FEC counters are the corrected-by-iteration histogram (3.E840–E84E). The correlation run verifies the expected ordering; this module is its instrument. Design points:
|
||||
|
||||
- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
|
||||
- Run at high line utilization: an uncorrected LDPC frame landing in idle/IFG leaves no host-visible trace; low-load runs undercount the host side.
|
||||
- Expected ordering as margin falls: corrected-iteration counters shift toward higher N first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1.
|
||||
- Log fast retrains alongside (IEEE 1.147) to separate FEC exhaustion from retrain hiccups.
|
||||
- Clear-on-read discipline: exactly one reader during a run.
|
||||
- Scope: documents the Aquantia oracle — fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out until documented.
|
||||
|
||||
@@ -72,6 +72,6 @@ Observed on a plugged, linked, healthy cable: verdicts OK×4 and per-pair length
|
||||
Chapters the datasheet TOC lists but the excerpts here omit — to source elsewhere:
|
||||
|
||||
1. The **ECD chapter** — now for confirming bit meanings rather than unblocking.
|
||||
2. The **1588 operation chapter** (in-PHY timestamping would measure path delay at the MDI, removing PHY-pipeline latency from the length equation; [../../open-questions.md](../../open-questions.md) §2).
|
||||
2. The **1588 operation chapter**.
|
||||
3. Datasheet **§1.20 loopback** (copper line loopback) and **§1.17 EEE/fast-retrain monitoring**.
|
||||
4. **Chapter 2 register summary.** The excerpt's TOC names them all.
|
||||
|
||||
@@ -33,7 +33,7 @@ Not available in safe space: pre-FEC counter, cable length — those live in the
|
||||
|
||||
## VCT/cable length: no confirmed-safe path
|
||||
|
||||
The problem is real (if both loop ends become Wiiteks, this PHY is the only length source), but every known VCT candidate lands in the exact danger zone, and the safe-looking option is a documentation error corrected here:
|
||||
This PHY offers no length capability the product needs (the length strategy lives in [../README.md](../README.md)); every known VCT candidate lands in the exact danger zone, and the safe-looking option is a documentation error corrected here:
|
||||
|
||||
- **The 88Q2220 TDR recipe is NOT safe.** Its registers (MMD 3, 0xFECA/0xFED9/0xFEDA/0xFEDE/0xFEDD) sit at `3.0xFExx` — inside the `3.0x8000+` µC-mapped window this doc identifies as the space that bricked both modules. Low-byte ≠ 0x64 does **not** make them safe; the whole high window is µC-intercept space, and the brick was a µC crash on access, not specifically the 0x64 decode. (The 0x64 pattern is the one *confirmed* kill; it is not the boundary of the danger.) Treat the entire `3.0x8000+` / `31.0xF000+` space as lethal-until-proven, not the `*.??64` addresses alone.
|
||||
- **The QCA-style CDT block** (`qcom/qcom.h`: 3.0x8064 status, 3.0x8065–0x8068 per-pair results, 3.0x8074–0x807E thresholds) is the most likely *intended* VCT — the ODM firmware emulates it — but it is centered on `3.0x8064`, the precise register that killed both originals. Hypothesis: the kill was a *cold* status read before running the test, and the real protocol is write-start → wait → read-status/results. Plausible, unproven, and not worth a live module.
|
||||
@@ -41,8 +41,7 @@ The problem is real (if both loop ends become Wiiteks, this PHY is the only leng
|
||||
|
||||
Routes that do not risk the module:
|
||||
|
||||
1. **NIC timestamp path-delay** for length — module-independent, works whatever modules are at the ends ([../../open-questions.md](../../open-questions.md) §2). This directly answers the same-module-both-ends case; prioritize the X710 retrain-stability experiment.
|
||||
2. **The real VCT register map** from Marvell/Wiitek under NDA — the only route to a *confirmed*-safe sequence.
|
||||
3. **Deliberate sacrifice** of a spare to test the QCA-CDT-start hypothesis, `rb_forensic.py` running — only if the capability is judged worth a module.
|
||||
1. **The real VCT register map** from Marvell/Wiitek under NDA — the only route to a *confirmed*-safe sequence.
|
||||
2. **Deliberate sacrifice** of a spare to test the QCA-CDT-start hypothesis, `rb_forensic.py` running — only if the capability is judged worth a module.
|
||||
|
||||
The Marvell 88E151x datasheet PDF in this directory is family reference material.
|
||||
|
||||
+7
-2
@@ -12,9 +12,14 @@ Which card is where: [../state.md](../state.md).
|
||||
|
||||
| NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict |
|
||||
|---|---|---|---|
|
||||
| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only, serial latch) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC (§2) |
|
||||
| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (single serial latch; 1588-event frames only, EtherType configurable) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC |
|
||||
| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set |
|
||||
| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform |
|
||||
| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes |
|
||||
|
||||
Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable) — [../open-questions.md](../open-questions.md) §1–2, §6.
|
||||
Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable). Full analyses live in each device directory.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
|
||||
- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules and the X520 stays an offline BCM rig ([connectx-5](connectx-5/README.md)).
|
||||
|
||||
@@ -12,7 +12,7 @@ Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
|
||||
- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
|
||||
- Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model.
|
||||
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
|
||||
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../../open-questions.md](../../open-questions.md) §2, §6).
|
||||
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
|
||||
|
||||
## Arrival notes
|
||||
|
||||
|
||||
+19
-10
@@ -1,8 +1,9 @@
|
||||
# Intel X520-DA2 (82599, ixgbe)
|
||||
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial PTP-only latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault ([../../open-questions.md](../../open-questions.md) §2). The ConnectX-5 is the product NIC.
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial 1588-event latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault. The ConnectX-5 is the product NIC.
|
||||
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s) — enough for 2×10G full duplex at the default mix.
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
|
||||
- Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
|
||||
- **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
|
||||
|
||||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||||
@@ -10,22 +11,30 @@ The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmwa
|
||||
Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
|
||||
|
||||
- **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
|
||||
- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→delay→read under a single swfw hold, removing the window. Proven on the FS BCM bridge ([../../open-questions.md](../../open-questions.md) §4).
|
||||
- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave — traffic that fires exactly around cable swaps, when diagnostics also run, so collisions are rare but correlated with the interesting moments. Serialization itself holds (driver SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`), and AT24-style word pointers do persist across STOP — the hazard is the other master *moving* the pointer or consuming the BCM 0x56 bridge's pending read data. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→STOP→in-kernel delay→read under a single swfw hold, closing the window by construction; the write/read primitives are split into locked/unlocked halves so `x` reuses them under one acquire.
|
||||
- **Compound op proven on the FS BCM** (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004); a ~3 ms hold is shorter than the driver's own SFP-identify holds. The delay sweep corroborates the hazard the op closes: at delay 0 the bridged read returns stale data (the prior read's 0x0004), at ≥200 µs it reads correctly. The >1 ms BCM delay is validated at 3 ms; treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.
|
||||
- **RollBall split transactions are field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks; the kernel's combined form is not required by the modules.
|
||||
|
||||
## Register/timestamp bench transport (patched driver)
|
||||
|
||||
For the §2 exploration the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
|
||||
For the timestamp/counter bench pokes the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
|
||||
|
||||
## Timestamping facts
|
||||
|
||||
**No all-packet exact RX timestamping, and no path to it — this disqualifies the X520 as product NIC.** A cable tester needs each frame's exact arrival time; the 82599 cannot give it for the 0x88b5 test traffic.
|
||||
|
||||
- RX latches **PTP frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood; `TSIP_UT_EN` (0x00800000) will not even hold (both are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
|
||||
- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial and PTP-only, so it stamps a fraction of frames at best and none of the raw traffic. Its only surviving use is the low-volume length probes (§2/§3).
|
||||
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset (length-probe use only).
|
||||
- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not. Measured facts (RQSMR maps 1:1; QPRC/QBRC exact, read-clear; the 2 s stats watchdog is a competing reader, `ixgbe_main.c:7780,7789,7822`) are kept in [../../open-questions.md](../../open-questions.md) §2, but the route is rejected.
|
||||
- RX latches **1588-recognized frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08 = Type 100b) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood — and the datasheet marks 100b *reserved* on the 82599 (it's an X550 define); `TSIP_UT_EN` (0x00800000) will not even hold (X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
|
||||
- RX latch eligibility is configurable, not hardwired to 0x88F7 (datasheet §7.9.3.2, §8.2.3.26.1, ETQF bit 30): a frame qualifies when its EtherType sits in an ETQF slot with the 1588 bit set and TSYNCRXCTL.Type admits its message class — valid encodings 000b (L2 sync/delay_req + msgID 2,3), 001b (V1/L4), 010b (V2 L2+L4), 101b (**every V2 event frame**: payload byte 0 low nibble with bit 3 clear). Declaring 0x88b5 as 1588 and shaping byte 0 would latch our own raw frames — datasheet-derived, untested on hardware, and irrelevant to the verdict: the serial ceiling is unchanged.
|
||||
- TX latch has no parser (§8.2.3.26.7): the advanced TX data descriptor's 1588 bit (MAC field, bit 1) stamps **any** marked frame into TXSTMPL/H — serial, TXTT cleared by TXSTMPH read. Both directions therefore stamp arbitrary frames one-at-a-time: a ping-pong probe gets PHY-adjacent hardware stamps on both ends, paced by the MMIO read loop.
|
||||
- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial, so it stamps a fraction of frames at best and none of the line-rate traffic. NIC-timestamp length measurement is discarded (length strategy: [../../modules/README.md](../../modules/README.md)), so the latch has no committed use.
|
||||
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal; the two SYSTIMEs start at driver load and can sit seconds apart.
|
||||
- SDP auxiliary elements (§7.9.4): an edge on SDP2/3 latches SYSTIME into AUXSTMP0/1 (two external-event timestamp channels); target-time registers assert an SDP pin when SYSTIME reaches a programmed value; SDP6/7 carry configurable clock outputs — hooks for external instrumentation on the same clock that stamps packets.
|
||||
- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not — it cannot place a single frame in time, so it cannot see the fault. Measured facts retained for the record: RQSMR maps queue→counter 1:1; QPRC/QBRC exact, read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`). Also rejected: wire-determinism reconstruction (per-frame times from the size schedule assume saturation regularity — blind exactly at loss/retrain events).
|
||||
|
||||
## RX steering facts
|
||||
|
||||
- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged.
|
||||
- **ETQF/ETQS L2 EtherType steering is proven on hardware**: dedicated ethertype→queue stage ahead of RSS/fdir; 7 streams land on 7 queues, unfiltered ethertypes fall to queue 0. Flow Director cannot match non-IP frames at all (a flex-word rule on raw 0x88b5 steers nothing — confirmed). Details and the FCoE-slot fix: [../../open-questions.md](../../open-questions.md) §1.
|
||||
- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged; `ETHTOOL_GRXCLSRULE`/`GRXCLSRLALL` round-trip the rules so stale-rule cleanup sees and clears them.
|
||||
- **ETQF/ETQS L2 EtherType steering is proven on hardware** (`~/work/phydiag-work/etqfbench`): a dedicated ethertype→queue stage checked at L2, ahead of RSS/fdir — ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE + QUEUE_EN, and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`). 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each; unfiltered ethertypes fall to queue 0 (non-IP frames hash to RSS 0); delete restores the fall-through.
|
||||
- **Slot budget fits**: 8 filters; the patch reserves index 3 (1588) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes moved behind the FCoE-enabled guard (without that, FCoE CRC pre-claims slot 2). Restore-on-link-up (`ixgbe_etqf_filter_restore`) and clear-on-close (`ixgbe_etqf_filter_exit`) are hooked so a ring-resize reset re-applies the rules.
|
||||
- **Flow Director is a dead end for raw L2 — confirmed on hardware**: 82599 fdir classifies IPv4/IPv6 only; a flex-word rule at the ethertype offset (`FDIRCTRL_FLEX_SHIFT = 0x6`) on a raw 0x88b5 stream steers nothing (`etqfbench -mode fdirneg`) — the flex word only *narrows* an IP-typed match.
|
||||
- Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap acceptable — raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix).
|
||||
|
||||
@@ -17,5 +17,5 @@
|
||||
## Timestamping
|
||||
|
||||
- Rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`) — PTP-only, so no committed-path bucketing here.
|
||||
- Genuinely shared PHC across ports (one oscillator) + PTP latches: the platform for the retrain-stability experiment and the scoped-but-unbuilt PTP path-delay length fallback ([../../open-questions.md](../../open-questions.md) §2) — probes must be PTP-shaped here too.
|
||||
- Genuinely shared PHC across ports (one oscillator) + PTP latches — no committed use: NIC-timestamp length measurement is discarded (length strategy: [../../modules/README.md](../../modules/README.md)).
|
||||
- No EEE on i40e/X710.
|
||||
|
||||
+12
-101
@@ -1,112 +1,23 @@
|
||||
# Open questions
|
||||
|
||||
The genuinely open, thinking-worthy problems and their current answers — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `nics/`, `modules/`, `measurement.md`); this assumes that context.
|
||||
The genuinely open, thinking-worthy problems — nothing here is resolved. Analyses and settled facts live in the device docs; each entry points at the doc that holds its detail. Read `state.md` first for placement context.
|
||||
|
||||
Framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. Which cards and modules are in play: `state.md`.
|
||||
## 1. Can ConnectX-5 MCIA reach the BCM? (decide on arrival)
|
||||
|
||||
## 1. Raw-L2 RX steering on 82599
|
||||
The decisive unknown for product diagnostics coverage ([nics/connectx-5/](nics/connectx-5/README.md)):
|
||||
|
||||
**Status: solved and proven on hardware — ETQF steers raw ethertypes to per-queue contexts.** The committed tool steers raw ethertype 0x88b5+ to per-queue NAPI contexts; stock ixgbe rejects ETHER_FLOW ntuple rules. The patched ixgbe maps exact-ethertype ETHER_FLOW inserts onto ETQF slots (`~/work/ixgbe-sff/ixgbe_ethtool.c`, `ixgbe_add_etqf_entry`), so `system.go` runs unchanged.
|
||||
- Does firmware MCIA accept **arbitrary I2C device addresses**? The BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
|
||||
- Can the BCM SMI **read data phase** (raw 2-byte read, no offset write) be framed through MCIA's offset model?
|
||||
- If no to either: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig.
|
||||
|
||||
- **ETQF/ETQS is a dedicated ethertype→RX-queue stage** checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses it — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`).
|
||||
- **Bench-confirmed** (`~/work/phydiag-work/etqfbench`): 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each on their queue; an 8th unfiltered ethertype falls to queue 0; delete restores the fall-through. `ETHTOOL_GRXCLSRULE`/`GRXCLSRLALL` round-trip the ETQF rules so the tool's stale-rule cleanup sees and clears them.
|
||||
- **Slot budget fits**: 8 filters; the patch reserves index 3 (1588, needed by §2) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes were moved behind the FCoE-enabled guard (`ixgbe_fcoe.c`) so the default config leaves slots 0/1/2 free; without that, FCoE CRC pre-claimed slot 2. Restore-on-link-up (`ixgbe_etqf_filter_restore`) and clear-on-close (`ixgbe_etqf_filter_exit`) are hooked so a ring-resize reset re-applies the rules.
|
||||
- **Flow Director is a dead end for raw L2 — confirmed on hardware.** 82599 fdir classifies IPv4/IPv6 only; a flex-word rule (`FDIRCTRL_FLEX_SHIFT = 0x6` = byte offset 12 = the ethertype field) targeting a raw 0x88b5 stream steers nothing — every frame falls to queue 0 (`etqfbench -mode fdirneg`). Non-IP frames never enter the fdir lookup; the flex word only *narrows* an IP-typed match.
|
||||
- Baseline behavior without filters: non-IP frames hash to RSS 0 → all land on queue 0.
|
||||
- Fallbacks (unneeded now): MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix).
|
||||
## 2. ECD run disturbance
|
||||
|
||||
## 2. Zero-baseline loss and timestamping on the 82599
|
||||
The BCM ECD run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear). Until the disturbance is characterized, length measurement is a between-runs operation, not a during-run one ([modules/fs/](modules/fs/README.md)).
|
||||
|
||||
**Status: throughput yes; all-packet exact RX timestamps no — and there is no substitute. A cable tester needs each frame's exact arrival time; aggregate counters/rates are honest only when nothing is wrong, which is useless for fault detection. The 82599 has one serial PTP-only latch and nothing else, so the X520 is disqualified as the product measurement NIC (diagnostics-only). The ConnectX-5 is the product path.**
|
||||
## 3. Pre-FEC correlation run
|
||||
|
||||
### Throughput
|
||||
The register question is answered (post-FEC vs corrected-by-iteration histogram located); the graded-noise correlation run that verifies the expected ordering has not happened. Design and instrument: [modules/fibergaga/](modules/fibergaga/README.md).
|
||||
|
||||
- PCIe Gen2 x8 ≈ 32 Gb/s raw, ~25–26 Gb/s/direction effective, vs 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already. Verify 5 GT/s ×8 trained (`lspci -vv`).
|
||||
- Loss attribution survives: 82599 exposes missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" still works.
|
||||
## 4. Wiitek VCT — pursue or leave dead?
|
||||
|
||||
### Timestamping is two separate uses — keep them distinct
|
||||
|
||||
| Use | Mechanism (committed) | 82599 |
|
||||
|---|---|---|
|
||||
| 1. Rate bucketing — every received frame | Per-packet MAC RX stamps via `SO_TIMESTAMPING` cmsg (`rx.go` `rateRun.add`); `rx_filter=ALL` enforced as a hard host check (`ts.go`; failed checks abort, `system.go` `reportChecks`) | **Cannot** — PTP-only (`TYPE_ALL` is X550+, `ixgbe_ptp.c:1043`); i40e also rejects `FILTER_ALL` with -ERANGE (verified upstream). Only the E810 qualified |
|
||||
| 2. Length probes — low volume | `probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry | Possible with PTP-shaped probes (below) |
|
||||
|
||||
**There is no software fallback for use 1** — software RX stamping was tried and cannot reach the needed precision. All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright. (measurement.md's read-time-stamping lesson covers the *NIC-counter* buckets, a separate system.)
|
||||
|
||||
### The full 82599 hardware space for use 1 — and why none of it substitutes for per-packet stamps
|
||||
|
||||
**A cable tester measures the receive precisely when it is misbehaving — loss bursts, late frames, timing excursions to correlate with pre-FEC errors and retrains. That signal lives in each frame's exact arrival time. Any aggregate (a rate, a count per window) is only honest when the receive is well-behaved, i.e. when there is nothing to measure. Exact per-frame timestamps are the requirement, full stop; nothing on the 82599 provides them for all packets.**
|
||||
|
||||
What X550 calls "timestamp all" is `TYPE_ALL` filtering **plus TSIP** (`TSYNCRXCTL_TSIP_UT_EN` — stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 has neither — confirmed on hardware: writing `TYPE_ALL` sticks in the register but never arms the latch on raw frames, and the `TSIP_UT_EN` bit will not even hold. All it has is one RX latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`) with attribute-capture (RXSATRL/H), no timestamp field in the RX descriptor.
|
||||
|
||||
The routes that were mapped here are all **rejected** for a cable tester:
|
||||
|
||||
1. ~~Per-queue hardware counters as bucket content.~~ **Rejected — it's a rate, not timestamps.** QPRC/QBRC deltas per window give aggregate throughput, honest only in zero-loss steady state (the FIFO smear that would perturb a boundary window is ~0 only when nothing is going wrong). It measured cleanly on the bench (RQSMR maps 1:1, read-clear, exact counts) but it cannot place a single frame in time, so it cannot see the fault a cable tester is built to catch. Measured facts retained below; the conclusion is dead.
|
||||
2. ~~The latch as an identifiable sampler.~~ **Not all-packet.** One serial latch, PTP-only, ~383 k stamps/s from a dedicated in-kernel spin loop against ~1.78 M pps/dir line rate. Useful only for the low-volume length probes (below), never for stamping the 0x88b5 test traffic.
|
||||
3. ~~Wire-determinism reconstruction.~~ **Rejected — same failure as counters.** Reconstructing per-frame wire times from the size schedule assumes saturation regularity, which a loss/retrain event breaks; it is blind at exactly the interesting moments.
|
||||
|
||||
**Verdict: the 82599 cannot do all-packet exact RX timestamping and has no path to it. The X520 is disqualified as the product measurement NIC — diagnostics / BCM-framing card only.** The ConnectX-5 (per-packet CQE stamps, `HWTSTAMP_FILTER_ALL`) is the product path (§6).
|
||||
|
||||
Bench-poke facts (measured, retained for the record even though the counter conclusion is dead):
|
||||
|
||||
- `TYPE_ALL` writable but inert for raw frames; `TSIP_UT_EN` will not hold — the latch is PTP-only silicon, not a hidden all-packet path.
|
||||
- RQSMR maps queue→counter 1:1; QPRC/QBRC exact and read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`).
|
||||
- Single PTP latch services ~383 k identifiable stamps/s; stream seq lands in `RXSATRH[31:16]` byte-swapped.
|
||||
|
||||
### Probes on the 82599
|
||||
|
||||
- The shared-PHC assumption dies, the method doesn't: each port has its own free-running SYSTIME (~6.4 ns granularity at 10G), but both share one crystal — relative drift ≈ 0, and a two-way exchange over the same cable cancels the unknown offset exactly, PTP-peer-delay style. `probe.go`'s existing both-direction averaging already provides this — but the two SYSTIMEs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first.
|
||||
- Probe frames must be 1588 L2 event frames (0x88F7): only those latch, so probing coexists with line-rate 0x88b5 traffic (0x88F7 recognition is ETQF index 3, reserved in §1). 6.4 ns quantization ≈ 1.3 m of round trip — average many probes.
|
||||
|
||||
### The decisive unknown is the module PHYs — testable before anything is built
|
||||
|
||||
The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the timestamp length method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches — probes must be PTP-shaped there too): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. ≲ 10–20 ns → viable; much more → close §3's timestamp path permanently.
|
||||
|
||||
The bench pokes (measured facts above) were run through the patched driver's debugfs — `reg_ops` returns read values, and `ts_bench` runs the latch poll/re-arm loop in-kernel because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM` (drivers: `~/work/phydiag-work/x520poke`, `ts_bench` in `~/work/ixgbe-sff/ixgbe_debugfs.c`). What they establish for the 82599: no all-packet exact-timestamp path exists, so the only surviving use of the latch is the low-volume **length probes** (below), where sparse PTP-shaped anchors are exactly the model — not the line-rate 0x88b5 measurement.
|
||||
|
||||
## 3. Cable-length strategy
|
||||
|
||||
**Status: consolidated — PHY features are the product path, NIC timestamps the fallback. The BCM path works.**
|
||||
|
||||
| Path | Status |
|
||||
|---|---|
|
||||
| BCM ECD | **Working on the bench** — recipe recovered from the OpenBCM SDK (`phy8481.c` cable-diag, same register family as the 84891L's command handler) and validated: per-pair verdicts + per-pair lengths in meters on a plugged healthy cable. Calibration and run-disturbance characterization remain (modules/fs/) |
|
||||
| Aquantia DSP/TDR (oracle only) | **Fully documented**: `1E.C884` length ±1 m, per-pair TDR verdicts and reflection distances (modules/fibergaga/) |
|
||||
| Marvell VCT (Wiiteks) | Undocumented and gated behind the brick trap; sacrificial-unit-only single-shot templates (modules/wiitek/). The BCM ECD result layout (verdict nibbles + per-pair length registers) is a fresh analogy for future targeted probes |
|
||||
| NIC timestamp path-delay | Module-independent; gated on the §2 retrain-stability experiment |
|
||||
|
||||
- **Decision:** product path = the PHY's own length machinery (BCM ECD, `1E.C884` on the oracle); NIC timestamp path-delay = fallback, pursued only if the §2 experiment passes. The ECD run blips the link, so length measurement is a between-runs operation, not a during-run one, unless characterization says otherwise.
|
||||
|
||||
## 4. Two-master I2C safety and the once-untested client assumptions
|
||||
|
||||
**Status: verified against source and hardware; the hardening item is built and proven.**
|
||||
|
||||
- **Per-op semaphore bracketing is real and is the weak point.** Each debugfs `w`/`r` acquires/releases swfw individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`); clients compose protocols from separate ops with userspace sleeps. Every multi-op sequence has windows where the driver can run its own I2C.
|
||||
- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask` — no mid-transaction bus corruption. But driver traffic is event-driven (SFP identify after module/link events, link setup, `ethtool -m`) — it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but correlated with the interesting moments.
|
||||
- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the risk is the other master *moving* the pointer — an interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence.
|
||||
- **The BCM 0x56 bridge's pending read data** (0x56 7-bit = 0xAC/0xAD write/read) across the >1 ms window was the one interleaving case without proof; the compound `x` op removes the window rather than reasoning about it, so the case is now closed by construction.
|
||||
- **The compound op is built and proven.** `sff_i2c` gains an `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`, `ixgbe_phy.c`): write, STOP, in-kernel delay, read — all under a single swfw hold, closing the interleave window entirely. The write/read primitives were refactored into locked/unlocked halves so `x` reuses them under one acquire. Validated on the FS BCM (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004). A ~3 ms hold is nothing — the driver holds swfw longer during SFP identify. The delay sweep corroborates the not-ready hazard the op closes: at delay 0 the bridged read returns stale data (0x0004 left over from the prior read), at ≥200 µs it reads correctly — precisely the stale-pointer/stale-data corruption an interleaved driver access would cause.
|
||||
- **RollBall split transactions: field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules.
|
||||
- **The >1 ms BCM delay**: validated on hardware at 3 ms. Treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.
|
||||
|
||||
## 5. Pre-FEC error visibility
|
||||
|
||||
**Status: answered from documentation — `3.E820` is post-FEC; the pre-FEC counters are the corrected-by-iteration histogram next door.** Register map: modules/fibergaga/. The correlation experiment is now verification, not classification.
|
||||
|
||||
Design points:
|
||||
|
||||
- Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic.
|
||||
- Run at high line utilization: an uncorrected LDPC frame landing in idle/IFG leaves no host-visible trace; low-load runs undercount the host side.
|
||||
- Expected ordering as margin falls: corrected-iteration counters shift toward higher N first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1.
|
||||
- Log fast retrains alongside (IEEE 1.147) to separate FEC exhaustion from retrain hiccups.
|
||||
- Clear-on-read discipline: exactly one reader during a run.
|
||||
- Scope: documents the Aquantia oracle — fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out until documented.
|
||||
|
||||
## 6. Architecture
|
||||
|
||||
**Status: resolved twice over — co-location is forced, and the single card is the ConnectX-5.**
|
||||
|
||||
- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation (§5), and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
|
||||
- **The single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (§2) and raw-L2 steering is not (§1) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only; txgbe is also PTP-only (verified in upstream `wx_ptp.c`); **ConnectX-4/5 (mlx5) is the only candidate meeting the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (`probe.go`'s assumption holds), native ETHER_FLOW ntuple steering, mature driver.
|
||||
- **Open on the ConnectX**: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only use 0x50/0x51) and whether the BCM SMI read data phase (raw 2-byte read, no offset write) can be framed. RollBall (0x51-resident, offset-model) fits.
|
||||
- **If MCIA can't reach the BCM**: product diagnostics via the Marvell/Aquantia modules only, with the X520 kept as an offline BCM rig — its bit-banged framing stays the only *certain* SMI transport. The X520 is **not** a fallback product NIC: it has no all-packet exact-timestamp path (§2), and a cable tester's rate buckets require exactly that.
|
||||
No confirmed-safe path exists (every candidate lands in the µC danger window). The open decision is whether the capability is worth the NDA route or a sacrificial unit — the product doesn't need it for length ([modules/wiitek/](modules/wiitek/README.md), [modules/README.md](modules/README.md)).
|
||||
|
||||
+7
-8
@@ -2,7 +2,7 @@
|
||||
|
||||
## Committed tree
|
||||
|
||||
AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — open-questions.md §2 for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (`probe.go`); framebuffer UI; harness.
|
||||
AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — nics/README.md for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (`probe.go`); framebuffer UI; harness.
|
||||
|
||||
## Stashes
|
||||
|
||||
@@ -27,20 +27,20 @@ AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet
|
||||
| Item | Status | Notes |
|
||||
|---|---|---|
|
||||
| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Port 0 `enp1s0f0` = new Wiitek (SN WAMZ012606X039U); port 1 `enp1s0f1` = FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) |
|
||||
| ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (open-questions.md §6). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) |
|
||||
| ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (nics/connectx-5/). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) |
|
||||
| Replacement Wiiteks | Arrived; one in X520 port 0 | Originals bricked by register exploration — modules/wiitek/ trap first |
|
||||
| FS SFP-10G-T-100 ×2 | In hand; one in X520 port 1 | BCM84891L, documented, robust |
|
||||
| Fibergaga SFP-10G-T-30M | In hand | Aquantia, RollBall, the documented oracle |
|
||||
| 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
|
||||
| E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
|
||||
|
||||
**The X520 is disqualified as the product NIC — measurement, not a narrow gap.** Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, open-questions.md §2). Per-queue counters were explored and **rejected** — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.
|
||||
**The X520 is disqualified as the product NIC — measurement, not a narrow gap.** Rate bucketing requires all-packet exact hardware RX timestamps (a cable tester measures the receive when it is misbehaving, so per-frame arrival times are the requirement; aggregate counters/rates are honest only in steady state and see nothing). The 82599 has one serial PTP-only latch and no all-packet path — TYPE_ALL is inert and TSIP does not exist (confirmed on hardware, nics/x520/). Per-queue counters were explored and **rejected** — a rate is not timestamps. The X520 is a diagnostics / BCM-framing card only; the ConnectX-5 is the product NIC.
|
||||
|
||||
## X520 diagnostics path — validated on hardware
|
||||
|
||||
In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`:
|
||||
|
||||
- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w <addr8> <bytes…>` / `r <addr8> <n>` / `x <waddr> <raddr> <delay_us> <n> <wbytes…>` single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: **ETQF steering** (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in `ixgbe_ethtool.c`, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); **`reg_ops` read-returns-value** and a **`ts_bench`** debugfs command (in-kernel latch poll/re-arm for the §2 pokes). `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`.
|
||||
- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w <addr8> <bytes…>` / `r <addr8> <n>` / `x <waddr> <raddr> <delay_us> <n> <wbytes…>` single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: **ETQF steering** (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in `ixgbe_ethtool.c`, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); **`reg_ops` read-returns-value** and a **`ts_bench`** debugfs command (in-kernel latch poll/re-arm for the timestamp bench pokes). `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`.
|
||||
- **Bench tools** (`~/work/phydiag-work/`): `etqfbench/` (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), `x520poke/` (Go; drives `reg_ops`/`ts_bench` for the timestamp/counter pokes), `compound_test.py` (proves the `x` compound op against the FS BCM).
|
||||
- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via `CMD_GET_SNR = 0x8030` invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
|
||||
- **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
|
||||
@@ -52,11 +52,11 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
|
||||
2. ~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source.
|
||||
3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
|
||||
4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).
|
||||
5. ~~RX steering: program/verify the ETQF path (open-questions.md §1).~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (`~/work/phydiag-work/etqfbench`). `system.go` runs unchanged.
|
||||
5. ~~RX steering: program/verify the ETQF path.~~ Done — the patched driver maps ETHER_FLOW onto ETQF slots; 7 streams → 7 queues proven on hardware (`~/work/phydiag-work/etqfbench`). `system.go` runs unchanged.
|
||||
|
||||
## RX flow-steering on ixgbe
|
||||
|
||||
**Solved on hardware.** cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (`ixgbe_ethtool.c`), so `system.go` runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (`~/work/phydiag-work/etqfbench`). Full analysis in open-questions.md §1.
|
||||
**Solved on hardware.** cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (`ixgbe_ethtool.c`), so `system.go` runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (`~/work/phydiag-work/etqfbench`). Full analysis in nics/x520/.
|
||||
|
||||
- **ETQF/ETQS L2 EtherType queue filters — proven.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
|
||||
- **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
|
||||
@@ -65,5 +65,4 @@ In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `ker
|
||||
## Open items
|
||||
|
||||
- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. Remaining work is characterizing the link blip the run causes (length is a between-runs operation until then). The FS ECD-chapter ask is now confirmation, not unblocking.
|
||||
- **Pre-FEC verification** on the Aquantia — counters documented (modules/fibergaga/); needs the graded-noise correlation run (open-questions.md §5).
|
||||
- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, short-cable calibration); scoped but unbuilt — the committed `probe.go` is the *filter-all* variant (raw-frame probes, needs all-packet rx stamping, E810-only); the X710/X520 variant means PTP-shaped probes. Superseded for the product if PHY DSP length pans out.
|
||||
- **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).
|
||||
|
||||
Reference in New Issue
Block a user