diff --git a/docs/README.md b/docs/README.md index f4a3e13..abc3abb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,12 +2,15 @@ Project knowledge that moves and versions with the repo. -- [goals.md](goals.md) — what cabletest is for and what it must output. -- [state.md](state.md) — where things are now: committed vs stashed, hardware in flight, bringup plan, open items. -- [hardware.md](hardware.md) — the box, interfaces, host tuning, how to run. -- [transports.md](transports.md) — how to reach module-PHY diagnostics, why the NIC choice governs it, and the register-level protocols. -- [modules.md](modules.md) — per-module diagnostics, register maps, cable-length physics, and the Marvell brick hazard. -- [measurement.md](measurement.md) — how to measure correctly; performance results and dead ends. -- [open-questions.md](open-questions.md) — genuinely open problems worth deeper thinking before the X520 hardware arrives. +General: -Vendor PDFs (BCM84891L transport + command handler, Marvell 88E151x datasheet) live alongside these. +- [goals.md](goals.md) — what cabletest is for and what it must output. +- [state.md](state.md) — where things are now: committed vs stashed, the box and what's in it, bringup progress, open items. **All current-placement state lives here and only here.** +- [hardware.md](hardware.md) — host rules: interface conventions, tuning, how to run. +- [measurement.md](measurement.md) — how to measure correctly; performance results and dead ends. +- [open-questions.md](open-questions.md) — the open problems, with their current answers. + +Per-device (each directory: device doc + its datasheets): + +- [nics/](nics/README.md) — the transport rule, the comparison table, and one directory per NIC: [x520](nics/x520/README.md), [connectx-5](nics/connectx-5/README.md), [x710](nics/x710/README.md), [e810](nics/e810/README.md), [wx1820](nics/wx1820/README.md). +- [modules/](modules/README.md) — module inventory, media-lies table, the shared IEEE register set, RollBall protocol, cable-length physics, and one directory per module: [fs](modules/fs/README.md), [wiitek](modules/wiitek/README.md), [fibergaga](modules/fibergaga/README.md). diff --git a/docs/goals.md b/docs/goals.md index 14fce56..f064336 100644 --- a/docs/goals.md +++ b/docs/goals.md @@ -2,15 +2,22 @@ cabletest finds Ethernet cables that misbehave under sustained full-duplex 10 Gbit load. It exchanges raw Ethernet frames (no IP, no ARP) across a variety of sizes and payloads between two directly-cabled ports on one host, and attributes every problem it can to the cable rather than the host. It runs as PID 1 on a dedicated appliance with a framebuffer UI; development happens on the same machine under Arch. -The target of measurement is the cable, not throughput. Line rate is a means to stress the physical layer. The outputs that matter, roughly in order: +The target of measurement is the cable, not throughput. Line rate is a means to stress the physical layer. + +## Outputs, in priority order 1. **Loss and error attribution** — reception gaps, link errors, NIC/driver counters as first-class output alongside application loss. Baseline loss must be exactly zero before a run counts; any host-side loss masks real cable faults. 2. **Noise tolerance** — a deliberately-bad "noise" cable intertwined with the test cable, driven by link up/down cycling, stresses the cable under test with alien crosstalk. -3. **Per-pair SNR** from the module PHYs — the leading indicator of a marginal cable before it drops frames. IEEE 802.3an standard registers on the Marvells; the BCM leaves those unpopulated and reports through its vendor command handler instead (transports.md). +3. **Per-pair SNR** from the module PHYs — the leading indicator of a marginal cable before it drops frames. IEEE 802.3an standard registers on the Marvells; the BCM leaves those unpopulated and reports through its vendor command handler instead (modules/fs/). 4. **Cable length** — sanity check and fault localization. Sources, in preference order: - Module PHY DSP estimate — works on a linked cable. - PHY TDR — localizes opens/shorts both-ended; healthy-cable length only single-ended. - - NIC timestamp path-delay — works linked, module-independent; needs per-packet or PTP-latch hardware timestamps and a short-cable calibration. -5. **Pre-FEC error visibility** — corrected-error counters that move before post-FEC loss appears. Vendor-specific; still unlocated on the module PHYs we can talk to. Standard latched PCS counters (errored blocks, BER, block-lock loss) are the working proxy under noise stress. + - NIC timestamp path-delay — works linked, module-independent; needs hardware timestamps and a short-cable calibration. +5. **Pre-FEC error visibility** — corrected-error counters that move before post-FEC loss appears. Vendor-specific; located on the Aquantia (open-questions.md §5), unlocated elsewhere. Standard latched PCS counters (errored blocks, BER, block-lock loss) are the working proxy under noise stress. -Design preferences that shaped the tool: plain sockets first, escalate only after measuring (AF_PACKET reached line rate on the shipping mix; the AF_XDP experiment is parked); pre-populated frame contents; goroutine-heavy is fine (many cores to burn); DPDK is off the table — it would kill the sysfs counters and link management the tester depends on. +## Design preferences that shaped the tool + +- Plain sockets first, escalate only after measuring: AF_PACKET reached line rate on the shipping mix; the AF_XDP experiment is parked. +- Pre-populated frame contents. +- Goroutine-heavy is fine — many cores to burn. +- DPDK is off the table: it would kill the sysfs counters and link management the tester depends on. diff --git a/docs/hardware.md b/docs/hardware.md index acec9c5..945b40d 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -1,42 +1,37 @@ -# Hardware and host +# Host rules -## The box -Single usable PCIe slot (Gen4 x8) holding the X520-DA2, which trains at its own Gen2 ceiling (5 GT/s ×8, 32 Gb/s); the ConnectX-5 replaces it when it arrives. The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome. +Box contents, card placement, and interface assignments live in [state.md](state.md). This file carries the rules that stay true regardless of what's plugged in. ## Interfaces -- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order — the X710 pair has already been enp4s0f\* and enp3s0f\* across reboots). Currently the X520 pair `enp1s0f0` (new Wiitek) / `enp1s0f1` (FS), ixgbe. -- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. The X710 pair `enp3s0f0np0` / `enp3s0f1np1` (i40e). -- **`enp88s0` (igc)**: this box's LAN uplink with the default route (its sibling port `enp89s0` is dark). Never repurpose or down it; identify by default route, not name. -- Both test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4. -## The media lies (mostly) -**Every test module is a copper RJ45 10GBASE-T module** — none are fiber, whatever the EEPROM claims; the real media is a 10GBASE-T PHY inside each module. But EEPROM honesty varies by vendor, with driver consequences: -- **The Wiitek lies**: LC connector, 850 nm SR, multimode lengths, fake optical DOM (the temperature is real, the "laser" powers are theater). The clone exists to pass NIC compatibility checks — and it works: stock ixgbe qualifies it without complaint. -- **The FS SFP-10G-T-100 is honest**: connector RJ45 (0x22), extended transceiver code 10GBASE-T Short Reach, 100 m copper, no fake DOM. Stock ixgbe rejects exactly that honesty as an unqualified type and **fails the whole port's probe** (error -95, no netdev at all) — the driver must load with `allow_unsupported_sfp=1`, which is therefore mandatory on the X520, and it's the truthful module that requires it. -- The 10Gtek claims `SFP-10G-SR`; a module whose part number reads SR can still be 10GBASE-T copper. +- **Test pair**: the two ports whose modules hold the cable under test. Find by driver name, never by ethN — names shift with kernel link order. +- **Noise pair**: driven bad on purpose, link up/down cycled, intertwined with the test cable to inject alien crosstalk. +- **LAN uplink**: carries the default route. **Never repurpose or down it**; identify by default route, not name. +- Test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4. -Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE. +Module identities, EEPROM honesty (and why the X520 needs `allow_unsupported_sfp=1`), and the physical-layer model live in [modules/](modules/README.md). ## Host tuning (resets every reboot; re-apply before trusting results) + Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults. -1. **CPU governor → `performance`** on all cores. - - Biggest single contributor. - - Under powersave, cores idle at 400 MHz and ramp too slowly, dropping frames at startup. -2. **Disable adaptive coalescing.** - - `ethtool -C adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. - - Syntax/values differ on ixgbe — re-derive for the X520. -3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair). - - Effect was not isolated in testing; don't claim it as a proven win. -4. **Socket buffers** need no sysctl change. - - The tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) to bypass the 4 MB `rmem_max` clamp. +1. **CPU governor → `performance`** on all cores. Biggest single contributor: under powersave, cores idle at 400 MHz, ramp too slowly, and drop frames at startup. +2. **Disable adaptive coalescing**: `ethtool -C adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC. +3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair). Effect was not isolated in testing; don't claim it as a proven win. +4. **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp. Verify a clean run shows `tx frames == rx frames` exactly before a run counts. ## irdma (dev host only) -`irdma` autoloads and binds the ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active"), which fails the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first. + +`irdma` autoloads and binds ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active") and failing the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first. ## Running -Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a enp1s0f0np0 -b enp1s0f1np1`. It starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1) and do not follow every run with a pgrep orphan check — the group kill is the guarantee. The tell for two competing instances (an orphan, or the user's own hand-run) is `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs — ask rather than killing something that might be theirs. `./harness` also dumps the panel to `shots/.png` for remote assessment. -Test runs use the **default config only** — no `-sizes`/`-streams` overrides, and no editing the hardcoded `frameSizes`/`numStreams`/`batchSize` (that's the same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case. +Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a -b `. + +- The harness starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1); do not follow runs with pgrep orphan checks — the group kill is the guarantee. +- It dumps the panel to `shots/.png` for remote assessment. +- The tell for two competing instances (an orphan, or the user's own hand-run): `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs. Ask rather than killing something that might be theirs. +- **Default config only** — no `-sizes`/`-streams` overrides, no editing the hardcoded `frameSizes`/`numStreams`/`batchSize` (same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case. +- The first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) — not residual error. diff --git a/docs/measurement.md b/docs/measurement.md index a606223..63afc2d 100644 --- a/docs/measurement.md +++ b/docs/measurement.md @@ -3,27 +3,50 @@ Hard-won rules about measuring correctly. Violating these produces numbers that look plausible and are wrong. ## Rate buckets: stamp at read time, never at ticker-fire time -`direction.capture()` stamps each bucket with `time.Now()` taken immediately after reading the counters — never a timestamp passed in from the sampler ticker. A rate is Δcounters/Δt; the sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator, and because E[1/x] > 1/E[x] the error biases the rate **upward**, it doesn't cancel. Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high; read-time stamping read within 2%. A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it. Never share one timestamp across directions "so buckets share an instant" — nothing needs it and it reintroduces the skew. + +- `direction.capture()` stamps each bucket with `time.Now()` taken immediately after reading the counters — never a timestamp passed in from the sampler ticker. +- Why: a rate is Δcounters/Δt. The sampler runs a variable delay after its tick, so a tick-time label pairs a right numerator with a wrong denominator — and because E[1/x] > 1/E[x], the error biases the rate **upward**; it doesn't cancel. +- Measured: tick-time stamping read 27.55 G / 4.88 Mpps against a true 19.86 G — 35% high. Read-time stamping read within 2%. +- A 1 s console interval hides it (jitter < 0.2%); a 16 ms panel window exposes it. +- Never share one timestamp across directions "so buckets share an instant" — nothing needs it, and it reintroduces the skew. + +This rule covers the **NIC-counter** buckets. The per-stream application buckets are a separate system stamped by per-packet MAC RX timestamps (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check); read-time stamping is not a substitute for those — software stamping was tried and cannot reach the needed precision. NICs without all-packet RX timestamping (X710, 82599) fail the host check; see open-questions.md §2. ## NIC counters are not monotonic -`/sys/class/net/*/statistics/*` run from boot, not process start, and reset to zero on driver stats resets. In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run, and a backward step underflows `now - base` to ~2^64. Accumulate only forward motion into a process-local total, then take every baseline from that total — never from a raw reading. Establish all baselines in one place before traffic starts. Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug. + +- `/sys/class/net/*/statistics/*` run from boot, not process start, and reset to zero on driver stats resets. +- In unsigned arithmetic that bites twice: a zero baseline charges the machine's whole lifetime to the run; a backward step underflows `now - base` to ~2^64. +- Rule: accumulate only forward motion into a process-local total, then take every baseline from that total — never from a raw reading. Establish all baselines in one place before traffic starts. +- Debugging tell: when a panel's background disagrees with rows drawn from the same value, suspect two vintages of one counter before a color bug. ## Only zero-baseline-loss runs count; small frames measure the host -Any nonzero baseline loss masks real cable faults, so a run counts only when `tx frames == rx frames` exactly with a clean cable. Small-frame runs are CPU/host-bound, not cable-bound, so cable conclusions drawn from them are false. Keep to the default config (see hardware.md) and judge from steady state — the first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) that can read far below line rate and is not residual error. + +- A run counts only when `tx frames == rx frames` exactly with a clean cable — any nonzero baseline loss masks real cable faults. +- Small-frame runs are CPU/host-bound, not cable-bound; cable conclusions drawn from them are false. +- Keep to the default config (hardware.md) and judge from steady state — the first ~5 s is settling transient, not residual error. ## Measured performance (AF_PACKET committed path, ice/E810 era) -- Full size mix at 7 flow-director streams: line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup. -- 64 B only: pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops), so only ~3.8 Gb/s. -- RX cannot be parallelized by RSS (hardware RSS on ice can't hash raw ethertypes) — Flow Director steering by ethertype to distinct queues is what gives multiple NAPI contexts. rxnfc/fdir programming has sharp edges; see the rxnfc notes below. -- Dead ends, measured and not to be re-attempted without new hardware (the binding constraint is total CPU across ~28 goroutines on 20 threads): - - **Splitting tx senders from rx streams** — raises tx but collapses rx, since rx scales with queue count, capped at 7 on that NIC. - - **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at tx. - - **Batch sizes above 64** — no gain, worse loss. -- The tx `sendmmsg` busy-spin on ENOBUFS is not worth chasing: it only exists when the tx ring is full, which means the wire is the ceiling, so recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable). -The AF_XDP experiment (stashed) removed the tx frame-generation ceiling (64 B tx ~14.5 Mpps/dir) and moved the bottleneck to RX (~4 Mpps/dir on AF_PACKET, ~13.5 with AF_XDP RX). It required per-packet MAC rx timestamps for honest buckets, delivered via an XDP-metadata kfunc — available on the E810 datapath, not the X710. With the test path off the E810 this whole path is parked. Note the committed AF_PACKET path *also* buckets by per-packet MAC rx stamps — via the `SO_TIMESTAMPING` cmsg, with `rx_filter=ALL` enforced as a hard host check — so the stash's novelty was the delivery mechanism and throughput, not the use of hardware stamps. Read-time stamping is how the *NIC-counter* buckets are labeled (the lesson above), not a substitute for the per-frame stamps; NICs without all-packet rx timestamping (X710, 82599) fail the host check and need a bucketing fallback — see open-questions.md §2. +| Config | Result | +|---|---| +| Full size mix, 7 flow-director streams | Line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup | +| 64 B only | pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s | +| AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX | -## rxnfc / Flow Director programming (ice), three traps that each cost a debugging round -1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement, so trusting its display gives an inverted rule that silently matches nothing. -2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192) on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted. +- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts. +- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path gets the same stamps via the `SO_TIMESTAMPING` cmsg). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps. + +## Dead ends — measured, do not re-attempt without new hardware + +The binding constraint was total CPU across ~28 goroutines on 20 threads. + +- **Splitting TX senders from RX streams** — raises TX but collapses RX (RX scales with queue count, capped at 7 on that NIC). +- **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at TX. +- **Batch sizes above 64** — no gain, worse loss. +- **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable). + +## rxnfc / Flow Director programming (ice) — three traps + +1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement — trusting its display gives an inverted rule that silently matches nothing. +2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192), on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted. 3. **ice rejects `RX_CLS_LOC_ANY`** with ENOSPC — allocate a free location yourself (capacity from `ETHTOOL_GRXCLSRLCNT`'s `data`; the CLI allocates downward from the top). When a rule inserts but steers nothing, diff the raw bytes of a CLI-made known-good rule against yours. diff --git a/docs/modules.md b/docs/modules.md deleted file mode 100644 index 0dc769e..0000000 --- a/docs/modules.md +++ /dev/null @@ -1,75 +0,0 @@ -# Module-PHY diagnostics and the register maps - -## DANGER: Marvell CUX3610 (Wiitek UF-RJ45-10G-100) modules brick on certain reads -Both original Wiiteks were bricked **permanently** by register exploration — the death survives cold power cycles and reseats. Read this before touching any replacement. - -The CUX3610 has an integrated microcontroller (Marvell Alaska-M, PHY ID 0x002B0BF4, firmware 12.1.5.0) that boots from onboard non-volatile storage. The high vendor register windows (3.0x8000+, 31.0xF000+) reached over RollBall are **µC-mapped peripheral space, not PHY-core MDIO**. A read of a trapped peripheral crashes the µC mid-operation and corrupts its persistent boot state, so it comes up FATAL forever after. - -**The trap: any register whose low byte is 0x64 in a high window (register high-byte ≥ 0x80), on any devad.** Module 1 died reading 3.0x8064, module 2 reading 31.0xF064. The mailbox writes the address bytes, then the µC dies executing the access — the command never even needs to complete. 3.0x8064 is a known QCA808x CDT (cable-diagnostic) status register; the ODM firmware appears to emulate/intercept a CDT block there and crash on a cold read. A sloppy `(reg & 0xFF) == 0x64` decode explains both deaths with one bug. - -Rules for any Marvell/RollBall module: -- **Never sweep vendor register windows. Never read `*.0x??64` (high-byte ≥ 0x80).** -- Single targeted reads only, on registers with a documented reason. -- Recovery after a brick needs a vendor firmware image + the boot-download protocol (registers 1.0xC050 boot status, 1.0xD0F0–D0F3 download) — we have no CUX3610 image, so bricks are effectively dead. A ghost i2c device at 7-bit 0x57 (returns zeros) is the ROM bootloader waiting for a download. - -## What the Marvell modules give (safely) -Proven on the CUX3610 before they died, and on the Fibergaga oracle — all IEEE 802.3an standard registers, safe on any Marvell module: -- **Per-pair SNR margin** — PMA 1.133–1.136 current, 1.137–1.140 minimum. - - On the CUX3610 the value is `reg − 0x8000` in 0.1 dB (saw A 9.1 / B 7.1 / C 7.7 / D 8.9 dB). - - Retrain-dependent — only compare within one training session. -- **Latched error/retrain detection** — PCS 3.32/3.33. - - Block-lock loss, BER saturate, errored blocks; clear-on-read. -- **Link-partner / master-slave** — 7.33. -- **FW version** 1.0xC011/12; **boot status** 1.0xC050. -- **Forcing a retrain** (to refresh SNR minimums): - - RollBall write `7.0 |= bit9` (restart AN) — proven safe. - - An `ip link` bounce does not drop the copper line, so it won't retrain. - -Not available in safe space: pre-FEC counter, cable length. Those live in the µC vendor space that killed the modules. - -## What the FS/BCM gives -All via the SMI bridge + command handler (transports.md carries the codes and the invocation traps). Proven on the X520: -- **Per-pair SNR** — CMD_GET_SNR 0x8030 invoked bare: DATA2–5 = absolute receiver SNR in 0.1 dB (bench ≈ 27–32 dB; the 10GBASE-T operating point is ≈ 26.5 dB, so margin ≈ value − 26.5). The IEEE registers 1.133–1.140 stay at 0x8080 — handler only. -- **Die temperature** (0x8031) and **supply rails** (0x802F) — module health. -- **Standard PCS 3.32/3.33** — block lock, latched errored-block/BER counters, clear-on-read: the noise-stress error proxy. -- **Retrain** — IEEE AN restart `7.0 |= bit9`, relinks in ~6 s. - -Probed on the bench unit (full command transcription: `bcm84891l-mdio-commands.md`); all GETs PASS; observed states: -- **EEE / AutogrEEEn — force off at bringup, and trust the wire over the handler.** `GET_EEE_MODE` (0x8008) once returned DATA1 = 0x0047 — decoding as 10G AutogrEEEn *variable latency* plus 5G/1G native EEE, Broadcom's local-only power saving that never shows in the IEEE advertisement registers (7.60/7.61 read 0 throughout) — then read 0x0000 later with no SET in between: either live-then-cleared or an unreliable return (caveat below). Mitigation applied and verified: `SET_EEE_MODE` (0x8009) with explicit all-off params `(0, 0, 0x7A12, 0x480, 0)` + AN restart → mode reads 0, link and SNR healthy. cabletest should apply this defensively at startup. Wire-truth check: arm `SET_EEE_STATISTICS` (0x801A) after link-up, read `GET_EEE_STATISTICS` (0x801B) — zero LPI events/time observed on the idle link; repeat under traffic. -- **Fast retrain: enabled** on 10G/5G/2.5G (0x800A → 1/1/1); IEEE 1.147 = 0x0019, retrain count bits zero since link-up. Keep it enabled and treat the 1.147 count as a per-run quality metric — a marginal cable that fast-retrains instead of dropping still shows up in the count. -- **Pair map: identity** (`GET_PAIR_SWAP` 0x8000 → DATA2 = 0x00E4 = A/B/C/D straight through) — MDI wiring verification works. -- **Faults pass through** (`GET_SUB_LF_RF_STATUS` 0x800D → 0; SET/CLEAR at 0x8011/0x8010 could mask them as IDLE — leave off). -- **Host interface clean**: XFI polarity normal, TX FIR = main tap only (0x3C), KR/SyncE/USXGMII/WOL all disabled, MAC/PHY frequency locked, jumbo enabled at 9K, temp warnings off. -- **1588 disabled** (0x8004 → 0); **limited-reach mode disabled** (0x8006 → 0), so its DATA3 linked-in-LR bit — the only handler-visible trace of the DSP length estimate — is unavailable without a config change. -- Pause-frame mode (0x801F/0x8020) is scoped to 2.5/5G idle-stuffing under 10G XFI WAP mode — inert in plain 10G operation. - -**Firmware-reliability caveat.** Several documented DATA1 returns on this ODM firmware are untrustworthy: die-temperature-like values (0x43/0x44/0x46/0x47) appear in DATA1 of commands that should return modes, and repeat reads of the same GET disagree. Corroborate anything load-bearing through IEEE registers (7.60/7.61 for EEE advertisement) or wire behavior (EEE statistics under traffic), and write every DATA register explicitly before any SET (the handler never clears them). - -**No cable length.** The handler catalog is complete (§1.25.1.1–45: pair swap through WOL) and contains no ECD, length, or skew command. Cable length, opens/shorts, pair skew, and polarity live in the separate ECD register mechanism whose chapter FS hasn't sent. The 1588 engine is the same story: the PHY is IEEE 1588-2008 v2-compliant with a one-command enable (0x8004/0x8005) but its operation registers are undocumented — in-PHY timestamping would measure path delay at the MDI, taking PHY-pipeline latency out of the length equation entirely (open-questions.md §2). Until either chapter lands, FS-side length comes only from the NIC timestamp path — in contrast to the Aquantia oracle, whose `1E.C884` length register is fully documented. - -## Marvell cable-length register candidates (unverified, single-shot only) -From kernel drivers for sibling parts — templates for a *future* single targeted probe on a sacrificial module, never a sweep, never near `*.0x??64`: - -- **TDR** from the 88Q2220 (`marvell-88q2xxx.c`, same 0x002b0bXX family, recipe cited from "the Marvell API"), all in MMD 3: - - TDR_RESET 0xFECA — init 0x0D90, clear 0x1D90. - - Calibration: 0xFED9 = 0x010E, 0xFEDA = 0x00EB, 0xFEDE = 0x0058. - - TDR_STATUS 0xFEDD — write 0x0002 to start; ~500 ms test. - - Read TDR_STATUS: bits[1:0] = 01 done; [7:4] verdict (0x7 OK / 0xE open / 0x3 short / 0x5 noise); [15:8] distance in meters. - - Single-pair on the Q2220; the 4-pair CUX3610 likely has per-pair variants nearby. Low bytes are nowhere near the 0x64 trap. -- **QCA-style CDT** (`qcom/qcom.h`): - - 3.0x8064 CDT_STATUS, 3.0x8065–0x8068 per-pair results, 3.0x8074–0x807E thresholds. - - This IS the trap address — only ever entered via the full start sequence, never a cold status read; treat as off-limits absent strong evidence. -- **Next doc-mining lead** for the 4-pair pattern: `gtAdvVct.c` in the Marvell qd-dsdt GPL dumps (e.g. github.com/andy928/xpenology `arch/arm/plat-feroceon/mv_hal/qd-dsdt/`). - -## Cable-length physics (applies to both vendors) -- **TDR** (classic VCT/CDT/ECD): - - Locates opens/shorts both-ended. - - But *healthy-cable length* needs a far-end reflection, and a plugged-in terminated far end absorbs the pulse — so "OK" pairs report no distance. Effectively single-ended for length. - - Every kernel implementation only reports distance for *fault* results. -- **DSP estimate**: - - Both vendors' PHYs compute linked-cable length continuously from insertion loss during training (Marvell "proactively determines cable length"; the BCM's limited-reach power mode switches "based on cable length" while linked). - - Works in the product's both-ends-plugged topology — the open question is only whether the estimate is exposed in a readable register. -- **NIC timestamp path-delay** — the module-independent fallback if the DSP length isn't reachable: - - Works linked. - - Needs per-packet or PTP-latch hardware timestamps. - - Needs a short-cable calibration to cancel fixed PHY latency (~4.8 ns/m one way; 50 m ≈ 240 ns). diff --git a/docs/modules/README.md b/docs/modules/README.md new file mode 100644 index 0000000..d889e17 --- /dev/null +++ b/docs/modules/README.md @@ -0,0 +1,58 @@ +# Modules + +Per-module diagnostics: capabilities, register maps, hazards. Each device directory holds its doc plus its datasheets. NIC-side transports: [../nics/](../nics/README.md). + +> ⚠ **Before touching any Wiitek/Marvell module, read the brick trap in [wiitek/](wiitek/README.md).** Two modules died permanently from register exploration. + +## The modules + +Counts and current placement: [../state.md](../state.md). + +| Module | PHY | Doc | Role | +|---|---|---|---| +| FS SFP-10G-T-100 | Broadcom BCM84891L | [fs/](fs/README.md) | Test set; documented, robust | +| Wiitek UF-RJ45-10G-100 | Marvell CUX3610 (Alaska-M) | [wiitek/](wiitek/README.md) | Test set; **brick-prone** | +| Fibergaga SFP-10G-T-30M | Aquantia AQR | [fibergaga/](fibergaga/README.md) | The documented oracle | +| 10Gtek | (claims SFP-10G-SR) | — | Copper RJ45 despite the SR part number; filler, not in the test set | + +## The media lies (mostly) + +**Every test module is a copper RJ45 10GBASE-T module** — none are fiber, whatever the EEPROM claims; the real media is a 10GBASE-T PHY inside each module. EEPROM honesty varies by vendor, with driver consequences: + +| Module | EEPROM claims | Stock ixgbe verdict | +|---|---|---| +| Wiitek | LC connector, 850 nm SR, multimode lengths, fake optical DOM (temperature real, "laser" powers theater) | Passes qualification | +| FS | Honest: RJ45 (0x22), 10GBASE-T Short Reach, 100 m copper, no fake DOM | **Rejected** — kills the whole port probe (error -95, no netdev) | +| 10Gtek | `SFP-10G-SR` part number | (not in the test set) | + +- `allow_unsupported_sfp=1` is mandatory on the X520 — and it's the *truthful* module that requires it ([../nics/x520/](../nics/x520/README.md)). +- Physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — never an optical model. +- The module PHYs keep the copper link trained on their own: an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). + +## IEEE 802.3an standard registers (safe on all copper module PHYs) + +Proven on the CUX3610 (before the originals died) and the Fibergaga; the BCM notably does **not** populate the SNR set ([fs/](fs/README.md)). + +| Registers | What | Notes | +|---|---|---| +| PMA 1.133–1.136 / 1.137–1.140 | Per-pair SNR margin, current / minimum | CUX3610 value = `reg − 0x8000` in 0.1 dB. Retrain-dependent — only compare within one training session | +| PCS 3.32/3.33 | Block-lock loss, BER saturate, errored blocks | Latched, clear-on-read | +| AN 7.33 | Link-partner status / master-slave | | +| PMA 1.147 | Fast retrain status/count | Per-run quality metric | +| AN 7.0 bit 9 | Restart AN — forces retrain, refreshes SNR minimums | Proven safe everywhere. An `ip link` bounce does not drop the copper line, so it won't retrain | + +## RollBall protocol (Marvell and Aquantia modules) + +Clause-45 access to the internal PHY over I2C A2h, all messages ≤ 4 B, offset-addressed: + +1. Unlock: password 0xFF×4 at A2h 0x7B. +2. Page-select: A2h 0x7F → 3. +3. Mailbox: A2h 0x80 (cmd) / 0x81 (data), poll for DONE (0x04). + +Split transactions (separate STOP per byte) are field-proven on both the Fibergaga and the original Wiiteks; the kernel `mdio-i2c` combined form is not required by the modules. + +## Cable-length physics (applies to all vendors) + +- **TDR** (classic VCT/CDT/ECD): locates opens/shorts both-ended. But *healthy-cable length* needs a far-end reflection, and a plugged-in terminated far end absorbs the pulse — "OK" pairs report no distance. Effectively single-ended for length; every kernel implementation only reports distance for *fault* results. +- **DSP estimate**: both vendors' PHYs compute linked-cable length continuously from insertion loss during training (Marvell "proactively determines cable length"; the BCM's limited-reach power mode switches "based on cable length" while linked). Works in the product's both-ends-plugged topology — the question is only whether the estimate is exposed in a readable register (Aquantia: yes, `1E.C884`; BCM/Marvell: not in documented space). +- **NIC timestamp path-delay** — the module-independent fallback: works linked, needs hardware timestamps and a short-cable calibration to cancel fixed PHY latency (~4.8 ns/m one way; 50 m ≈ 240 ns). Viability gated on PHY-latency stability across retrains ([../open-questions.md](../open-questions.md) §2). diff --git a/docs/modules/fibergaga/README.md b/docs/modules/fibergaga/README.md new file mode 100644 index 0000000..cfc1bdd --- /dev/null +++ b/docs/modules/fibergaga/README.md @@ -0,0 +1,21 @@ +# Fibergaga SFP-10G-T-30M (Aquantia AQR) + +The documented oracle. RollBall module, PHY ID 0x31c31c13, proven over the i40e oracle path ([../../nics/x710/](../../nics/x710/README.md)). Its PHY family register reference is vendored in this directory (`aquantia-aqr107-aqr109-datasheet.pdf` — the AQR107-AQR109 register reference). + +## Documented registers + +| Registers | What | +|---|---| +| 3.E820/3.E821 | PCS Receive Vendor **Uncorrected** Frame Counter (post-FEC) | +| 3.E840–3.E84E | Corrected Frame N Iteration Counters, N = 1…10 LDPC iterations — the pre-FEC leading indicator ([../../open-questions.md](../../open-questions.md) §5) | +| 3.E816 | RS Corrected Error Frame Counter (system-side, secondary) | +| 1E.C884 bits 7:0 | **Estimated cable length in meters, ±1 m**, RO, from the cable diagnostic engine | +| 1E.C800 | Per-pair TDR fault verdicts (open/short/mismatch/cross-pair) | +| 1E.C801–C807 | Distances of the four worst reflections per pair, ±1 m | +| 1E.C880–C883 | Reflection impedance/magnitude | +| PMA 1.133–1.140, PCS 3.32/3.33 | Standard set ([../README.md](../README.md)), proven live | + +## Open bench questions + +- Whether `1E.C884` is continuously valid on a linked cable or only after an explicit diag run. +- The pre-FEC correlation experiment ([../../open-questions.md](../../open-questions.md) §5) — this module is its instrument. diff --git a/docs/aquantia-aqr107-aqr109-datasheet.pdf b/docs/modules/fibergaga/aquantia-aqr107-aqr109-datasheet.pdf similarity index 100% rename from docs/aquantia-aqr107-aqr109-datasheet.pdf rename to docs/modules/fibergaga/aquantia-aqr107-aqr109-datasheet.pdf diff --git a/docs/10GBase-T Coper SFP Registers Operation's Reference for BCM84891.pdf b/docs/modules/fs/10GBase-T Coper SFP Registers Operation's Reference for BCM84891.pdf similarity index 100% rename from docs/10GBase-T Coper SFP Registers Operation's Reference for BCM84891.pdf rename to docs/modules/fs/10GBase-T Coper SFP Registers Operation's Reference for BCM84891.pdf diff --git a/docs/BCM84891L-MDIO Command Descriptions-.pdf b/docs/modules/fs/BCM84891L-MDIO Command Descriptions-.pdf similarity index 100% rename from docs/BCM84891L-MDIO Command Descriptions-.pdf rename to docs/modules/fs/BCM84891L-MDIO Command Descriptions-.pdf diff --git a/docs/modules/fs/README.md b/docs/modules/fs/README.md new file mode 100644 index 0000000..78d10fc --- /dev/null +++ b/docs/modules/fs/README.md @@ -0,0 +1,62 @@ +# FS SFP-10G-T-100 (Broadcom BCM84891L) + +Documented, robust — survived the whole poking campaign un-bricked. Honest EEPROM (RJ45/10GBASE-T codes), which is what trips stock ixgbe qualification ([../README.md](../README.md)). + +References in this directory: [bcm84891l-mdio-commands.md](bcm84891l-mdio-commands.md) (command handler, transcribed), [bcm84891l-sfp-registers.md](bcm84891l-sfp-registers.md) (transport reference, transcribed), and their source PDFs. + +## SMI transport (I2C→MDIO bridge) + +PHY at I2C 8-bit **0xAC write / 0xAD read**. + +| Op | Frames | +|---|---| +| Clause-45 write | one I2C write to 0xAC: `[000+DevAD, RegH, RegL, DataH, DataL]` | +| Clause-45 read | I2C write to 0xAC: `[001+DevAD, RegH, RegL]` → **delay >1 ms** (3 ms proven) → I2C read 2 B from 0xAD | + +- Single-byte reads see the bridge as inert (returns 0) — the DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead. Treat 0x0000 reads as retry-with-longer-delay: 0 is also the bridge's not-ready signature. +- Recorded for a future multi-byte firmware transport (mlx5 MCIA shape): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`. + +## MDIO command handler + +Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-commands.md). Traps: + +- **SET commands execute stale DATA1** — the handler never clears DATA registers; write every parameter register explicitly before any SET. +- Poll STATUS ~100 ms; frozen up to 2 s during 10GBASE-T training — only run after link-up. + +## Proven diagnostics + +| Capability | How | Observed | +|---|---|---| +| Per-pair SNR | `CMD_GET_SNR` 0x8030, **invoked bare** (writing the documented DATA1 display flag returns zeros; IEEE 1.133–1.140 never populate — constant 0x8080) | DATA2–5 = SNR A–D ×0.1 dB; ≈ 27–32 dB absolute on bench, 0.1–0.4 dB jitter. 10GBASE-T operating point ≈ 26.5 dB, so margin ≈ value − 26.5 | +| Die temperature | `CMD_GET_CURRENT_TEMP` 0x8031 | ~68–70 °C on bench | +| Supply rails | `GET_CURRENT_VOLTAGE` 0x802F | 0.8 V and 1.88 V rails, tenths of mV | +| Error counters | IEEE PCS 3.32/3.33 — block lock, latched errored-block/BER, clear-on-read | The noise-stress error proxy | +| Retrain on demand | IEEE AN restart `7.0 \|= bit9` | Relinks in ~5–6 s | + +## Probed configuration states + +| Item | Command | State | +|---|---|---| +| EEE / AutogrEEEn | 0x8008/0x8009 | Once read 0x0047 (10G AutogrEEEn variable latency + 5G/1G native — local-only, invisible in IEEE 7.60/7.61, which read 0); later reads 0. **Forced all-off** via SET with explicit params `(0, 0, 0x7A12, 0x480, 0)` + AN restart, verified. cabletest should apply this defensively at bringup | +| EEE wire-truth | arm 0x801A after link-up, read 0x801B | Zero LPI events/duration on idle link; repeat under traffic | +| Fast retrain | 0x800A (datasheet titles it EMI_MODE; description is fast retrain) | Enabled 10G/5G/2.5G; IEEE 1.147 = 0x0019, count bits zero. Keep enabled; read the 1.147 count per run — a marginal cable that fast-retrains still gets counted | +| Pair map | 0x8000 | DATA2 = 0x00E4 = identity (A/B/C/D straight through) — MDI wiring verification works | +| Fault pass-through | 0x800D | 0 = LF/RF pass to the line (0x8011/0x8010 could mask them as IDLE — leave off) | +| Host interface | 0x802B/0x802D/0x8027 | XFI polarity normal; TX FIR = main tap 0x3C only; MAC/PHY frequency locked; KR/SyncE/USXGMII/WOL disabled; jumbo on at 9K; temp warnings off | +| 1588 | 0x8004 | Disabled (engine registers undocumented — see asks below) | +| Limited reach | 0x8006 | Disabled — its DATA3 "linked in LR mode" bit, the only handler-visible trace of the DSP length estimate, is unavailable without a config change | + +## Firmware-reliability caveat + +Several documented DATA1 returns on this ODM firmware are untrustworthy: die-temperature-like values (0x43/0x44/0x46/0x47) appear in DATA1 of commands that should return modes, and repeat reads of the same GET disagree. Corroborate anything load-bearing through IEEE registers (7.60/7.61 for EEE advertisement) or wire behavior (EEE statistics under traffic), and write every DATA register explicitly before any SET. + +## No cable length — and the asks to FS + +The handler catalog is complete (§1.25.1.1–45) and contains no ECD, length, or skew command. Cable length, opens/shorts, pair skew, and polarity live in the separate ECD register mechanism whose chapter FS hasn't sent; the 1588 engine is the same story (one-command enable, undocumented operation — in-PHY timestamping would measure path delay at the MDI, removing PHY-pipeline latency from the length equation; [../../open-questions.md](../../open-questions.md) §2). Until either chapter lands, FS-side length comes only from the NIC timestamp path. + +Asks, in value order: + +1. The **ECD chapter** — shorts/opens/cable length invocation. +2. The **1588 operation chapter**. +3. Datasheet **§1.20 loopback** (copper line loopback) and **§1.17 EEE/fast-retrain monitoring**. +4. **Chapter 2 register summary.** The excerpt's TOC names them all. diff --git a/docs/bcm84891l-mdio-commands.md b/docs/modules/fs/bcm84891l-mdio-commands.md similarity index 100% rename from docs/bcm84891l-mdio-commands.md rename to docs/modules/fs/bcm84891l-mdio-commands.md diff --git a/docs/bcm84891l-sfp-registers.md b/docs/modules/fs/bcm84891l-sfp-registers.md similarity index 96% rename from docs/bcm84891l-sfp-registers.md rename to docs/modules/fs/bcm84891l-sfp-registers.md index 157eb1d..3d589dc 100644 --- a/docs/bcm84891l-sfp-registers.md +++ b/docs/modules/fs/bcm84891l-sfp-registers.md @@ -7,7 +7,7 @@ Transcribed (visually, tables intact) from the FS-provided 4-page ## 1. Serial ID memory (A0h) Standard SFP MSA serial ID at I2C A0h. Notable factory values (the module is -honest about copper — cf. hardware.md): +honest about copper — cf. ../README.md): | Addr | Field | Value | Meaning | |---|---|---|---| @@ -76,4 +76,4 @@ Reference sequence "set PHY to work in 10GBE rate" (write in order): The EEE-advertisement line means the reference bring-up (and possibly the module firmware's own defaults) leaves 10GBASE-T EEE negotiable — verify and disable -for measurement runs (see modules.md probe list). +for measurement runs (see the probe results in README.md). diff --git a/docs/modules/wiitek/README.md b/docs/modules/wiitek/README.md new file mode 100644 index 0000000..1207403 --- /dev/null +++ b/docs/modules/wiitek/README.md @@ -0,0 +1,45 @@ +# Wiitek UF-RJ45-10G-100 (Marvell CUX3610) + +Test-set module; lying fake-SR EEPROM (passes stock ixgbe qualification). Reached over RollBall ([../README.md](../README.md)). + +## ⚠ DANGER: bricks permanently on certain reads + +Both original Wiiteks were bricked **permanently** by register exploration — the death survives cold power cycles and reseats. + +- The CUX3610's integrated microcontroller (Marvell Alaska-M, PHY ID 0x002B0BF4, firmware 12.1.5.0) boots from onboard non-volatile storage. The high vendor register windows (3.0x8000+, 31.0xF000+) reached over RollBall are **µC-mapped peripheral space, not PHY-core MDIO**. A read of a trapped peripheral crashes the µC mid-operation and corrupts its persistent boot state; it comes up FATAL forever after. +- **The trap: any register whose low byte is 0x64 in a high window (register high-byte ≥ 0x80), on any devad.** Module 1 died reading 3.0x8064, module 2 reading 31.0xF064. The mailbox writes the address bytes, then the µC dies executing the access — the command never needs to complete. 3.0x8064 is a known QCA808x CDT status register; the ODM firmware appears to emulate/intercept a CDT block there and crash on a cold read. A sloppy `(reg & 0xFF) == 0x64` decode explains both deaths with one bug. +- Rules for any Marvell/RollBall module: + - **Never sweep vendor register windows. Never read `*.0x??64` (high-byte ≥ 0x80).** + - Single targeted reads only, on registers with a documented reason. +- Recovery needs a vendor firmware image + the boot-download protocol (1.0xC050 boot status, 1.0xD0F0–D0F3 download) — we have no CUX3610 image, so bricks are effectively dead. A ghost I2C device at 7-bit 0x57 (returns zeros) is the ROM bootloader waiting for a download. + +The `rollball_ixgbe.py` client hard-guards the blacklist and raises *before* touching hardware — it structurally cannot repeat the kill. + +## Safe register set + +The shared IEEE 802.3an set ([../README.md](../README.md)) — SNR margin (offset-0x8000 encoding; saw A 9.1 / B 7.1 / C 7.7 / D 8.9 dB), PCS latches, AN status, retrain. Plus, proven safe in vendor space: + +| Registers | What | +|---|---| +| 1.0xC011/12 | Firmware version | +| 1.0xC050 | Boot status | + +Not available in safe space: pre-FEC counter, cable length — those live in the µC vendor space that killed the modules. + +## VCT/TDR candidates (unverified; single-shot, sacrificial unit only) + +Templates from kernel drivers for sibling parts — never a sweep, never near `*.0x??64`: + +- **88Q2220 TDR recipe** (`marvell-88q2xxx.c`, same 0x002b0bXX family, cited from "the Marvell API"), all MMD 3: + + | Register | Value / meaning | + |---|---| + | TDR_RESET 0xFECA | init 0x0D90, clear 0x1D90 | + | 0xFED9 / 0xFEDA / 0xFEDE | Calibration: 0x010E / 0x00EB / 0x0058 | + | TDR_STATUS 0xFEDD | Write 0x0002 to start; ~500 ms test. Read: bits[1:0]=01 done; [7:4] verdict (0x7 OK / 0xE open / 0x3 short / 0x5 noise); [15:8] distance in meters | + + Single-pair on the Q2220; the 4-pair CUX3610 likely has per-pair variants nearby. Low bytes nowhere near the 0x64 trap. +- **QCA-style CDT** (`qcom/qcom.h`): 3.0x8064 CDT_STATUS, 3.0x8065–0x8068 per-pair results, 3.0x8074–0x807E thresholds. **This IS the trap address** — only ever entered via the full start sequence, never a cold status read; treat as off-limits absent strong evidence. +- Doc-mining lead for the 4-pair pattern: `gtAdvVct.c` in the Marvell qd-dsdt GPL dumps (e.g. github.com/andy928/xpenology `arch/arm/plat-feroceon/mv_hal/qd-dsdt/`). + +The Marvell 88E151x datasheet PDF in this directory is family reference material. diff --git a/docs/marvell-ethernet-phys-alaska-88e151x-datasheet.pdf b/docs/modules/wiitek/marvell-ethernet-phys-alaska-88e151x-datasheet.pdf similarity index 100% rename from docs/marvell-ethernet-phys-alaska-88e151x-datasheet.pdf rename to docs/modules/wiitek/marvell-ethernet-phys-alaska-88e151x-datasheet.pdf diff --git a/docs/nics/README.md b/docs/nics/README.md new file mode 100644 index 0000000..6f49b7a --- /dev/null +++ b/docs/nics/README.md @@ -0,0 +1,21 @@ +# NICs + +The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path gets the hardware timestamps it requires. Per-device directories hold the details. + +## The transport rule + +A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols. + +## Comparison + +Which card is where: [../state.md](../state.md). + +| NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict | +|---|---|---|---| +| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only) | The certain diagnostics transport; product-capable only if the bucket exploration validates | +| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set | +| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform | +| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes | +| [WX1820 (txgbe)](wx1820/README.md) | Kernel `/dev/i2c-N` — ideal | No (PTP-only) | Disqualified | + +Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable) — [../open-questions.md](../open-questions.md) §1–2, §6. diff --git a/docs/nics/connectx-5/README.md b/docs/nics/connectx-5/README.md new file mode 100644 index 0000000..620569e --- /dev/null +++ b/docs/nics/connectx-5/README.md @@ -0,0 +1,22 @@ +# ConnectX-5 (mlx5) — the product NIC candidate + +Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set: + +- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — the committed bucketing and `probe.go` run unchanged. +- **One shared clock across both ports** — `probe.go`'s shared-PHC assumption holds. +- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed. +- Mature driver; SFP+ modules drop into SFP28 cages at 10G. + +## Open questions (decide on arrival) + +- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51. +- Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model. +- RollBall (0x51-resident, offset-model) fits MCIA — low risk. +- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig — or the X520 as fallback product NIC if the bucket exploration validates ([../../open-questions.md](../../open-questions.md) §2, §6). + +## Arrival notes + +- Ports may ship in InfiniBand mode: `mlxconfig -d set LINK_TYPE_P1=2 LINK_TYPE_P2=2`, reboot. +- Check whether auto-loading `mlx5_ib` interferes with channel changes (the irdma lesson, hardware.md). +- Force 10G if SFP28 autoneg is stubborn: `ethtool -s speed 10000 autoneg off`. +- Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway. diff --git a/docs/nics/e810/README.md b/docs/nics/e810/README.md new file mode 100644 index 0000000..2806248 --- /dev/null +++ b/docs/nics/e810/README.md @@ -0,0 +1,9 @@ +# Intel E810 (ice) + +The NIC the committed measurement path was built against: its datapath delivers per-packet MAC RX timestamps (`rx_filter=ALL`), which bucketing and `probe.go` require. A patched ice + `sff_i2c` exists for read-side diagnostics work. + +## Module I2C: multi-byte framing works, writes are policy-blocked + +- topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. **Reads work perfectly.** +- Every write returns aq 1 (EPERM) — tested identically against EEPROM 0x50 and BCM 0x56, so the block is write-vs-read, not address-scoped. Intel policy, survives NVM update; NVM 5.01 hardened the 3.10 silent-drop into an explicit EPERM. +- Dead for anything needing writes: RollBall unlock needs a write; BCM SMI reads *start* with a write. diff --git a/docs/nics/wx1820/README.md b/docs/nics/wx1820/README.md new file mode 100644 index 0000000..da08559 --- /dev/null +++ b/docs/nics/wx1820/README.md @@ -0,0 +1,5 @@ +# Wangxun WX1820 (txgbe) — evaluated, disqualified + +- Module I2C is architecturally ideal: kernel `sff`/phylink exposes `/dev/i2c-N` — arbitrary framing, zero patches. +- **Disqualified as product NIC**: PTP-only RX timestamping (verified in upstream `wx_ptp.c` — `HWTSTAMP_FILTER_ALL` falls to -ERANGE), the same limitation as the 82599 with a younger driver. +- Flow-steering/counter parity unverified. diff --git a/docs/nics/x520/README.md b/docs/nics/x520/README.md new file mode 100644 index 0000000..33b3e42 --- /dev/null +++ b/docs/nics/x520/README.md @@ -0,0 +1,25 @@ +# Intel X520-DA2 (82599, ixgbe) + +The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse. Cannot run the committed measurement path — PTP-only RX timestamping ([../../open-questions.md](../../open-questions.md) §2) — unless the bucket-timestamping exploration validates an equivalent. + +- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s) — enough for 2×10G full duplex at the default mix. +- **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it. + +## `sff_i2c` diagnostics transport (patched driver, validated) + +Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe//sff_i2c` (`w ` / `r `). + +- **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte. +- **Two-master windows**: each debugfs op takes the semaphore separately; serialization against the driver's own I2C holds per-op, but multi-op protocol sequences have unlocked gaps, and driver I2C is event-driven (SFP identify, link setup, `ethtool -m`) — firing exactly around cable swaps. Hardening item: a compound op under one semaphore hold ([../../open-questions.md](../../open-questions.md) §4). + +## Timestamping facts + +- RX latches **PTP frames only** (`TYPE_ALL`/TSIP are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`): one latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H), no RX-descriptor timestamp field. +- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset. +- Per-queue hardware counters QPRC/QBRC (16 slots via RQSMR, **read-clear**, `ixgbe_main.c:7777,7787`) count at queue write-back, upstream of ring residence — the basis of bucket-exploration route 1. +- Full analysis and bench pokes: [../../open-questions.md](../../open-questions.md) §2. + +## RX steering facts + +- ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only) — the tool fails at startup as committed. +- **ETQF/ETQS L2 EtherType filters are the answer on paper**: dedicated ethertype→queue stage ahead of RSS/fdir, 8 slots, 7 free (index 3 reserved for 1588). Flow Director cannot match non-IP frames at all. Analysis and patch shape: [../../open-questions.md](../../open-questions.md) §1. diff --git a/docs/nics/x710/README.md b/docs/nics/x710/README.md new file mode 100644 index 0000000..e6b5e1f --- /dev/null +++ b/docs/nics/x710/README.md @@ -0,0 +1,21 @@ +# Intel X710 (i40e) + +## Module I2C: 1-byte-offset only — dead end for BCM SMI + +- `reg_address > 0xFF` returns AQ retval 14 (hard boundary). +- Emits at most `[dev, offset, data]` (2 payload bytes); the BCM SMI needs 3–5 byte frames in one transaction. +- MDIO interface modes don't reach the module — no PHY on the NIC MDIO pins; copper SFPs wire only I2C. +- Single-byte **writes work** (unlike the E810) — which is what makes the oracle path below possible. + +## AQ oracle path (stock driver) — proved the RollBall client logic + +- Transport: `/sys/kernel/debug/i40e//command`, `send aq_cmd 0 0 0 0 0`, flags 0x2000. +- Opcode 0x0629 get / 0x0628 set PHY register. param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value. +- Result comes back in the dmesg "AQ desc WB" line, 9th field. +- Proven on the Fibergaga, link up: per-pair SNR (PMA 1.133–1.136), PCS 3.32/3.33, the 3.E820 counter ([../../modules/fibergaga/](../../modules/fibergaga/README.md)). + +## Timestamping + +- Rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`) — PTP-only, so no committed-path bucketing here. +- Genuinely shared PHC across ports (one oscillator) + PTP latches: the platform for the retrain-stability experiment and the scoped-but-unbuilt PTP path-delay length fallback ([../../open-questions.md](../../open-questions.md) §2) — probes must be PTP-shaped here too. +- No EEE on i40e/X710. diff --git a/docs/open-questions.md b/docs/open-questions.md index 11da9fb..ca0bb99 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -1,97 +1,108 @@ # Open questions -The genuinely open, thinking-worthy problems — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `transports.md`, `modules.md`, `measurement.md`); this assumes that context. +The genuinely open, thinking-worthy problems and their current answers — not mechanical tasks. Read the rest of `docs/` first (`state.md`, `nics/`, `modules/`, `measurement.md`); this assumes that context. -Short framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. We're swapping the E810 out for an Intel X520-DA2 (82599, `ixgbe`) so the host can bit-bang the SFP module I2C bus (the E810/i40e firmware blocked or couldn't frame it). Diagnostics clients and a raw-i2c kernel patch are prebuilt but untested. Test modules: FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented) and replacement Wiiteks (Marvell CUX3610, brick-prone). All modules are copper lying as fiber. +Framing: cabletest stresses 10GBASE-T copper cables at full-duplex 10G and attributes loss/errors to the cable. Which cards and modules are in play: `state.md`. -## 1. Raw-L2 RX steering on 82599 (gates whether cabletest runs on the X520 at all) -The tool steers raw ethertype 0x88b5 to per-queue NAPI contexts via Flow Director; ixgbe rejects ETHER_FLOW. But `FDIRCTRL_FLEX_SHIFT` is already set to 0x6 = byte offset 12 = the ethertype field, so the hardware samples it. Open: will a non-IP frame classify into an fdir flow_type the rule engine matches, and does pairing `flex_bytes` with a permissive base flow type need a driver change? Alternative: MAC→VMDq-pool steering (distinct dest MACs → queues, fully raw). Think through both against the 82599 datasheet before committing; UDP/bare-IP encap is the last resort we want to avoid. See the steering section of state.md. +## 1. Raw-L2 RX steering on 82599 -**Answered on paper from the driver source — needs only bench confirmation: use the 82599's L2 EtherType queue filters (ETQF/ETQS), not Flow Director.** -- The flex-byte lead is a dead end: 82599 Flow Director classifies IPv4/IPv6 packets only. The flex word *narrows* an IP-typed match; it cannot create a non-IP one, `FLEX_SHIFT` notwithstanding — non-IP frames never enter the fdir lookup. (One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7.) -- ETQF[0–7]/ETQS[0–7] are a dedicated ethertype→RX-queue stage checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses exactly this machinery — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports which ETQF index matched (`RXDADV_PKTTYPE_ETQF`). -- Slot budget fits: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588) is worth reserving (item 2 needs it) → 7 free slots = exactly the 7 streams. -- Patch shape: teach the ixgbe ethtool ntuple path to map exact-ethertype ETHER_FLOW inserts onto free ETQF slots, so `system.go` works unchanged. We already carry a driver patch; this is far smaller than fdir surgery. -- Explains the no-filter baseline too: non-IP frames hash to RSS 0 → all land on queue 0. -- MAC→VMDq remains the fallback (ETQF even has a pool field if pools ever matter). -- UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred on the X520 (no framing change, no header overhead in the size mix), but if it disappoints on bench, encap is fine. +**Status: answered on paper — use ETQF, bench-confirm.** The tool steers raw ethertype 0x88b5+ to per-queue NAPI contexts; ixgbe rejects ETHER_FLOW ntuple rules. -## 2. Can 82599 sustain zero-baseline-loss 2×10G full duplex, and what happens to timestamping? -It's older/Gen2 silicon. The whole tool depends on *exactly zero* host-side loss (see measurement.md). Separately: the cable-length-via-timestamp idea assumed two ports on one NIC sharing a PHC with usable TX+RX hardware timestamps — 82599's PTP is limited (largely PTP-only, sparse). Re-examine whether the timestamp path-delay length measurement is even viable on this NIC, or whether it dies with the E810. +- **Flow Director is a dead end for raw L2.** 82599 fdir classifies IPv4/IPv6 only; the flex word (`FDIRCTRL_FLEX_SHIFT = 0x6` = byte offset 12 = the ethertype field) *narrows* an IP-typed match but cannot create a non-IP one — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test; datasheet §7.1.2.7. +- **ETQF/ETQS is a dedicated ethertype→RX-queue stage** checked at L2, ahead of RSS/fdir: ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE (bits 22:16) + QUEUE_EN (`~/work/ixgbe-sff/ixgbe_type.h:381,1772–1806`). ixgbe already uses it — 0x88F7→1588 latch (`ixgbe_ptp.c:1099`), FCoE/FIP (`ixgbe_fcoe.c:656,691`), LLDP/FC under SR-IOV (`ixgbe_sriov.c:1828`) — and the RX descriptor reports the matched index (`RXDADV_PKTTYPE_ETQF`). +- **Slot budget fits**: 8 filters; with no FCoE/DCB/SR-IOV only index 3 (1588, needed by §2) is reserved → 7 free = exactly the 7 streams. +- **Patch shape**: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots in the ixgbe ethtool path, so `system.go` works unchanged. Far smaller than fdir surgery; we already carry a driver patch. +- Baseline behavior without filters: non-IP frames hash to RSS 0 → all land on queue 0. +- Fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap is acceptable — a fallback, not a last resort — so raw-L2 steering is not a hard requirement of NIC choice. ETQF stays preferred (no framing change, no header overhead in the size mix). -**Throughput arithmetic says yes at the default config.** X520-DA2 is PCIe Gen2 x8: ~32 Gb/s raw per direction, ~25–26 Gb/s effective after 8b/10b + TLP overhead, against 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already, so it tests nothing new here. On install verify the link actually trained 5 GT/s ×8 (`lspci -vv`). Loss attribution survives the move: 82599 has missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters, so "prove host-side zero" still works. +## 2. Zero-baseline loss and timestamping on the 82599 -**Timestamping is two separate uses — keep them distinct.** -- *Use 1: rate bucketing — every received frame.* The committed path buckets by per-packet MAC RX timestamps (`rx.go` `rateRun.add` on the cmsg stamp), obtained via `SO_TIMESTAMPING` with `rx_filter=ALL` — and that filter is a hard host check (`ts.go`; any failed check aborts the run, `system.go` `reportChecks`). Only NICs that can timestamp *all* RX packets qualify: the E810 could; i40e rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`), and 82599 is PTP-only (`TYPE_ALL` is X550+). As committed, the timestamp check therefore fails on both the interim X710 pair and the incoming X520. **There is no software fallback — software RX stamping was tried and cannot reach the needed precision.** All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright, and whether its other hardware can still produce honest buckets is worked through below — that answer drives the NIC choice in item 6. (measurement.md's read-time-stamping lesson is about the *NIC-counter* buckets, a separate system; it is not a substitute for the per-frame stamps.) -- *Use 2: length probes — low volume.* Also already built and committed (`probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry). It rides the same `rx_filter=ALL` for its receive stamps, so on PTP-only NICs the probes must be reshaped as L2 1588 event frames (0x88F7) under a PTP rx filter. The existing both-direction averaging also cancels a stable inter-port clock offset, so `probe.go`'s architecture survives the 82599's per-port clocks — but the two free-running SYSTIMs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first. +**Status: throughput yes; per-packet timestamps no — exploration routes mapped, plus one decisive experiment runnable on the X710 ahead of any build-out.** -**The full 82599 timestamp-hardware space for use 1 — what exists, what might, and the routes to honest buckets without per-packet stamps.** What X550 calls "timestamp all" is two mechanisms together: `TYPE_ALL` filtering plus TSIP (`TSYNCRXCTL_TSIP_UT_EN` — the stamp *prepended into the packet buffer*, which is what makes per-packet delivery scale). The 82599 documents neither: its RX 1588 unit is a single latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`) plus attribute-capture registers (RXSATRL/H, unused by the driver), and its advanced RX descriptor has no timestamp field. Literal per-packet stamps are absent from the silicon. What remains, best first: +### Throughput -1. **Per-queue hardware counters as bucket content — likely sufficient, and the key insight is *where* they count.** QPRC/QBRC (16 slots; queues map to slots via RQSMR; 7 streams fit) increment at queue write-back — *upstream of ring residence*, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate, more than an entire 16 ms bucket, while the RX FIFO ahead of the counters bounds their smear at 512 KB ≈ 400 µs worst-case and ≈ 0 in zero-loss steady state. A sampler in the patched driver (hrtimer at 1–16 ms, snapshot all slots plus ktime into a page the tool maps) yields buckets whose *content* is hardware-counted and whose *boundaries* carry read-time labels — the same honesty discipline measurement.md already proved for counter rates, with µs kernel jitter against 16 ms buckets. Constraint: QPRC/QBRC are read-clear (`ixgbe_main.c:7777,7787`), so exactly one reader — the sampler owns the reads and feeds the driver's accumulators. Nothing of value is lost: `rateRun` only ever carried frames/bytes per bucket. -2. **The latch as an identifiable sampling instrument.** Reading RXSTMPH re-arms the latch. Anchor frames shaped as L2 PTP V2 events with our sequence counter placed in the PTP sequenceID field get true MAC-time stamps, and RXSATRL/H capture the identifying header bytes — each stamp attributable to a specific frame. Even ~1 k stamps/s of ground truth continuously validates route 1's boundaries and measures FIFO+ring residence live. The same machinery is the on-card probe path (below). -3. **Wire-determinism reconstruction — the reserve.** At saturation the TX ring drains at exactly line rate, so every frame's wire time is computable from sequence numbers plus the known size schedule, anchored by sparse latch stamps: exact per-frame times with no per-frame hardware. Breaks off-saturation; reach for it only if routes 1–2 fall short. +- PCIe Gen2 x8 ≈ 32 Gb/s raw, ~25–26 Gb/s/direction effective, vs 20 Gb/s/direction needed for 2×10G full duplex plus descriptor/writeback traffic. The full-size mix (~1.78 Mpps/dir) fits; the 64 B case was host-bound on the E810 already. Verify 5 GT/s ×8 trained (`lspci -vv`). +- Loss attribution survives: 82599 exposes missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" still works. -Bench pokes that decide the map, first day on the card: -- Write `TYPE_ALL` (0x08) into TSYNCRXCTL on the 82599 and blast raw frames — the encoding is defined silicon-wide in the driver header but documented only for X550; if the reserved encoding secretly arms the latch on raw frames, route 2 sheds its PTP-shaping constraint (RXSATR captures whatever bytes sit at the attribute offsets — put the stream seq there). Poke `TSIP_UT_EN` the same way and look for a buffer prepend. Expected dead; minutes to know. -- RQSMR mapping, QPRC read-clear semantics, and coexistence with the watchdog stats task. -- Latch service rate: sustainable identifiable stamps/s from a dedicated poller. -- Route-1 sampler prototype, validated against route-2 anchors, and against software stamps at low load where software is honest. +### Timestamping is two separate uses — keep them distinct -If route 1 validates, the X520 delivers honest buckets after all — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the fallback should MCIA fail to reach the BCM on the ConnectX (item 6). +| Use | Mechanism (committed) | 82599 | +|---|---|---| +| 1. Rate bucketing — every received frame | Per-packet MAC RX stamps via `SO_TIMESTAMPING` cmsg (`rx.go` `rateRun.add`); `rx_filter=ALL` enforced as a hard host check (`ts.go`; failed checks abort, `system.go` `reportChecks`) | **Cannot** — PTP-only (`TYPE_ALL` is X550+, `ixgbe_ptp.c:1043`); i40e also rejects `FILTER_ALL` with -ERANGE (verified upstream). Only the E810 qualified | +| 2. Length probes — low volume | `probe.go`: 64 B probes on stream 0xffff every 200 ms, TX stamp via MSG_ERRQUEUE, min-delta tracking, both-direction averaging cancelling the measured ~790 ns PHY TX/RX asymmetry | Possible with PTP-shaped probes (below) | -**Probes: the shared-PHC assumption dies, the method doesn't (verified in `ixgbe_ptp.c`).** Each 82599 port is its own PCI function with its own free-running SYSTIME/PHC, and the NIC timestamps *only* PTP frames — `HWTSTAMP_FILTER_ALL` is X550+ (`ixgbe_ptp.c:1043`; 82599 gets -ERANGE) — with a single latch register per direction and ~6.4 ns granularity at 10G. Consequences: -- Probe frames must be 1588 L2 event frames (ethertype 0x88F7). They coexist with line-rate 0x88b5 traffic since only 0x88F7 latches — length probing can run *during* the stress test. (0x88F7 recognition is ETQF index 3 — reserved in item 1.) -- Two SYSTIMEs but one crystal: relative drift ≈ 0. A two-way exchange over the same cable (A→B then B→A) cancels the unknown offset exactly, PTP-peer-delay style. -- 6.4 ns quantization ≈ 1.3 m of round trip — average many probes. +**There is no software fallback for use 1** — software RX stamping was tried and cannot reach the needed precision. All-packet hardware RX stamping is a hard NIC requirement; the 82599 lacks the committed mechanism outright. (measurement.md's read-time-stamping lesson covers the *NIC-counter* buckets, a separate system.) -**The decisive unknown is the module PHYs, not the NIC — and it's testable today, before the X520 arrives.** The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the timestamp path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches, in hand now — but probes must be PTP-shaped here too, since i40e also stamps only PTP): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. Spread ≲ 10–20 ns → viable; much more → close item 3's timestamp path permanently. +### The full 82599 hardware space for use 1 — routes to honest buckets without per-packet stamps -## 3. Consolidate the cable-length strategy — currently three partial paths -- **BCM ECD/DSP register** — real, but the ECD chapter is missing from FS's docs. What's the fallback if FS doesn't deliver: mine the Broadcom SDK/patents, or lean on the QCA/Marvell CDT analogy? -- **Marvell VCT/DSP** — undocumented, and gated behind a firmware trap that permanently bricks the module on certain reads (`*.0x??64`, high window; see modules.md). Is there *any* safe route, or is it strictly vendor-docs-or-sacrificial-unit? -- **Timestamp path-delay** — see item 2. +What X550 calls "timestamp all" is `TYPE_ALL` filtering **plus TSIP** (`TSYNCRXCTL_TSIP_UT_EN` — stamp prepended into the packet buffer, which is what makes per-packet delivery scale). The 82599 documents neither: one RX latch (RXSTMPL/H, locked until RXSTMPH is read, `ixgbe_ptp.c:742`), attribute-capture registers (RXSATRL/H, unused by the driver), no timestamp field in the RX descriptor. Routes, best first: -Decide which is the product path vs. nice-to-have. +1. **Per-queue hardware counters as bucket content — likely sufficient.** QPRC/QBRC (16 slots; queues map via RQSMR; 7 streams fit) increment at queue write-back — *upstream of ring residence*, which is exactly where software stamping died: a full 4096-descriptor ring is ~18 ms of residence at per-queue line rate (more than one 16 ms bucket), while the RX FIFO ahead of the counters bounds smear at 512 KB ≈ 400 µs worst-case, ≈ 0 in zero-loss steady state. Implementation: an hrtimer sampler in the patched driver (1–16 ms cadence, snapshot slots + ktime into a page the tool maps) — hardware-counted content, read-time-labeled boundaries, the same honesty discipline measurement.md proved. Constraint: QPRC/QBRC are read-clear (`ixgbe_main.c:7777,7787`) — exactly one reader; the sampler owns the reads and feeds the driver's accumulators. Nothing lost: `rateRun` only ever carried frames/bytes per bucket. +2. **The latch as an identifiable sampler.** Reading RXSTMPH re-arms it. Anchor frames shaped as L2 PTP V2 events carrying our sequence counter in the PTP sequenceID field get MAC-time stamps; RXSATRL/H capture the identifying bytes. Even ~1 k stamps/s continuously validates route 1's boundaries and measures FIFO+ring residence live. +3. **Wire-determinism reconstruction — the reserve.** At saturation the TX ring drains at exactly line rate; per-frame wire times are computable from sequence numbers + the size schedule, anchored by sparse latch stamps. Breaks off-saturation; only if 1–2 fall short. -- **BCM fallback, checked:** OpenBCM's `phy8481.c` covers the copper XGPHY family only through BCM8488x — no 84891, and no ECD code in the retrievable portion — and the kernel's Broadcom ECD (`bcm-phy-lib`) is the BCM54xx GbE register model; neither transfers. SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route. -- **The Aquantia-based oracle gets a fully documented path the Wiiteks can't have.** The Fibergaga's PHY family register reference is now vendored in this folder (`aquantia-aqr107-aqr109-datasheet.pdf`, see item 5), and it delivers the DSP estimate outright: **`1E.C884` "Global Status" bits 7:0 = estimated cable length in meters, ±1 m, RO, "estimated from the cable diagnostic engine."** Plus full TDR results: per-pair fault verdicts (open/short/mismatch/cross-pair) at `1E.C800`, distances of the four worst reflections per pair (±1 m) at `1E.C801–C807`, reflection impedance/magnitude at `1E.C880–C883`. Bench question: whether C884 is continuously valid on a linked cable or only after an explicit diag run. This also proves at least one vendor exposes the DSP length in a plain register — useful precedent for the FS/Broadcom ECD ask. The CUX3610 stays sacrificial-unit-only. -- **Proposed decision:** product path = the PHY's own length machinery — BCM ECD if/when FS delivers, the documented `1E.C884` estimate on the Aquantia oracle; NIC timestamp path-delay = the module-independent fallback, pursued only if the item-2 retrain-stability experiment passes. +Bench pokes, first day on the card: -## 4. Two-master I2C safety + correctness of the untested clients -The `ixgbe` driver polls the SFP EEPROM (DOM/qualification) on its own; our `sff_i2c` transactions share the bus. The swfw semaphore *should* serialize us, but verify the reasoning. Also scrutinize framing assumptions we couldn't test: -- the BCM SMI read's >1 ms inter-transaction delay adequacy; -- the EEPROM-read assumption that the word-address pointer persists across STOP; -- whether RollBall's mailbox read (set-pointer-then-read as two separate transactions) matches what the module expects vs. the kernel `mdio-i2c` combined-transfer form. +- Write `TYPE_ALL` (0x08) into TSYNCRXCTL and blast raw frames — defined silicon-wide in the header, documented only for X550; if the reserved encoding secretly arms the latch, route 2 sheds its PTP-shaping constraint (put the stream seq at the RXSATR capture offsets). Poke `TSIP_UT_EN` and look for a buffer prepend. Expected dead; minutes to know. +- RQSMR mapping, QPRC read-clear semantics, coexistence with the watchdog stats task. +- Latch service rate: sustainable identifiable stamps/s. +- Route-1 sampler prototype, validated against route-2 anchors and against software stamps at low load (where software is honest). -**Verified against the source — per-op bracketing is real, and it is the weak point.** Each debugfs `w`/`r` acquires and releases the swfw semaphore individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`), and both clients compose their protocols from separate ops with userspace sleeps between them (`bcm_ixgbe.py` `mdio_read`: `w`, 3 ms sleep, `r`; `eeprom()`: pointer-write then read; same shape in `rollball_ixgbe.py`). Every multi-op sequence therefore has windows where the driver can run its own I2C. Working through the concerns: -- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`, so no mid-transaction bus corruption. But that traffic is event-driven — SFP identify after module/link events, link setup, `ethtool -m` — i.e. it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but *correlated with the interesting moments*. -- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the real risk is the other master *moving* it. An interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence (`eeprom()`, RollBall mailbox reads). -- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe — the driver never addresses 0x56 and I2C devices are address-filtered — but that's the one interleaving case with no proof. -- **One fix kills all of it**: add a compound debugfs op (write bytes, STOP, optional delay in µs, read n) executed under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds the semaphore longer during SFP identify. This is the same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the i2c bus across its multi-transfer sequences). -- **RollBall split transactions are already field-proven**: the i40e oracle path drove RollBall entirely with single-byte, separate-STOP AQ transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules; only the interleaving above needs fixing. -- **The >1 ms figure cannot be validated from experience** — the E810 never executed an SMI read (reads start with an I2C *write*, which was EPERM-blocked), so the whole SMI path including the delay is untested. Make it self-calibrating at bringup: sweep the delay 0.5→5 ms on a known-nonzero read (PHY ID), find where 0x0000 stops appearing, run at 3×; and treat 0x0000 results as retry-with-longer-delay, since 0 is also the bridge's not-ready signature. +If route 1 validates, the X520 delivers honest buckets — its disqualification narrows from "cannot measure" to "committed mechanism unavailable, equivalent required," which matters as the MCIA fallback (§6). -## 5. Pre-FEC error visibility — design the experiment -No module exposes a pre-FEC counter in reachable space; the Aquantia `3.e820` clear-on-read counter is the best candidate but is unclassified (pre- vs post-FEC) because it won't move on a healthy short link. cabletest's own noise pair can create a marginal channel — design the correlation (module counter vs. host CRC/loss under graded noise) that would classify it and turn it into a real leading indicator. +### Probes on the 82599 -**Answered from documentation: `3.E820` is post-FEC, and the pre-FEC counters are documented next door.** The AQR107-AQR109 register reference (December 2017, rev 1.2.4) is vendored in this folder as `aquantia-aqr107-aqr109-datasheet.pdf` (source mirror: assets.sourcengine.com/datasheets/5fe9c033-cf7a-4b7b-b950-80efc0bbb681.pdf). Verified against it: -- `3.E820`/`3.E821` = **PCS Receive Vendor Uncorrected Frame Counter** 1/2 (§6.7.225–226) — uncorrectable LDPC frames, i.e. it moves *with* host CRC/loss, not before. -- `3.E840–3.E84E` = **PCS Receive Vendor Corrected Frame N Iteration Counter** for N = 1…10 LDPC iterations (§6.7.228–242; N=1–5 are 32-bit register pairs, N=6–10 single 16-bit) — a full corrected-by-iteration histogram, the best possible leading indicator: mass shifting toward higher iteration counts = margin eroding while everything is still corrected and the host sees nothing. -- `3.E816` = PCS Receive Vendor RS Corrected Error Frame Counter (§6.7.223) — likely the system-side/RS path, secondary. +- The shared-PHC assumption dies, the method doesn't: each port has its own free-running SYSTIME (~6.4 ns granularity at 10G), but both share one crystal — relative drift ≈ 0, and a two-way exchange over the same cable cancels the unknown offset exactly, PTP-peer-delay style. `probe.go`'s existing both-direction averaging already provides this — but the two SYSTIMEs start at driver load and can sit seconds apart, outside the `probeMaxDelay` accept window, so they need a one-time offset estimate (or phc2sys) first. +- Probe frames must be 1588 L2 event frames (0x88F7): only those latch, so probing coexists with line-rate 0x88b5 traffic (0x88F7 recognition is ETQF index 3, reserved in §1). 6.4 ns quantization ≈ 1.3 m of round trip — average many probes. -First action: add the corrected-iteration counters to the oracle client. +### The decisive unknown is the module PHYs — testable before anything is built + +The path crosses two 10GBASE-T PHYs (~2–3 µs pipeline each); the timestamp length method needs that latency stable across retrains, but LDPC frame alignment quantizes at ~320 ns ≈ 65 m equivalent — if latency shifts by alignment quanta per training, no calibration survives a re-plug and the path is dead on *every* NIC. Experiment on the X710 pair (genuinely shared PHC, PTP latches — probes must be PTP-shaped there too): fixed cable, force N retrains (`7.0 |= bit9`), measure round-trip spread. ≲ 10–20 ns → viable; much more → close §3's timestamp path permanently. + +## 3. Cable-length strategy + +**Status: consolidated — PHY features are the product path, NIC timestamps the fallback.** + +| Path | Status | +|---|---| +| BCM ECD/DSP | Real, but the ECD chapter is missing from FS's docs (modules/fs/ ask list). The handler exposes only a 1-bit trace of the DSP estimate (limited-reach linked bit, and only with LR mode enabled) | +| Aquantia DSP/TDR (oracle only) | **Fully documented**: `1E.C884` length ±1 m, per-pair TDR verdicts and reflection distances (modules/fibergaga/). Proves at least one vendor exposes DSP length in a plain register — precedent for the FS ask | +| Marvell VCT (Wiiteks) | Undocumented and gated behind the brick trap; sacrificial-unit-only single-shot templates (modules/wiitek/) | +| NIC timestamp path-delay | Module-independent; gated on the §2 retrain-stability experiment | + +- Fallback doc-mining, checked: OpenBCM's `phy8481.c` covers the copper XGPHY family only through BCM8488x — no 84891, no ECD code in the retrievable portion; the kernel's Broadcom ECD (`bcm-phy-lib`) is the BCM54xx GbE register model. Neither transfers; SDK/patent mining looks low-yield. FS delivering the ECD chapter stays the primary route. +- **Decision:** product path = the PHY's own length machinery (BCM ECD when FS delivers; `1E.C884` on the oracle); NIC timestamp path-delay = fallback, pursued only if the §2 experiment passes. + +## 4. Two-master I2C safety and the once-untested client assumptions + +**Status: verified against source and hardware; one hardening item remains.** + +- **Per-op semaphore bracketing is real and is the weak point.** Each debugfs `w`/`r` acquires/releases swfw individually (`~/work/ixgbe-sff/ixgbe_phy.c:63–108`); clients compose protocols from separate ops with userspace sleeps. Every multi-op sequence has windows where the driver can run its own I2C. +- **Serialization itself holds**: the driver's own SFP traffic uses the same byte primitives under the same `phy_semaphore_mask` — no mid-transaction bus corruption. But driver traffic is event-driven (SFP identify after module/link events, link setup, `ethtool -m`) — it fires exactly around cable swaps, when diagnostics also run. Collisions are rare but correlated with the interesting moments. +- **Pointer persistence, resolved**: AT24-style word pointers do persist across STOP; the risk is the other master *moving* the pointer — an interleaved driver read of 0xA0/0xA2 silently corrupts any pointer-set→STOP→read sequence. +- **The BCM 0x56 bridge's pending read data** across the >1 ms window is probably safe (the driver never addresses 0x56; I2C is address-filtered) — the one interleaving case without proof. +- **Remaining hardening item**: a compound debugfs op (write bytes, STOP, optional delay µs, read n) under a single swfw hold. A ~1.5 ms hold is nothing — the driver holds it longer during SFP identify. Same shape as the kernel's fix for the identical RollBall-vs-DOM-poll race (`mdio-i2c` locks the bus across multi-transfer sequences). +- **RollBall split transactions: field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks. The kernel's combined form is not required by the modules. +- **The >1 ms BCM delay**: validated on hardware at 3 ms. Treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature. + +## 5. Pre-FEC error visibility + +**Status: answered from documentation — `3.E820` is post-FEC; the pre-FEC counters are the corrected-by-iteration histogram next door.** Register map: modules/fibergaga/. The correlation experiment is now verification, not classification. + +Design points: -The correlation experiment becomes verification rather than classification. Design points that stand regardless: - Dose axis = *measured* SNR margin (1.133–6), not the noise-knob position — plot counter rates against margin; don't assume the knob is monotonic. -- Run at high line utilization: an uncorrected LDPC frame that lands in idle/IFG leaves no host-visible trace, so low-load runs undercount the host side of the correlation. -- Expected ordering as margin falls: corrected counters rise first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1 counts. -- Log fast retrains alongside (Clause 45 fast-retrain status/count, ~1.147 — verify address) to separate FEC exhaustion from retrain hiccups. +- Run at high line utilization: an uncorrected LDPC frame landing in idle/IFG leaves no host-visible trace; low-load runs undercount the host side. +- Expected ordering as margin falls: corrected-iteration counters shift toward higher N first (host still perfect — the leading-indicator regime), then E820 + PCS 3.32/33 errored blocks + host CRC/loss together. One uncorrected LDPC frame is 2048 line bits (~320 ns) and can clip multiple packets — expect burstiness, not 1:1. +- Log fast retrains alongside (IEEE 1.147) to separate FEC exhaustion from retrain hiccups. - Clear-on-read discipline: exactly one reader during a run. -- Scope: this documents the Aquantia-based Fibergaga (the oracle) — which is fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out of this until documented. +- Scope: documents the Aquantia oracle — fine; proving the pipeline is the oracle's job. The CUX3610 is Alaska-M with its own map and stays out until documented. -## 6. Architecture sanity check -Is putting *both* test traffic and module diagnostics on the X520 right, given its steering/timestamp weaknesses, or is there a cleaner split (traffic on a NIC that steers well, diagnostics on the X520)? The tension: the modules under test are where the traffic physically flows, so splitting is awkward — but worth pressure-testing the assumption. +## 6. Architecture -**Resolved: co-location is forced, not chosen.** The outputs that matter — SNR under load, the pre-FEC correlation (item 5), diagnosing the module that carried the stress — all require the diagnostics I2C to reach the very modules the traffic flows through, and module I2C is only reachable through the NIC hosting the module. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path; the split is structurally impossible, not merely awkward. The chassis seconds it: one usable slot (hardware.md). +**Status: resolved twice over — co-location is forced, and the single card is the ConnectX-5.** -**But the single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (item 2: software stamping was tried, not precise enough) and raw-L2 steering is not (item 1: encap acceptable) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only, so it cannot carry the committed measurement path; and among the transports.md candidates, txgbe is also PTP-only (verified in upstream `wx_ptp.c`), leaving **ConnectX-4/5 (mlx5) as the only candidate that can meet the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (probe.go's assumption holds), native ETHER_FLOW ntuple steering (no driver patch, no encap needed), mature driver. Its open questions are exactly the diagnostics ones transports.md flagged: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only ever use 0x50/0x51) and whether the BCM SMI read's data phase (raw 2-byte read, no offset write) can be framed at all. RollBall (0x51-resident, offset-model) fits. If MCIA can't reach the BCM, the fork is: product diagnostics via Marvell/Aquantia modules only, with the X520 kept as an offline BCM diagnostics rig — its arbitrary bit-banged framing stays the only *certain* SMI transport, and it remains the bring-up card for proving the clients either way. +- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation (§5), and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot. +- **The single card is not the X520.** Two constraints — all-packet hardware RX stamps are non-negotiable (§2) and raw-L2 steering is not (§1) — flip the requirement weights the X520 was chosen under. The 82599 stamps PTP only; txgbe is also PTP-only (verified in upstream `wx_ptp.c`); **ConnectX-4/5 (mlx5) is the only candidate meeting the full set**: CQE-stamps every RX packet (`HWTSTAMP_FILTER_ALL`), one shared clock across both ports (`probe.go`'s assumption holds), native ETHER_FLOW ntuple steering, mature driver. +- **Open on the ConnectX**: whether firmware MCIA accepts arbitrary I2C device addresses (the BCM sits at 0x56; kernel paths only use 0x50/0x51) and whether the BCM SMI read data phase (raw 2-byte read, no offset write) can be framed. RollBall (0x51-resident, offset-model) fits. +- **If MCIA can't reach the BCM**: product diagnostics via the Marvell/Aquantia modules only, with the X520 kept as an offline BCM rig — its bit-banged framing stays the only *certain* SMI transport — or, if the §2 bucket exploration validates, the X520 itself as fallback product NIC. diff --git a/docs/state.md b/docs/state.md index 6057af0..984280c 100644 --- a/docs/state.md +++ b/docs/state.md @@ -1,46 +1,73 @@ # Current state ## Committed tree -AF_PACKET raw sockets everywhere (`sock.go`), flow-director steering, per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — see open-questions.md §2 for what that demands of the NIC), read-time-stamped NIC-counter rates, hardware-timestamped length probes (`probe.go`), framebuffer UI, harness. The zero-copy AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata) lives uncommitted in `stash@{0}`; it was built for the era when the test path ran on the E810, whose datapath delivers per-packet MAC timestamps. It is an experiment, not a requirement. -`stash@{1}` holds the phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, and a STATUS.md campaign log. Working copies extracted to `~/work/phydiag-work/` during the module work. The knowledge from that campaign is now in these docs; the code should eventually move into the repo properly. +AF_PACKET raw sockets everywhere (`sock.go`); flow-director steering; per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL` as a hard host check — open-questions.md §2 for what that demands of the NIC); read-time-stamped NIC-counter rates; hardware-timestamped length probes (`probe.go`); framebuffer UI; harness. -## Hardware in flight -- **Intel X520-DA2 installed** in the box's single PCIe slot (E810 out; PCIe trained 5 GT/s ×8). Port 0 `enp1s0f0` holds a new Wiitek (SN WAMZ012606X039U), port 1 `enp1s0f1` the FS (SN S2433774168), cable between them linked at 10G. Stock ixgbe runs with `allow_unsupported_sfp=1` — mandatory, and it's the *FS* that needs it (honest 10GBASE-T EEPROM fails Intel qualification; see hardware.md). The patched `sff_i2c` driver is not yet loaded. The X520 was chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md. -- **ConnectX-5 ordered (dual SFP28, PCIe x8) — the product NIC candidate.** Rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement); the 82599 stamps PTP frames only, and with raw-L2 steering relaxed (encap acceptable), mlx5 is the one driver that meets the full set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: the MCIA diagnostics questions in open-questions.md §6 (BCM at I2C 0x56, SMI read data phase). On arrival: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`), SFP+ modules drop into SFP28 cages at 10G, and check whether `mlx5_ib` autoloading interferes with channel changes (the irdma lesson). -- **The X520 arrives first and carries two campaigns**: the diagnostics bring-up (certain arbitrary I2C framing for proving the BCM/RollBall clients), and the 82599 bucket-timestamping exploration in open-questions.md §2 — per-queue hardware counters as bucket content plus latch-anchor sampling. If that validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX. -- **Replacement Wiitek (Marvell CUX3610) modules arrived** — one is in X520 port 0. Both original units were bricked by register exploration — see modules.md for the trap before ever touching one. -- **In hand** (all copper RJ45 10GBASE-T modules with cloned/lying fiber EEPROMs — see hardware.md; none are actually fiber): - - 2× FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented diagnostics). - - 1× Fibergaga SFP-10G-T-30M (Aquantia, RollBall, the proven oracle module). - - 1× 10Gtek, EEPROM claims SFP-10G-SR — still a copper RJ45 module; filler, not part of the test set. +## Stashes + +- `stash@{0}` — AF_XDP conversion (both directions, hand-rolled XSK + hand-assembled XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement. +- `stash@{1}` — phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly. + +## The box + +- Single usable PCIe slot (Gen4 x8): holds the X520-DA2, trained at the card's Gen2 ceiling (5 GT/s ×8, 32 Gb/s). The ConnectX-5 replaces it on arrival. +- X710 on a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case bandwidth warning). +- Many CPU cores; goroutine-heavy designs welcome. + +| Interface | Device | Role (rules: hardware.md) | +|---|---|---| +| `enp1s0f0` | X520 port 0 (ixgbe) | Test pair — new Wiitek module | +| `enp1s0f1` | X520 port 1 (ixgbe) | Test pair — FS module | +| `enp3s0f0np0` / `enp3s0f1np1` | X710 (i40e) | Noise pair (has been `enp4s0f*` across reboots) | +| `enp88s0` | igc | LAN uplink, default route; sibling `enp89s0` is dark | + +## Hardware + +| Item | Status | Notes | +|---|---|---| +| X520-DA2 | **Installed** in the single PCIe slot (E810 out); PCIe 5 GT/s ×8 | Port 0 `enp1s0f0` = new Wiitek (SN WAMZ012606X039U); port 1 `enp1s0f1` = FS (SN S2433774168); cable linked at 10G. Stock ixgbe needs `allow_unsupported_sfp=1` — the *FS* trips qualification (hardware.md) | +| ConnectX-5 | **Ordered** (dual SFP28, PCIe x8) — the product NIC candidate | mlx5 is the one driver meeting the full requirement set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: MCIA diagnostics questions (open-questions.md §6). Arrival notes: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`); SFP+ drops into SFP28 cages at 10G; check `mlx5_ib` vs channel changes (the irdma lesson) | +| Replacement Wiiteks | Arrived; one in X520 port 0 | Originals bricked by register exploration — modules/wiitek/ trap first | +| FS SFP-10G-T-100 ×2 | In hand; one in X520 port 1 | BCM84891L, documented, robust | +| Fibergaga SFP-10G-T-30M | In hand | Aquantia, RollBall, the documented oracle | +| 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set | +| E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work | + +**The X520 cannot be the product NIC**: rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement) and the 82599 stamps PTP only. It carries two interim campaigns: the diagnostics bring-up (certain arbitrary I2C framing) and the 82599 bucket-timestamping exploration (open-questions.md §2). If that exploration validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX. ## X520 diagnostics path — validated on hardware -In `~/work/` alongside the ice-sff/phydiag-work artifacts, ready to fold into the repo's `kernel/`: -- **Patched ixgbe** (`~/work/ixgbe-sff/`): adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions built from the existing bit-bang primitives, swfw-semaphore bracketed) and an `sff_i2c` debugfs file with `w ` / `r ` commands, result read back from the fd. **Proven on the card after one real bug**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible until now); `raw_read` now releases SDA after each ACK, mirroring the release block stock uses after clocking out a byte. `load-ixgbe` rebuilds + swaps the module and passes `allow_unsupported_sfp=1` (mandatory for the FS). -- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): BCM SMI framing proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, and per-pair SNR ≈ [32, 27.5, 30, 27.6] dB on the bench cable via `CMD_GET_SNR = 0x8030` invoked with no DATA1 write (see transports.md for the code-table trap and the pause-mode side quest). -- **RollBall client for the Wiiteks** (`~/work/phydiag-work/rollball_ixgbe.py`): the same `sff_i2c` transport, RollBall unlock/page/mailbox, per-pair IEEE SNR (PMA 1.133–1.136), untested on this card. The `*.0x??64` (high-byte ≥ 0x80) brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill. -## Bringup plan when the X520 arrives -1. Card installed, modules seated, `./load-ixgbe` with `allow_unsupported_sfp=1` (mandatory — the FS's honest 10GBASE-T EEPROM fails Intel qualification and kills the port probe; `ixgbe_main.c:165`), `sff_i2c` present. Done. -2. FS/BCM proven: SMI transport, EEPROM/PHY-ID/STATUS sanity, command handler (temp, voltage, per-pair SNR). The IEEE-register SNR path is dead on the BCM — the command handler is the source. Done. -3. Replacement Wiiteks: IEEE-standard registers only (SNR, latches) unless/until a documented recipe exists for more; the VCT templates in modules.md are candidates for single-shot targeted probes on a sacrificial unit only. +In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`: + +- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w ` / `r `). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`. +- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via `CMD_GET_SNR = 0x8030` invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/. +- **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/). +- **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill. + +## Bringup plan + +1. ~~Card installed, modules seated, `./load-ixgbe`, `sff_i2c` present.~~ Done. +2. ~~FS/BCM proven: SMI transport, sanity reads, command handler (temp, voltage, per-pair SNR), config sweep, EEE forced off.~~ Done. The IEEE-register SNR path is dead on the BCM — the command handler is the source. +3. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only. 4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs). 5. RX steering: program/verify the ETQF path (open-questions.md §1) before cabletest traffic runs. -## Open: RX flow-steering on ixgbe needs serious investigation before the X520 carries test traffic -cabletest fans RX across queues by steering **raw ethertype** (0x88b5 base) to distinct queues with Flow Director rules (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). The obvious path fails: ixgbe/82599's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — **no ETHER_FLOW** — so an ETHER_FLOW rule insert is rejected (EINVAL) and the tool fails at startup (no fallback, by design). This is a datapath issue affecting *both* module types, independent of diagnostics. +## RX flow-steering on ixgbe -But 82599 has hardware steering machinery the current code doesn't use, and **there is good reason to think we can keep true raw-L2 ethertype steering without falling back to encapsulation.** This needs bench investigation on the actual card; do not assume the answer is UDP encap. Leads, most promising first (full analysis in open-questions.md §1): -- **L2 EtherType queue filters (ETQF/ETQS) — the answer on paper.** A dedicated ethertype→RX-queue stage checked at L2 ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588), already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto free ETQF slots so `system.go` works unchanged. Bench-confirm on the card. -- **Flow Director flex-byte match — ruled out on paper.** 82599 fdir classifies IPv4/IPv6 only; the flex word narrows an IP-typed match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding — non-IP frames never enter the fdir lookup. One-shot bench confirmation alongside the ETQF test. -- **MAC → VMDq pool steering.** 82599 assigns unicast MAC (RAR) entries to VMDq pools and pools to RX queues (`hw_set_rar` VMDq pool/queue selection). Distinct dest MACs per stream → distinct queues, fully raw Ethernet, no IP at all. Fallback if ETQF disappoints. -- **RSS on a flexible field** — 82599 RSS hashes IP tuples, not raw L2; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline). +cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW — so the insert is rejected and the tool fails at startup (no fallback, by design). This affects the datapath regardless of module type. -Fallbacks if raw-L2 steering genuinely can't be made to work: minimal bare-IPv4 framing (no UDP) steered by IP_USER_FLOW src/dst IP (lighter than the old UDP-encap plan), or single-queue RX (caps near the old ~1.6 Mpps single-NAPI ceiling, loses small-frame headroom and per-stream isolation). Encap is acceptable as a fallback (see open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice. +Leads, most promising first (full analysis in open-questions.md §1): + +- **ETQF/ETQS L2 EtherType queue filters — the answer on paper.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, 7 free in our config (index 3 reserved for 1588); already used by ixgbe for 0x88F7/FCoE/LLDP. Patch: map exact-ethertype ETHER_FLOW ntuple inserts onto ETQF slots so `system.go` is unchanged. Bench-confirm. +- **Flow Director flex-byte match — ruled out on paper.** fdir classifies IPv4/IPv6 only; the flex word narrows an IP match but can't create a non-IP one, `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) notwithstanding. +- **MAC → VMDq pool steering** — distinct dest MACs → pools → queues, fully raw. Fallback if ETQF disappoints. +- **RSS** — hashes IP tuples only; non-IP frames hash to 0 and land on queue 0. Ruled out (but explains the no-filter baseline). + +Fallbacks if raw-L2 steering can't work: minimal bare-IPv4 framing steered by IP_USER_FLOW, or single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable as a fallback (open-questions.md §1) — ETQF is still the plan, but steering does not constrain NIC choice. ## Open items -- **ECD register chapter**: the one missing document for BCM cable length. FS has been responsive; the narrow ask is "which registers invoke ECD / report the DSP cable-length estimate, and does it run with link up?" -- **Pre-FEC classification** on the Aquantia (3.e820): needs a marginal channel; cabletest's noise stress will provide one once diagnostics are integrated. -- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, 1 ft calibration cancels PHY latency); scoped but unbuilt — the committed `probe.go` is the *filter-all* variant of the same idea (raw-frame probes, needs all-packet rx stamping, so E810-only); the X710/X520 variant means reshaping probes as PTP frames. Superseded for the product if BCM ECD/DSP length pans out. -- E810 leaves the box when the X520 arrives; the patched ice + `sff_i2c` remains useful only if an E810 returns for read-side work. + +- **ECD register chapter** from FS — the one missing document for BCM cable length; the wider ask list is in modules/fs/. +- **Pre-FEC verification** on the Aquantia — counters documented (modules/fibergaga/); needs the graded-noise correlation run (open-questions.md §5). +- **X710 PTP path-delay length measurement**: viable fallback for linked-cable length (PTP-latch timestamps both ports, same oscillator, short-cable calibration); scoped but unbuilt — the committed `probe.go` is the *filter-all* variant (raw-frame probes, needs all-packet rx stamping, E810-only); the X710/X520 variant means PTP-shaped probes. Superseded for the product if PHY DSP length pans out. diff --git a/docs/transports.md b/docs/transports.md deleted file mode 100644 index 41389e7..0000000 --- a/docs/transports.md +++ /dev/null @@ -1,80 +0,0 @@ -# Module-PHY transports - -The diagnostics we want (SNR, cable length, error counters) live inside the SFP module's PHY, reachable only over the module's I2C sideband. Which NIC carries the test path decides whether the host can talk to that PHY at all. This is the single biggest architectural lever, and it was learned the hard way. - -## The rule -A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. NICs whose firmware mediates I2C impose limits that break the module command protocols. - -| NIC (driver) | Module I2C | Notes | -|---|---|---| -| **X520 / 82599 (ixgbe)** | host bit-bangs the bus — full control | `ixgbe_phy.c` wiggles I2CCTL pins directly; no firmware in the path. A debugfs patch gives arbitrary transactions. **The chosen NIC.** | -| **Wangxun WX1820 (txgbe)** | kernel `sff`/phylink — `/dev/i2c-N`, zero patches | Architecturally ideal, but no AF_XDP (irrelevant now the committed path is AF_PACKET); younger driver — flow-steering/counter parity unverified. | -| **ConnectX-4/5 (mlx5)** | firmware MCIA, but writes allowed | Offset model: RollBall fits; BCM SMI read data-phase uncertain. Has per-packet HW timestamps (would restore the timestamp length path). | -| **E810 (ice)** | multi-byte framing, but **writes EPERM-blocked** | Intel policy, blanket across all device addresses, survives NVM update. Reads fine. Dead for anything needing writes. | -| **X710 (i40e)** | 1-byte-offset I2C only | AQ 0x0628/0x0629 EXTERNAL_MODULE; reg_address caps at 0xFF. Can't frame multi-byte SMI. Writes work but framing doesn't. | - -### How the dead ends were proven -- **i40e** — 1-byte-offset only, can't frame the SMI: - - reg_address > 0xFF returns AQ retval 14 (hard boundary). - - Emits at most `[dev, offset, data]` (2 payload bytes); the BCM SMI needs 3–5 byte frames in one transaction. - - MDIO interface modes don't reach the module — no PHY on the NIC MDIO pins; copper SFPs wire only I2C. -- **E810** — frames multi-byte fine, but writes are policy-blocked: - - topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. Reads work perfectly. - - Every write returns aq 1 (EPERM), tested identically against both the EEPROM address 0x50 and the BCM address 0x56 — so the block is write-vs-read, not address-scoped. - - NVM 5.01 hardened it: the silent-drop on 3.10 became an explicit EPERM. - -## BCM84891L (FS SFP-10G-T-100) — the documented, safe path -Docs are in this folder (`10GBase-T...BCM84891.pdf` transport, `BCM84891L-MDIO Command Descriptions-.pdf` command handler). Broadcom PHY, robust — survived the whole poking campaign un-bricked, unlike the Marvell modules. - -**Transport (I2C→MDIO SMI bridge), PHY at I2C 7-bit 0x56 (8-bit 0xAC wr / 0xAD rd):** -- Clause-45 write: I2C-write to 0x56 the 5-byte frame `[000+DevAD, RegH, RegL, DataH, DataL]`. -- Clause-45 read, in two transactions: - - I2C-write `[001+DevAD, RegH, RegL]` to 0x56. - - Delay **>1 ms**. - - I2C-read 2 bytes. -- Single-byte reads see it as inert (returns 0) — the multi-byte DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead. - -**MDIO Command Handler** — status-gated handshake, inherently safe (the opposite of blind register pokes). Registers in MMD 0x1E: -- **Registers**: CMD 0x4005, STATUS 0x4037, DATA1–5 0x4038–0x403C. -- **STATUS codes**: - - CMD_RECEIVED 0x0001 - - IN_PROGRESS 0x0002 - - COMPLETE_PASS 0x0004 - - COMPLETE_ERROR 0x0008 - - SYSTEM_BUSY 0xBBBB -- **Procedure**: - 1. Poll STATUS until idle (not IN_PROGRESS/BUSY). - 2. Write params to DATAn. - 3. Write (cmdcode | bit15) to CMD. - 4. Poll STATUS for PASS/ERROR. - 5. Read DATAn for results. - - Poll ~100 ms; STATUS is frozen up to 2 s during 10GBASE-T training, so only run after link up. -- **Commands** (codes verified on hardware — the PDF's command-table columns misalign under text extraction; the check that settled it was GET_CURRENT_VOLTAGE = 0x802F returning the 0.8/1.88 V rails in tenths of mV): - - **CMD_GET_SNR = 0x8030** — invoke with **no DATA1 write**; writing the documented display flag makes this firmware return zeros. Bare invocation returns DATA2–5 = per-pair SNR (channels A–D) in tenths of dB. Bench: 27–32 dB absolute, ~0.1–0.4 dB read-to-read jitter. This is *absolute* receiver SNR, not margin (10GBASE-T operating point ≈ 26.5 dB). - - The IEEE PMA SNR registers 1.133–1.140 read a constant 0x8080 on the BCM — never populated, display flag inert. **On FS modules SNR comes from the command handler, not standard registers.** - - **CMD_GET_CURRENT_TEMP = 0x8031** (die runs ~70 °C on bench), **GET_CURRENT_VOLTAGE = 0x802F**. - - **Trap: SET commands execute stale DATA1.** The handler doesn't clear DATA registers between commands, so invoking any SET without writing its params applies leftover garbage (0x8020 SET_PAUSE_FRAME_MODE was hit this way under the old wrong GET_SNR constant — harmlessly, since pause mode is scoped to 2.5/5G idle-stuffing under XFI WAP mode and is inert at plain 10G). - - The >1 ms SMI read delay: 3 ms works reliably. - - Full command catalog with semantics: `bcm84891l-mdio-commands.md`. Probe-priority list: modules.md "What the FS/BCM gives". - - **Enhanced Cable Diagnostics** (shorts/opens/**cable length**) is a headline feature, but its invocation is NOT in the command-code table (0x8000–0x805B) — it's a separate ECD register mechanism whose chapter isn't in the docs FS sent. The FS ask list, in value order: the ECD chapter, the 1588 operation chapter (in-PHY timestamping; enable exists at 0x8004/5), datasheet §1.20 loopback (copper line loopback) and §1.17 EEE/fast-retrain monitoring, and Chapter 2 register summary — the excerpt's TOC names them all. - -**Correct BCM SMI framing over the E810 topo-I2C** (recorded in case a write-capable multi-byte transport reappears): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, then read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`. - -## RollBall (Marvell/Aquantia modules) over i40e — the proven oracle path -Stock, unpatched, works today on the X710 noise pair; the control experiment that proved our client logic correct. - -- **AQ transport** via `/sys/kernel/debug/i40e//command`: - - Command form: `send aq_cmd 0 0 0 0 0`, flags 0x2000. - - Opcode 0x0629 get / 0x0628 set PHY register. - - param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value. - - Result comes back in the dmesg "AQ desc WB" line, 9th field. - - Single-byte writes work here (unlike the E810). This 1-byte model drives RollBall (all messages ≤4 B) but cannot frame the BCM SMI. -- **RollBall sequence**: - - Unlock: password 0xFF×4 at A2h 0x7B. - - Page-select: A2h 0x7F → 3. - - Mailbox: A2h 0x80 (cmd) / 0x81 (data), poll for DONE (0x04). - - This is a clause-45 read/write of the internal PHY. On ixgbe with true multi-byte I2C it becomes the kernel's own `mdio-i2c` path — cleaner than the i40e byte-at-a-time framing. -- **Proven on the Fibergaga SFP-10G-T-30M** (Aquantia AQR, PHY ID 0x31c31c13), with link up: - - Per-pair SNR at PMA 1.133–136. - - PCS block-lock and errored-block counters at 3.32/3.33. - - `3.e820` is a clear-on-read error counter — pre- vs post-FEC unclassified, needs a marginal channel to move it.