The diagnostics we want (SNR, cable length, error counters) live inside the SFP module's PHY, reachable only over the module's I2C sideband. Which NIC carries the test path decides whether the host can talk to that PHY at all. This is the single biggest architectural lever, and it was learned the hard way.
## The rule
A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. NICs whose firmware mediates I2C impose limits that break the module command protocols.
| NIC (driver) | Module I2C | Notes |
|---|---|---|
| **X520 / 82599 (ixgbe)** | host bit-bangs the bus — full control | `ixgbe_phy.c` wiggles I2CCTL pins directly; no firmware in the path. A debugfs patch gives arbitrary transactions. **The chosen NIC.** |
| **Wangxun WX1820 (txgbe)** | kernel `sff`/phylink — `/dev/i2c-N`, zero patches | Architecturally ideal, but no AF_XDP (irrelevant now the committed path is AF_PACKET); younger driver — flow-steering/counter parity unverified. |
| **ConnectX-4/5 (mlx5)** | firmware MCIA, but writes allowed | Offset model: RollBall fits; BCM SMI read data-phase uncertain. Has per-packet HW timestamps (would restore the timestamp length path). |
| **E810 (ice)** | multi-byte framing, but **writes EPERM-blocked** | Intel policy, blanket across all device addresses, survives NVM update. Reads fine. Dead for anything needing writes. |
| **X710 (i40e)** | 1-byte-offset I2C only | AQ 0x0628/0x0629 EXTERNAL_MODULE; reg_address caps at 0xFF. Can't frame multi-byte SMI. Writes work but framing doesn't. |
### How the dead ends were proven
- **i40e** — 1-byte-offset only, can't frame the SMI:
- reg_address > 0xFF returns AQ retval 14 (hard boundary).
- Emits at most `[dev, offset, data]` (2 payload bytes); the BCM SMI needs 3–5 byte frames in one transaction.
- MDIO interface modes don't reach the module — no PHY on the NIC MDIO pins; copper SFPs wire only I2C.
- **E810** — frames multi-byte fine, but writes are policy-blocked:
- topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. Reads work perfectly.
- Every write returns aq 1 (EPERM), tested identically against both the EEPROM address 0x50 and the BCM address 0x56 — so the block is write-vs-read, not address-scoped.
- NVM 5.01 hardened it: the silent-drop on 3.10 became an explicit EPERM.
## BCM84891L (FS SFP-10G-T-100) — the documented, safe path
Docs are in this folder (`10GBase-T...BCM84891.pdf` transport, `BCM84891L-MDIO Command Descriptions-.pdf` command handler). Broadcom PHY, robust — survived the whole poking campaign un-bricked, unlike the Marvell modules.
- Clause-45 write: I2C-write to 0x56 the 5-byte frame `[000+DevAD, RegH, RegL, DataH, DataL]`.
- Clause-45 read, in two transactions:
- I2C-write `[001+DevAD, RegH, RegL]` to 0x56.
- Delay **>1 ms**.
- I2C-read 2 bytes.
- Single-byte reads see it as inert (returns 0) — the multi-byte DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead.
**MDIO Command Handler** — status-gated handshake, inherently safe (the opposite of blind register pokes). Registers in MMD 0x1E:
- **Registers**: CMD 0x4005, STATUS 0x4037, DATA1–5 0x4038–0x403C.
- **STATUS codes**:
- CMD_RECEIVED 0x0001
- IN_PROGRESS 0x0002
- COMPLETE_PASS 0x0004
- COMPLETE_ERROR 0x0008
- SYSTEM_BUSY 0xBBBB
- **Procedure**:
1. Poll STATUS until idle (not IN_PROGRESS/BUSY).
2. Write params to DATAn.
3. Write (cmdcode | bit15) to CMD.
4. Poll STATUS for PASS/ERROR.
5. Read DATAn for results.
- Poll ~100 ms; STATUS is frozen up to 2 s during 10GBASE-T training, so only run after link up.
- **Commands** (codes verified on hardware — the PDF's command-table columns misalign under text extraction; the check that settled it was GET_CURRENT_VOLTAGE = 0x802F returning the 0.8/1.88 V rails in tenths of mV):
- **CMD_GET_SNR = 0x8030** — invoke with **no DATA1 write**; writing the documented display flag makes this firmware return zeros. Bare invocation returns DATA2–5 = per-pair SNR (channels A–D) in tenths of dB. Bench: 27–32 dB absolute, ~0.1–0.4 dB read-to-read jitter. This is *absolute* receiver SNR, not margin (10GBASE-T operating point ≈ 26.5 dB).
- The IEEE PMA SNR registers 1.133–1.140 read a constant 0x8080 on the BCM — never populated, display flag inert. **On FS modules SNR comes from the command handler, not standard registers.**
- **CMD_GET_CURRENT_TEMP = 0x8031** (die runs ~70 °C on bench), **GET_CURRENT_VOLTAGE = 0x802F**.
- **Trap: SET commands execute stale DATA1.** The handler doesn't clear DATA registers between commands, so invoking any SET without writing its params applies leftover garbage (0x8020 SET_PAUSE_FRAME_MODE was hit this way under the old wrong GET_SNR constant — harmlessly, since pause mode is scoped to 2.5/5G idle-stuffing under XFI WAP mode and is inert at plain 10G).
- Full command catalog with semantics: `bcm84891l-mdio-commands.md`. Probe-priority list: modules.md "What the FS/BCM gives".
- **Enhanced Cable Diagnostics** (shorts/opens/**cable length**) is a headline feature, but its invocation is NOT in the command-code table (0x8000–0x805B) — it's a separate ECD register mechanism whose chapter isn't in the docs FS sent. The FS ask list, in value order: the ECD chapter, the 1588 operation chapter (in-PHY timestamping; enable exists at 0x8004/5), datasheet §1.20 loopback (copper line loopback) and §1.17 EEE/fast-retrain monitoring, and Chapter 2 register summary — the excerpt's TOC names them all.
**Correct BCM SMI framing over the E810 topo-I2C** (recorded in case a write-capable multi-byte transport reappears): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, then read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`.
## RollBall (Marvell/Aquantia modules) over i40e — the proven oracle path
Stock, unpatched, works today on the X710 noise pair; the control experiment that proved our client logic correct.
- **AQ transport** via `/sys/kernel/debug/i40e/<pci>/command`:
- This is a clause-45 read/write of the internal PHY. On ixgbe with true multi-byte I2C it becomes the kernel's own `mdio-i2c` path — cleaner than the i40e byte-at-a-time framing.
- **Proven on the Fibergaga SFP-10G-T-30M** (Aquantia AQR, PHY ID 0x31c31c13), with link up:
- Per-pair SNR at PMA 1.133–136.
- PCS block-lock and errored-block counters at 3.32/3.33.
-`3.e820` is a clear-on-read error counter — pre- vs post-FEC unclassified, needs a marginal channel to move it.
