# Module-PHY transports

The diagnostics we want (SNR, cable length, error counters) live inside the SFP module's PHY, reachable only over the module's I2C sideband. Which NIC carries the test path decides whether the host can talk to that PHY at all. This is the single biggest architectural lever, and it was learned the hard way.

## The rule
A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. NICs whose firmware mediates I2C impose limits that break the module command protocols.

| NIC (driver) | Module I2C | Notes |
|---|---|---|
| **X520 / 82599 (ixgbe)** | host bit-bangs the bus — full control | `ixgbe_phy.c` wiggles I2CCTL pins directly; no firmware in the path. A debugfs patch gives arbitrary transactions. **The chosen NIC.** |
| **Wangxun WX1820 (txgbe)** | kernel `sff`/phylink — `/dev/i2c-N`, zero patches | Architecturally ideal, but no AF_XDP (irrelevant now the committed path is AF_PACKET); younger driver — flow-steering/counter parity unverified. |
| **ConnectX-4/5 (mlx5)** | firmware MCIA, but writes allowed | Offset model: RollBall fits; BCM SMI read data-phase uncertain. Has per-packet HW timestamps (would restore the timestamp length path). |
| **E810 (ice)** | multi-byte framing, but **writes EPERM-blocked** | Intel policy, blanket across all device addresses, survives NVM update. Reads fine. Dead for anything needing writes. |
| **X710 (i40e)** | 1-byte-offset I2C only | AQ 0x0628/0x0629 EXTERNAL_MODULE; reg_address caps at 0xFF. Can't frame multi-byte SMI. Writes work but framing doesn't. |

### How the dead ends were proven
- **i40e** — 1-byte-offset only, can't frame the SMI:
  - reg_address > 0xFF returns AQ retval 14 (hard boundary).
  - Emits at most `[dev, offset, data]` (2 payload bytes); the BCM SMI needs 3–5 byte frames in one transaction.
  - MDIO interface modes don't reach the module — no PHY on the NIC MDIO pins; copper SFPs wire only I2C.
- **E810** — frames multi-byte fine, but writes are policy-blocked:
  - topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. Reads work perfectly.
  - Every write returns aq 1 (EPERM), tested identically against both the EEPROM address 0x50 and the BCM address 0x56 — so the block is write-vs-read, not address-scoped.
  - NVM 5.01 hardened it: the silent-drop on 3.10 became an explicit EPERM.

## BCM84891L (FS SFP-10G-T-100) — the documented, safe path
Docs are in this folder (`10GBase-T...BCM84891.pdf` transport, `BCM84891L-MDIO Command Descriptions-.pdf` command handler). Broadcom PHY, robust — survived the whole poking campaign un-bricked, unlike the Marvell modules.

**Transport (I2C→MDIO SMI bridge), PHY at I2C 7-bit 0x56 (8-bit 0xAC wr / 0xAD rd):**
- Clause-45 write: I2C-write to 0x56 the 5-byte frame `[000+DevAD, RegH, RegL, DataH, DataL]`.
- Clause-45 read, in two transactions:
  - I2C-write `[001+DevAD, RegH, RegL]` to 0x56.
  - Delay **>1 ms**.
  - I2C-read 2 bytes.
- Single-byte reads see it as inert (returns 0) — the multi-byte DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead.

**MDIO Command Handler** — status-gated handshake, inherently safe (the opposite of blind register pokes). Registers in MMD 0x1E:
- **Registers**: CMD 0x4005, STATUS 0x4037, DATA1–5 0x4038–0x403C.
- **STATUS codes**:
  - CMD_RECEIVED 0x0001
  - IN_PROGRESS 0x0002
  - COMPLETE_PASS 0x0004
  - COMPLETE_ERROR 0x0008
  - SYSTEM_BUSY 0xBBBB
- **Procedure**:
  1. Poll STATUS until idle (not IN_PROGRESS/BUSY).
  2. Write params to DATAn.
  3. Write (cmdcode | bit15) to CMD.
  4. Poll STATUS for PASS/ERROR.
  5. Read DATAn for results.
  - Poll ~100 ms; STATUS is frozen up to 2 s during 10GBASE-T training, so only run after link up.
- **Commands**:
  - **CMD_GET_SNR** (GET commands 0x8000+; detail §1.25.1.24): DATA2/10..DATA5/10 = per-pair SNR dB (channels A–D).
  - Also pair swap/skew/polarity, die temp, voltage, 1588.
  - **Enhanced Cable Diagnostics** (shorts/opens/**cable length**) is a headline feature, but its invocation is NOT in the command-code table (0x8000–0x805B) — it's a separate ECD register mechanism whose chapter isn't in the docs FS sent. This is the one outstanding ask to FS.

**Correct BCM SMI framing over the E810 topo-I2C** (recorded in case a write-capable multi-byte transport reappears): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, then read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`.

## RollBall (Marvell/Aquantia modules) over i40e — the proven oracle path
Stock, unpatched, works today on the X710 noise pair; the control experiment that proved our client logic correct.

- **AQ transport** via `/sys/kernel/debug/i40e/<pci>/command`:
  - Command form: `send aq_cmd <flags> <opcode> 0 0 0 0 <param0> <param1> <param2> 0`, flags 0x2000.
  - Opcode 0x0629 get / 0x0628 set PHY register.
  - param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value.
  - Result comes back in the dmesg "AQ desc WB" line, 9th field.
  - Single-byte writes work here (unlike the E810). This 1-byte model drives RollBall (all messages ≤4 B) but cannot frame the BCM SMI.
- **RollBall sequence**:
  - Unlock: password 0xFF×4 at A2h 0x7B.
  - Page-select: A2h 0x7F → 3.
  - Mailbox: A2h 0x80 (cmd) / 0x81 (data), poll for DONE (0x04).
  - This is a clause-45 read/write of the internal PHY. On ixgbe with true multi-byte I2C it becomes the kernel's own `mdio-i2c` path — cleaner than the i40e byte-at-a-time framing.
- **Proven on the Fibergaga SFP-10G-T-30M** (Aquantia AQR, PHY ID 0x31c31c13), with link up:
  - Per-pair SNR at PMA 1.133–136.
  - PCS block-lock and errored-block counters at 3.32/3.33.
  - `3.e820` is a clear-on-read error counter — pre- vs post-FEC unclassified, needs a marginal channel to move it.