43 lines
3.7 KiB
Markdown
43 lines
3.7 KiB
Markdown
# ConnectX-5 (mlx5) — the product NIC candidate
|
||
|
||
Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
|
||
|
||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — the committed bucketing and `probe.go` run unchanged.
|
||
- **One shared clock across both ports** — `probe.go`'s shared-PHC assumption holds.
|
||
- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
|
||
- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
|
||
|
||
## MCIA register format (verified in kernel headers)
|
||
|
||
The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope — MCIA's definition lives in NDA space; `struct mlx5_ifc_mcia_reg_bits` (`include/linux/mlx5/mlx5_ifc.h`) is the working reference:
|
||
|
||
- **`i2c_device_address` is a full 8-bit field** — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
|
||
- **8-bit `status` per access** — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
|
||
- **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested.
|
||
- **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.
|
||
|
||
## BCM SMI through MCIA framing — module side proven
|
||
|
||
MCIA has no raw-read encoding (every read emits an offset write first), so SMI reads work only if the BCM bridge tolerates that framing. **It does — proven on hardware** (X520 bit-bang emulation of MCIA's write-offset-then-read, `~/work/phydiag-work/bcm_mcia_emu.py` against the FS BCM):
|
||
|
||
- A 1-byte offset write (0x00 and 0xFF tried) injected between SMI command and read phase leaves the pending read data intact — immediately and after 3 ms; the expected register value (PHY ID 0x3590/0x5081) reads back correctly every time.
|
||
- A cold offset-write+read returns the last-latched SMI data — the offset byte is ignored, not parsed as a new command frame.
|
||
- Normal SMI operation recovers cleanly afterward; no lasting disturbance.
|
||
- Caveat: emulated with STOP-separated transactions; a repeated-START combined transfer (if that's what firmware emits) is untested.
|
||
|
||
SMI *writes* fit MCIA's model directly if firmware emits a 1-byte offset: an MCIA write emits `[offset][data…]` as one I2C write, so choosing `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly.
|
||
|
||
## Open questions (decide on arrival)
|
||
|
||
- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (`MLX5_I2C_ADDR_LOW/HIGH`). The register format allows it and the per-access status byte makes rejection observable.
|
||
- Whether MCIA writes emit exactly **one offset byte** (needed for the SMI-frame synthesis above), and whether reads use STOP-separated or repeated-START framing.
|
||
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
|
||
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
|
||
|
||
## Arrival notes
|
||
|
||
- Ports may ship in InfiniBand mode: `mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2`, reboot.
|
||
- Check whether auto-loading `mlx5_ib` interferes with channel changes (the irdma lesson, hardware.md).
|
||
- Force 10G if SFP28 autoneg is stubborn: `ethtool -s <dev> speed 10000 autoneg off`.
|
||
- Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.
|