ConnectX-5 (mlx5) — the product NIC candidate
Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
- CQE-stamps every RX packet (
HWTSTAMP_FILTER_ALL) — the committed bucketing andprobe.gorun unchanged. - One shared clock across both ports —
probe.go's shared-PHC assumption holds. - Native ETHER_FLOW ntuple steering — no driver patch, no encap needed.
- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
Open questions (decide on arrival)
- Whether firmware MCIA accepts arbitrary I2C device addresses — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
- Whether the BCM SMI read data phase (raw 2-byte read, no offset write) can be framed through MCIA's offset model.
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig — or the X520 as fallback product NIC if the bucket exploration validates (../../open-questions.md §2, §6).
Arrival notes
- Ports may ship in InfiniBand mode:
mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2, reboot. - Check whether auto-loading
mlx5_ibinterferes with channel changes (the irdma lesson, hardware.md). - Force 10G if SFP28 autoneg is stubborn:
ethtool -s <dev> speed 10000 autoneg off. - Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.