ConnectX-5 (mlx5) — the product NIC candidate
Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
- CQE-stamps every RX packet (
HWTSTAMP_FILTER_ALL) — the committed bucketing andprobe.gorun unchanged. - One shared clock across both ports —
probe.go's shared-PHC assumption holds. - Native ETHER_FLOW ntuple steering — no driver patch, no encap needed.
- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
MCIA register format (verified in kernel headers)
The public PRM documents only the data path plus the ACCESS_REGISTER envelope — MCIA's definition lives in NDA space; struct mlx5_ifc_mcia_reg_bits (include/linux/mlx5/mlx5_ifc.h) is the working reference:
i2c_device_addressis a full 8-bit field — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.- 8-bit
statusper access — firmware reports the result, so an address rejection or I2C failure is observable, not silent. - Every read is offset-addressed: 16-bit
device_address+ 16-bitsize, 12 data dwords (48 B max per transaction). There is no raw-read encoding — a read with no preceding offset cannot even be requested. llock bit — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.
Open questions (decide on arrival)
- Whether firmware MCIA accepts arbitrary I2C device addresses — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (
MLX5_I2C_ADDR_LOW/HIGH). - Whether the BCM SMI read data phase survives MCIA's framing. The format settles half the question: a raw no-offset read has no encoding, so the only path is the bridge tolerating an offset write before the read phase. That is a module-side property — testable now on the X520 by bit-bang-emulating MCIA's write-offset-then-read framing against the FS BCM, before the card arrives.
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is not a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require (../x520/README.md).
Arrival notes
- Ports may ship in InfiniBand mode:
mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2, reboot. - Check whether auto-loading
mlx5_ibinterferes with channel changes (the irdma lesson, hardware.md). - Force 10G if SFP28 autoneg is stubborn:
ethtool -s <dev> speed 10000 autoneg off. - Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.