cabletest/docs/nics/connectx-5/README.md

ConnectX-5 (mlx5) — the product NIC candidate

Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:

• CQE-stamps every RX packet (HWTSTAMP_FILTER_ALL) — the committed bucketing and probe.go run unchanged.
• One shared clock across both ports — probe.go's shared-PHC assumption holds.
• Native ETHER_FLOW ntuple steering — no driver patch, no encap needed.
• Mature driver; SFP+ modules drop into SFP28 cages at 10G.

MCIA register format (verified in kernel headers)

The public PRM documents only the data path plus the ACCESS_REGISTER envelope — MCIA's definition lives in NDA space; struct mlx5_ifc_mcia_reg_bits (include/linux/mlx5/mlx5_ifc.h) is the working reference:

• i2c_device_address is a full 8-bit field — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
• 8-bit status per access — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
• Every read is offset-addressed: 16-bit device_address + 16-bit size, 12 data dwords (48 B max per transaction). There is no raw-read encoding — a read with no preceding offset cannot even be requested.
• l lock bit — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.

Open questions (decide on arrival)

• Whether firmware MCIA accepts arbitrary I2C device addresses — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (MLX5_I2C_ADDR_LOW/HIGH).
• Whether the BCM SMI read data phase survives MCIA's framing. The format settles half the question: a raw no-offset read has no encoding, so the only path is the bridge tolerating an offset write before the read phase. That is a module-side property — testable now on the X520 by bit-bang-emulating MCIA's write-offset-then-read framing against the FS BCM, before the card arrives.
• RollBall (0x51-resident, offset-model) fits MCIA — low risk.
• If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is not a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require (../x520/README.md).

Arrival notes

• Ports may ship in InfiniBand mode: mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2, reboot.
• Check whether auto-loading mlx5_ib interferes with channel changes (the irdma lesson, hardware.md).
• Force 10G if SFP28 autoneg is stubborn: ethtool -s <dev> speed 10000 autoneg off.
• Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.