cabletest/docs/nics/connectx-5/README.md

ConnectX-5 (mlx5) — the product NIC candidate

Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:

• CQE-stamps every RX packet (HWTSTAMP_FILTER_ALL) — the committed bucketing and probe.go run unchanged.
• One shared clock across both ports — probe.go's shared-PHC assumption holds.
• Native ETHER_FLOW ntuple steering — no driver patch, no encap needed.
• Mature driver; SFP+ modules drop into SFP28 cages at 10G.

MCIA register format (verified in kernel headers)

The public PRM documents only the data path plus the ACCESS_REGISTER envelope — MCIA's definition lives in NDA space; struct mlx5_ifc_mcia_reg_bits (include/linux/mlx5/mlx5_ifc.h) is the working reference:

• i2c_device_address is a full 8-bit field — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
• 8-bit status per access — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
• Every read is offset-addressed: 16-bit device_address + 16-bit size, 12 data dwords (48 B max per transaction). There is no raw-read encoding — a read with no preceding offset cannot even be requested.
• l lock bit — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.

BCM SMI through MCIA framing — module side proven

MCIA has no raw-read encoding (every read emits an offset write first), so SMI reads work only if the BCM bridge tolerates that framing. It does — proven on hardware (X520 bit-bang emulation of MCIA's write-offset-then-read, ~/work/phydiag-work/bcm_mcia_emu.py against the FS BCM):

• A 1-byte offset write (0x00 and 0xFF tried) injected between SMI command and read phase leaves the pending read data intact — immediately and after 3 ms; the expected register value (PHY ID 0x3590/0x5081) reads back correctly every time.
• A cold offset-write+read returns the last-latched SMI data — the offset byte is ignored, not parsed as a new command frame.
• Normal SMI operation recovers cleanly afterward; no lasting disturbance.
• Caveat: emulated with STOP-separated transactions; a repeated-START combined transfer (if that's what firmware emits) is untested.

SMI writes fit MCIA's model directly if firmware emits a 1-byte offset: an MCIA write emits [offset][data…] as one I2C write, so choosing device_address = first SMI frame byte synthesizes the 3–5 byte command frame exactly.

Open questions (decide on arrival)

• Whether firmware MCIA accepts arbitrary I2C device addresses — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (MLX5_I2C_ADDR_LOW/HIGH). The register format allows it and the per-access status byte makes rejection observable.
• Whether MCIA writes emit exactly one offset byte (needed for the SMI-frame synthesis above), and whether reads use STOP-separated or repeated-START framing.
• RollBall (0x51-resident, offset-model) fits MCIA — low risk.
• If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is not a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require (../x520/README.md).

Arrival notes

• Ports may ship in InfiniBand mode: mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2, reboot.
• Check whether auto-loading mlx5_ib interferes with channel changes (the irdma lesson, hardware.md).
• Force 10G if SFP28 autoneg is stubborn: ethtool -s <dev> speed 10000 autoneg off.
• Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.