From 4e5ba0bccfc7bfebf883eb95efc1db5c135416ef Mon Sep 17 00:00:00 2001 From: flamingcow Date: Wed, 12 Aug 2026 21:37:02 -0700 Subject: [PATCH] Sharpen MCIA open question from mlx5_ifc.h register format; drop offline-rig and instrumentation-flow framing --- docs/nics/README.md | 2 +- docs/nics/connectx-5/README.md | 15 ++++++++++++--- docs/nics/x520/README.md | 2 +- docs/open-questions.md | 6 +++--- 4 files changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/nics/README.md b/docs/nics/README.md index 2f8990b..8683ef0 100644 --- a/docs/nics/README.md +++ b/docs/nics/README.md @@ -22,4 +22,4 @@ Requirement weights behind the verdicts: all-packet hardware RX timestamps are n ## Architecture - **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot. -- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules and the X520 stays an offline BCM rig ([connectx-5](connectx-5/README.md)). +- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules only ([connectx-5](connectx-5/README.md)). diff --git a/docs/nics/connectx-5/README.md b/docs/nics/connectx-5/README.md index fa71a41..b139f0e 100644 --- a/docs/nics/connectx-5/README.md +++ b/docs/nics/connectx-5/README.md @@ -7,12 +7,21 @@ Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set: - **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed. - Mature driver; SFP+ modules drop into SFP28 cages at 10G. +## MCIA register format (verified in kernel headers) + +The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope — MCIA's definition lives in NDA space; `struct mlx5_ifc_mcia_reg_bits` (`include/linux/mlx5/mlx5_ifc.h`) is the working reference: + +- **`i2c_device_address` is a full 8-bit field** — 0x56 is expressible in the register; the open part is purely whether firmware accepts it. +- **8-bit `status` per access** — firmware reports the result, so an address rejection or I2C failure is observable, not silent. +- **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested. +- **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem. + ## Open questions (decide on arrival) -- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51. -- Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model. +- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (`MLX5_I2C_ADDR_LOW/HIGH`). +- Whether the BCM SMI **read data phase** survives MCIA's framing. The format settles half the question: a raw no-offset read has no encoding, so the only path is the bridge *tolerating* an offset write before the read phase. That is a module-side property — **testable now on the X520** by bit-bang-emulating MCIA's write-offset-then-read framing against the FS BCM, before the card arrives. - RollBall (0x51-resident, offset-model) fits MCIA — low risk. -- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)). +- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)). ## Arrival notes diff --git a/docs/nics/x520/README.md b/docs/nics/x520/README.md index 839cbeb..656496a 100644 --- a/docs/nics/x520/README.md +++ b/docs/nics/x520/README.md @@ -28,7 +28,7 @@ For the timestamp/counter bench pokes the driver also carries `reg_ops` `read