Sharpen MCIA open question from mlx5_ifc.h register format; drop offline-rig and instrumentation-flow framing

This commit is contained in:
flamingcow
2026-08-12 21:37:02 -07:00
parent 65b1e18004
commit 4e5ba0bccf
4 changed files with 17 additions and 8 deletions
+1 -1
View File
@@ -22,4 +22,4 @@ Requirement weights behind the verdicts: all-packet hardware RX timestamps are n
## Architecture
- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules and the X520 stays an offline BCM rig ([connectx-5](connectx-5/README.md)).
- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules only ([connectx-5](connectx-5/README.md)).
+12 -3
View File
@@ -7,12 +7,21 @@ Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
## MCIA register format (verified in kernel headers)
The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope — MCIA's definition lives in NDA space; `struct mlx5_ifc_mcia_reg_bits` (`include/linux/mlx5/mlx5_ifc.h`) is the working reference:
- **`i2c_device_address` is a full 8-bit field** — 0x56 is expressible in the register; the open part is purely whether firmware accepts it.
- **8-bit `status` per access** — firmware reports the result, so an address rejection or I2C failure is observable, not silent.
- **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested.
- **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem.
## Open questions (decide on arrival)
- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
- Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model.
- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (`MLX5_I2C_ADDR_LOW/HIGH`).
- Whether the BCM SMI **read data phase** survives MCIA's framing. The format settles half the question: a raw no-offset read has no encoding, so the only path is the bridge *tolerating* an offset write before the read phase. That is a module-side property — **testable now on the X520** by bit-bang-emulating MCIA's write-offset-then-read framing against the FS BCM, before the card arrives.
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)).
## Arrival notes
+1 -1
View File
@@ -28,7 +28,7 @@ For the timestamp/counter bench pokes the driver also carries `reg_ops` `read <r
- TX latch has no parser (§8.2.3.26.7): the advanced TX data descriptor's 1588 bit (MAC field, bit 1) stamps **any** marked frame into TXSTMPL/H — serial, TXTT cleared by TXSTMPH read. Both directions therefore stamp arbitrary frames one-at-a-time: a ping-pong probe gets PHY-adjacent hardware stamps on both ends, paced by the MMIO read loop.
- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial, so it stamps a fraction of frames at best and none of the line-rate traffic. NIC-timestamp length measurement is discarded (length strategy: [../../modules/README.md](../../modules/README.md)), so the latch has no committed use.
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal; the two SYSTIMEs start at driver load and can sit seconds apart.
- SDP auxiliary elements (§7.9.4): an edge on SDP2/3 latches SYSTIME into AUXSTMP0/1 (two external-event timestamp channels); target-time registers assert an SDP pin when SYSTIME reaches a programmed value; SDP6/7 carry configurable clock outputs — hooks for external instrumentation on the same clock that stamps packets.
- SDP auxiliary elements (§7.9.4): an edge on SDP2/3 latches SYSTIME into AUXSTMP0/1; target-time registers assert an SDP pin when SYSTIME reaches a programmed value; SDP6/7 carry configurable clock outputs. No product use.
- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not — it cannot place a single frame in time, so it cannot see the fault. Measured facts retained for the record: RQSMR maps queue→counter 1:1; QPRC/QBRC exact, read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (`ixgbe_main.c:7780,7789,7822`). Also rejected: wire-determinism reconstruction (per-frame times from the size schedule assume saturation regularity — blind exactly at loss/retrain events).
## RX steering facts