28 lines
3.5 KiB
Markdown
28 lines
3.5 KiB
Markdown
# NICs
|
||
|
||
The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path can report an honest rate. Per-device directories hold the details.
|
||
|
||
## The transport rule
|
||
|
||
A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols (E810: writes blocked outright; CX-5: writes work but the firmware's own module traffic quantizes every mailbox read to ~150 ms steps).
|
||
|
||
## The timestamp requirement is dissolved
|
||
|
||
All-packet hardware RX timestamps existed for one consumer — bucketing received frames by arrival time so the displayed rate stayed honest under host read-jitter. The smoothed-bucket plan ([x520](x520/README.md)) computes the rate from read-time buckets with a backward excess-fill instead, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. The audit confirmed the rate buckets were the stamps' sole consumer, and the committed tree implements the plan; the capability that disqualified the X520 — and the only thing the firmware-managed NICs offered over it — is not required. Raw-L2 steering was never a hard requirement either (encap acceptable).
|
||
|
||
## Comparison
|
||
|
||
Which card is where: [../state.md](../state.md).
|
||
|
||
| NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict |
|
||
|---|---|---|---|
|
||
| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven**, fast (<25 ms RollBall) | No (single serial latch; 1588-event frames only) — no longer required | **The product NIC** under the smoothed-bucket rate plan: only card with unmediated host I2C, and the FS links here (mixed FS+Wiitek pair proven) |
|
||
| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA — writes via VSC/ICMD only (fwctl read-only), every mailbox read quantized to ~150 ms steps by firmware module traffic | Yes (CQE-stamps everything) | Retired: its stamps are no longer needed and its I2C mediation is the cost; the FS never links on it either (host-interface deadlock, device page) |
|
||
| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; noise pair |
|
||
| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes; the below-the-AQ bit-bang idea (device page) is retired with the timestamp requirement |
|
||
|
||
## Architecture
|
||
|
||
- **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot.
|
||
- **The single card is the X520.** With the timestamp requirement dissolved, the selection weights return to where the X520 originally won: unmediated host I2C is the one capability no firmware-managed card provides (E810 blocks writes; mlx5 mediates and quantizes), and it is non-negotiable for the module protocols. Gen2 ×8 (32 Gb/s raw) clears 2×10G full duplex; ETQF steering and loss attribution are proven on hardware. The general lesson, learned twice: NICs that stamp every packet are all firmware-managed, and firmware in the module-I2C path is what breaks or slows the diagnostics — the two headline capabilities appear to be mutually exclusive off the shelf.
|