Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
This commit is contained in:
flamingcow
@@ -0,0 +1,21 @@
|
||||
# NICs
|
||||
|
||||
The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path gets the hardware timestamps it requires. Per-device directories hold the details.
|
||||
|
||||
## The transport rule
|
||||
|
||||
A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols.
|
||||
|
||||
## Comparison
|
||||
|
||||
Which card is where: [../state.md](../state.md).
|
||||
|
||||
| NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict |
|
||||
|---|---|---|---|
|
||||
| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only) | The certain diagnostics transport; product-capable only if the bucket exploration validates |
|
||||
| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set |
|
||||
| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform |
|
||||
| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes |
|
||||
| [WX1820 (txgbe)](wx1820/README.md) | Kernel `/dev/i2c-N` — ideal | No (PTP-only) | Disqualified |
|
||||
|
||||
Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable) — [../open-questions.md](../open-questions.md) §1–2, §6.
|
||||
@@ -0,0 +1,22 @@
|
||||
# ConnectX-5 (mlx5) — the product NIC candidate
|
||||
|
||||
Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
|
||||
|
||||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — the committed bucketing and `probe.go` run unchanged.
|
||||
- **One shared clock across both ports** — `probe.go`'s shared-PHC assumption holds.
|
||||
- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
|
||||
- Mature driver; SFP+ modules drop into SFP28 cages at 10G.
|
||||
|
||||
## Open questions (decide on arrival)
|
||||
|
||||
- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
|
||||
- Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model.
|
||||
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
|
||||
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig — or the X520 as fallback product NIC if the bucket exploration validates ([../../open-questions.md](../../open-questions.md) §2, §6).
|
||||
|
||||
## Arrival notes
|
||||
|
||||
- Ports may ship in InfiniBand mode: `mlxconfig -d <pci> set LINK_TYPE_P1=2 LINK_TYPE_P2=2`, reboot.
|
||||
- Check whether auto-loading `mlx5_ib` interferes with channel changes (the irdma lesson, hardware.md).
|
||||
- Force 10G if SFP28 autoneg is stubborn: `ethtool -s <dev> speed 10000 autoneg off`.
|
||||
- Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Intel E810 (ice)
|
||||
|
||||
The NIC the committed measurement path was built against: its datapath delivers per-packet MAC RX timestamps (`rx_filter=ALL`), which bucketing and `probe.go` require. A patched ice + `sff_i2c` exists for read-side diagnostics work.
|
||||
|
||||
## Module I2C: multi-byte framing works, writes are policy-blocked
|
||||
|
||||
- topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. **Reads work perfectly.**
|
||||
- Every write returns aq 1 (EPERM) — tested identically against EEPROM 0x50 and BCM 0x56, so the block is write-vs-read, not address-scoped. Intel policy, survives NVM update; NVM 5.01 hardened the 3.10 silent-drop into an explicit EPERM.
|
||||
- Dead for anything needing writes: RollBall unlock needs a write; BCM SMI reads *start* with a write.
|
||||
@@ -0,0 +1,5 @@
|
||||
# Wangxun WX1820 (txgbe) — evaluated, disqualified
|
||||
|
||||
- Module I2C is architecturally ideal: kernel `sff`/phylink exposes `/dev/i2c-N` — arbitrary framing, zero patches.
|
||||
- **Disqualified as product NIC**: PTP-only RX timestamping (verified in upstream `wx_ptp.c` — `HWTSTAMP_FILTER_ALL` falls to -ERANGE), the same limitation as the 82599 with a younger driver.
|
||||
- Flow-steering/counter parity unverified.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Intel X520-DA2 (82599, ixgbe)
|
||||
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse. Cannot run the committed measurement path — PTP-only RX timestamping ([../../open-questions.md](../../open-questions.md) §2) — unless the bucket-timestamping exploration validates an equivalent.
|
||||
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s) — enough for 2×10G full duplex at the default mix.
|
||||
- **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
|
||||
|
||||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||||
|
||||
Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
|
||||
|
||||
- **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
|
||||
- **Two-master windows**: each debugfs op takes the semaphore separately; serialization against the driver's own I2C holds per-op, but multi-op protocol sequences have unlocked gaps, and driver I2C is event-driven (SFP identify, link setup, `ethtool -m`) — firing exactly around cable swaps. Hardening item: a compound op under one semaphore hold ([../../open-questions.md](../../open-questions.md) §4).
|
||||
|
||||
## Timestamping facts
|
||||
|
||||
- RX latches **PTP frames only** (`TYPE_ALL`/TSIP are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`): one latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H), no RX-descriptor timestamp field.
|
||||
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset.
|
||||
- Per-queue hardware counters QPRC/QBRC (16 slots via RQSMR, **read-clear**, `ixgbe_main.c:7777,7787`) count at queue write-back, upstream of ring residence — the basis of bucket-exploration route 1.
|
||||
- Full analysis and bench pokes: [../../open-questions.md](../../open-questions.md) §2.
|
||||
|
||||
## RX steering facts
|
||||
|
||||
- ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only) — the tool fails at startup as committed.
|
||||
- **ETQF/ETQS L2 EtherType filters are the answer on paper**: dedicated ethertype→queue stage ahead of RSS/fdir, 8 slots, 7 free (index 3 reserved for 1588). Flow Director cannot match non-IP frames at all. Analysis and patch shape: [../../open-questions.md](../../open-questions.md) §1.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Intel X710 (i40e)
|
||||
|
||||
## Module I2C: 1-byte-offset only — dead end for BCM SMI
|
||||
|
||||
- `reg_address > 0xFF` returns AQ retval 14 (hard boundary).
|
||||
- Emits at most `[dev, offset, data]` (2 payload bytes); the BCM SMI needs 3–5 byte frames in one transaction.
|
||||
- MDIO interface modes don't reach the module — no PHY on the NIC MDIO pins; copper SFPs wire only I2C.
|
||||
- Single-byte **writes work** (unlike the E810) — which is what makes the oracle path below possible.
|
||||
|
||||
## AQ oracle path (stock driver) — proved the RollBall client logic
|
||||
|
||||
- Transport: `/sys/kernel/debug/i40e/<pci>/command`, `send aq_cmd <flags> <opcode> 0 0 0 0 <param0> <param1> <param2> 0`, flags 0x2000.
|
||||
- Opcode 0x0629 get / 0x0628 set PHY register. param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value.
|
||||
- Result comes back in the dmesg "AQ desc WB" line, 9th field.
|
||||
- Proven on the Fibergaga, link up: per-pair SNR (PMA 1.133–1.136), PCS 3.32/3.33, the 3.E820 counter ([../../modules/fibergaga/](../../modules/fibergaga/README.md)).
|
||||
|
||||
## Timestamping
|
||||
|
||||
- Rejects `HWTSTAMP_FILTER_ALL` with -ERANGE (verified in upstream `i40e_ptp.c`) — PTP-only, so no committed-path bucketing here.
|
||||
- Genuinely shared PHC across ports (one oscillator) + PTP latches: the platform for the retrain-stability experiment and the scoped-but-unbuilt PTP path-delay length fallback ([../../open-questions.md](../../open-questions.md) §2) — probes must be PTP-shaped here too.
|
||||
- No EEE on i40e/X710.
|
||||
Reference in New Issue
Block a user