Resolve X520 open questions on hardware: ETQF raw-L2 steering and single-hold compound I2C proven; 82599 has no all-packet exact-timestamp path so it is disqualified as product NIC (diagnostics-only), counter route rejected for a cable tester
This commit is contained in:
flamingcow
+1
-1
@@ -12,7 +12,7 @@ Which card is where: [../state.md](../state.md).
|
||||
|
||||
| NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict |
|
||||
|---|---|---|---|
|
||||
| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only) | The certain diagnostics transport; product-capable only if the bucket exploration validates |
|
||||
| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (PTP-only, serial latch) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC (§2) |
|
||||
| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set |
|
||||
| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform |
|
||||
| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes |
|
||||
|
||||
@@ -12,7 +12,7 @@ Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set:
|
||||
- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51.
|
||||
- Whether the BCM SMI **read data phase** (raw 2-byte read, no offset write) can be framed through MCIA's offset model.
|
||||
- RollBall (0x51-resident, offset-model) fits MCIA — low risk.
|
||||
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig — or the X520 as fallback product NIC if the bucket exploration validates ([../../open-questions.md](../../open-questions.md) §2, §6).
|
||||
- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only, X520 kept as an offline BCM rig. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../../open-questions.md](../../open-questions.md) §2, §6).
|
||||
|
||||
## Arrival notes
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Intel X520-DA2 (82599, ixgbe)
|
||||
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse. Cannot run the committed measurement path — PTP-only RX timestamping ([../../open-questions.md](../../open-questions.md) §2) — unless the bucket-timestamping exploration validates an equivalent.
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial PTP-only latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault ([../../open-questions.md](../../open-questions.md) §2). The ConnectX-5 is the product NIC.
|
||||
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s) — enough for 2×10G full duplex at the default mix.
|
||||
- **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it.
|
||||
@@ -10,16 +10,22 @@ The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmwa
|
||||
Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
|
||||
|
||||
- **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
|
||||
- **Two-master windows**: each debugfs op takes the semaphore separately; serialization against the driver's own I2C holds per-op, but multi-op protocol sequences have unlocked gaps, and driver I2C is event-driven (SFP identify, link setup, `ethtool -m`) — firing exactly around cable swaps. Hardening item: a compound op under one semaphore hold ([../../open-questions.md](../../open-questions.md) §4).
|
||||
- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→delay→read under a single swfw hold, removing the window. Proven on the FS BCM bridge ([../../open-questions.md](../../open-questions.md) §4).
|
||||
|
||||
## Register/timestamp bench transport (patched driver)
|
||||
|
||||
For the §2 exploration the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
|
||||
|
||||
## Timestamping facts
|
||||
|
||||
- RX latches **PTP frames only** (`TYPE_ALL`/TSIP are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`): one latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H), no RX-descriptor timestamp field.
|
||||
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset.
|
||||
- Per-queue hardware counters QPRC/QBRC (16 slots via RQSMR, **read-clear**, `ixgbe_main.c:7777,7787`) count at queue write-back, upstream of ring residence — the basis of bucket-exploration route 1.
|
||||
- Full analysis and bench pokes: [../../open-questions.md](../../open-questions.md) §2.
|
||||
**No all-packet exact RX timestamping, and no path to it — this disqualifies the X520 as product NIC.** A cable tester needs each frame's exact arrival time; the 82599 cannot give it for the 0x88b5 test traffic.
|
||||
|
||||
- RX latches **PTP frames only** — confirmed on hardware. Writing `TYPE_ALL` (0x08) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood; `TSIP_UT_EN` (0x00800000) will not even hold (both are X550+; `HWTSTAMP_FILTER_ALL` → -ERANGE, `ixgbe_ptp.c:1043`). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
|
||||
- The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial and PTP-only, so it stamps a fraction of frames at best and none of the raw traffic. Its only surviving use is the low-volume length probes (§2/§3).
|
||||
- Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal — two-way exchange cancels the inter-port offset (length-probe use only).
|
||||
- **Per-queue counters are not a substitute**: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not. Measured facts (RQSMR maps 1:1; QPRC/QBRC exact, read-clear; the 2 s stats watchdog is a competing reader, `ixgbe_main.c:7780,7789,7822`) are kept in [../../open-questions.md](../../open-questions.md) §2, but the route is rejected.
|
||||
|
||||
## RX steering facts
|
||||
|
||||
- ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only) — the tool fails at startup as committed.
|
||||
- **ETQF/ETQS L2 EtherType filters are the answer on paper**: dedicated ethertype→queue stage ahead of RSS/fdir, 8 slots, 7 free (index 3 reserved for 1588). Flow Director cannot match non-IP frames at all. Analysis and patch shape: [../../open-questions.md](../../open-questions.md) §1.
|
||||
- Stock ntuple ETHER_FLOW is rejected (`ixgbe_flowspec_to_flow_type`: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged.
|
||||
- **ETQF/ETQS L2 EtherType steering is proven on hardware**: dedicated ethertype→queue stage ahead of RSS/fdir; 7 streams land on 7 queues, unfiltered ethertypes fall to queue 0. Flow Director cannot match non-IP frames at all (a flex-word rule on raw 0x88b5 steers nothing — confirmed). Details and the FCoE-slot fix: [../../open-questions.md](../../open-questions.md) §1.
|
||||
|
||||
Reference in New Issue
Block a user