Docs cleanup: stale-claim conflicts fixed (e810 stamps/probe.go, CX-5 candidate title, committed-path perf table now the X520 20G result with E810-era rows marked historical, ice-pair ring note), irdma/ice notes moved to e810, state.md halved (done-checklist, RX-steering and standing-goals duplication out; ECD suppression fact into modules/fs), changelog framing flattened to present tense
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
# ConnectX-5 (mlx5) — the product NIC candidate
|
||||
# ConnectX-5 (mlx5) — retired
|
||||
|
||||
## Status: retired — the X520 smoothed-bucket plan removes the need for this card
|
||||
|
||||
@@ -17,9 +17,9 @@ The all-packet stamps were this card's only edge, and the smoothed-bucket rate p
|
||||
- Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
|
||||
- The MCIA `l` lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.
|
||||
|
||||
Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set:
|
||||
Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 was the one driver meeting the old requirement set (all-packet stamps + module-I2C writes):
|
||||
|
||||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports; the committed bucketing and `probe.go` run unchanged.
|
||||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports.
|
||||
- **One hardware oscillator across both ports, but two PHC devices** — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
|
||||
- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
|
||||
- Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).
|
||||
@@ -85,19 +85,19 @@ Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31
|
||||
|
||||
**The deadlock is module-typed, not port-typed — confirmed by cage swap**: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).
|
||||
|
||||
Paths forward, in escalation order:
|
||||
Paths forward if this card ever returns, in escalation order:
|
||||
|
||||
1. **Fibergaga + Wiitek as the test pair.** Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — [../../modules/fibergaga/README.md](../../modules/fibergaga/README.md)) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
|
||||
2. **EEPROM surgery on the FS** (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.
|
||||
|
||||
## Arrival notes (resolved)
|
||||
## Arrival notes
|
||||
|
||||
- Ports arrived in Ethernet mode (`enp1s0f*np*` netdevs from first boot); no `mlxconfig` LINK_TYPE change needed.
|
||||
- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs.
|
||||
- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, [../e810/README.md](../e810/README.md)) — re-verify when cabletest's channel setup first runs.
|
||||
- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
|
||||
- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
|
||||
|
||||
## Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`)
|
||||
## Host-setup deltas from the ice/ixgbe path (the stashed implementation's `system.go`/`counters.go`)
|
||||
|
||||
Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
|
||||
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
# Intel E810 (ice)
|
||||
|
||||
The NIC the committed measurement path was built against: its datapath delivers per-packet MAC RX timestamps (`rx_filter=ALL`), which bucketing and `probe.go` require. A patched ice + `sff_i2c` exists for read-side diagnostics work.
|
||||
Out of the box. The measurement path was originally built against it — its datapath delivers per-packet MAC RX timestamps (`rx_filter=ALL`), which the committed rate design no longer needs ([../README.md](../README.md)). A patched ice + `sff_i2c` exists for read-side diagnostics work.
|
||||
|
||||
## Host notes (if it returns)
|
||||
|
||||
- `irdma` autoloads and binds ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active") and failing the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma.
|
||||
- ice refuses channel changes while ntuple rules exist — cabletest clears its own stale rules first.
|
||||
|
||||
## Module I2C: multi-byte framing works, writes are policy-blocked
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Intel X520-DA2 (82599, ixgbe)
|
||||
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate design below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The committed tree implements this design.
|
||||
**The product NIC** — the only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path). Its one gap — a single serial 1588-event latch, no all-packet RX timestamps — doesn't matter under the smoothed-bucket rate design below: the displayed rate is a smoothed throughput headline computed from software-read counts, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. The committed tree implements this design.
|
||||
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
|
||||
- Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
|
||||
@@ -19,12 +19,11 @@ The per-frame hardware RX timestamp requirement existed for exactly one consumer
|
||||
- **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
|
||||
- The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.
|
||||
|
||||
**The audit cleared**: the rate buckets were the *only* consumer of hardware RX stamps in the committed path (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`, nothing else); "late" is and stays a sequence-number notion. The implementation: buckets keyed by one shared host clock read once per drained batch (`rateEpoch`, `rxStats.commit`), and a **settled window** — each completed bucket enters once, donates its excess above line rate backward once (`fillBack`, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window: a recompute shows every excess twice (as the donation, then again unspent when its bucket reaches the display slot) and the headline reads above line rate — observed live as 20.0–20.2 G on a 20 G wire. The whole hardware-timestamp machinery is gone: the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`). The forward pass is not built.
|
||||
"Late" is a sequence-number notion, never a rate one. The implementation: buckets keyed by one shared host clock read once per drained batch (`rateEpoch`), and a **settled window** — each completed bucket enters once, donates its excess above line rate backward once (`fillBack`, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window — the recompute form double-counts moved excess and reads above line rate (measurement.md). The committed path requests no hardware timestamps at all; the forward pass is not built.
|
||||
|
||||
What this enables and retires:
|
||||
What this design carries in practice:
|
||||
|
||||
- **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal.
|
||||
- **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears.
|
||||
- **The FS links and runs full diagnostics here** (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative.
|
||||
- **Built on the committed X520/BCM/ECD implementation**, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
|
||||
|
||||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||||
|
||||
Reference in New Issue
Block a user