Docs cleanup: stale-claim conflicts fixed (e810 stamps/probe.go, CX-5 candidate title, committed-path perf table now the X520 20G result with E810-era rows marked historical, ice-pair ring note), irdma/ice notes moved to e810, state.md halved (done-checklist, RX-steering and standing-goals duplication out; ECD suppression fact into modules/fs), changelog framing flattened to present tense
This commit is contained in:
flamingcow
+1
-1
@@ -5,7 +5,7 @@ Project knowledge that moves and versions with the repo.
|
||||
General:
|
||||
|
||||
- [goals.md](goals.md) — what cabletest is for and what it must output.
|
||||
- [state.md](state.md) — where things are now: committed vs stashed, the box and what's in it, bringup progress, open items. **All current-placement state lives here and only here.**
|
||||
- [state.md](state.md) — where things are now: committed vs stashed, the box and what's in it, out-of-repo artifacts, every-boot steps. **All current-placement state lives here and only here.**
|
||||
- [hardware.md](hardware.md) — host rules: interface conventions, tuning, how to run.
|
||||
- [measurement.md](measurement.md) — how to measure correctly; performance results and dead ends.
|
||||
- [open-questions.md](open-questions.md) — the genuinely open problems, as pointers into the docs that hold their detail.
|
||||
|
||||
+1
-5
@@ -17,15 +17,11 @@ Required for zero baseline loss; without them baseline loss is 0.02–0.3% and m
|
||||
|
||||
1. **CPU governor → `performance`** on all cores. Biggest single contributor: under powersave, cores idle at 400 MHz, ramp too slowly, and drop frames at startup.
|
||||
2. **Disable adaptive coalescing**: `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC.
|
||||
3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair). Effect was not isolated in testing; don't claim it as a proven win.
|
||||
3. **Rings deep**, raised toward the max via `ethtool -G` (the tool asks RX 8160 / TX 4096; ixgbe takes it as-is). Effect was not isolated in testing; don't claim it as a proven win.
|
||||
4. **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp.
|
||||
|
||||
Verify a clean run shows `tx frames == rx frames` exactly before a run counts.
|
||||
|
||||
## irdma (dev host only)
|
||||
|
||||
`irdma` autoloads and binds ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active") and failing the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.
|
||||
|
||||
## Running
|
||||
|
||||
Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a <portA> -b <portB>`.
|
||||
|
||||
+9
-6
@@ -31,27 +31,30 @@ This rule covers the **NIC-counter** buckets and the per-stream receive buckets
|
||||
- Small-frame runs are CPU/host-bound, not cable-bound; cable conclusions drawn from them are false.
|
||||
- Keep to the default config (hardware.md) and judge from steady state — the first ~5 s is settling transient, not residual error.
|
||||
|
||||
## Measured performance (AF_PACKET committed path, ice/E810 era)
|
||||
## Measured performance
|
||||
|
||||
The committed X520 path at the default mix: steady 20.00–20.04 Gb/s (line rate both directions, ~1.78 Mpps/dir), zero lost/corrupt/link/internal over a 90 s run, including startup.
|
||||
|
||||
Host-side limits measured on the earlier E810 (ice) datapath — CPU facts, not NIC-specific:
|
||||
|
||||
| Config | Result |
|
||||
|---|---|
|
||||
| Full size mix, 7 flow-director streams | Line rate (10.0–10.3 Gb/s/dir), ~1.78 Mpps/dir, zero loss including startup |
|
||||
| 64 B only | pps-bound at ~5.4 Mpps/dir (frame generation is the limit, not receive drops) → only ~3.8 Gb/s |
|
||||
| AF_XDP experiment (stashed) | 64 B TX ~14.5 Mpps/dir; RX ~4 Mpps/dir on AF_PACKET vs ~13.5 with AF_XDP RX |
|
||||
|
||||
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts.
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path no longer uses hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
- RX cannot be parallelized by RSS — hardware RSS can't hash raw ethertypes. Flow-steering by ethertype to distinct queues is what gives multiple NAPI contexts (ETQF on the X520, Flow Director on the E810).
|
||||
- The AF_XDP experiment delivered per-packet MAC RX stamps via an XDP-metadata kfunc (E810 datapath only; the committed path uses no hardware stamps at all). With the test path off the E810 it is parked; its novelty was delivery mechanism and throughput, not the use of hardware stamps.
|
||||
|
||||
## Dead ends — measured, do not re-attempt without new hardware
|
||||
|
||||
The binding constraint was total CPU across ~28 goroutines on 20 threads.
|
||||
Measured in the E810 era; the binding constraint was total CPU across ~28 goroutines on 20 threads.
|
||||
|
||||
- **Splitting TX senders from RX streams** — raises TX but collapses RX (RX scales with queue count, capped at 7 on that NIC).
|
||||
- **CPU pinning** — the Go scheduler beats manual placement; E-cores are poor at TX.
|
||||
- **Batch sizes above 64** — no gain, worse loss.
|
||||
- **Chasing the TX `sendmmsg` busy-spin on ENOBUFS** — it only exists when the TX ring is full, which means the wire is the ceiling; recovering that CPU buys no packets. POLLOUT is inert under PACKET_QDISC_BYPASS (skb freed on ENOBUFS, socket always reports writable).
|
||||
|
||||
## rxnfc / Flow Director programming (ice) — three traps
|
||||
## rxnfc flow-rule programming — three traps (found on ice; 1 and 2 are kernel-ABI-generic)
|
||||
|
||||
1. **Mask polarity is inverted vs `ethtool -n` display.** In raw `m_u` bytes a *set* bit means "must match": a working "match ethertype, ignore MACs" rule has `m_u.ether_spec` = dst 00×6, src 00×6, proto ff ff. `ethtool -n` prints the complement — trusting its display gives an inverted rule that silently matches nothing.
|
||||
2. **`rule_locs` sits at offset 188, not `sizeof(struct ethtool_rxnfc)` (192), on amd64** — `rule_locs[]` follows `rule_cnt` at 188. Reading from 192 yields garbage locations, so existing rules are never found/deleted.
|
||||
|
||||
@@ -85,7 +85,7 @@ The ECD register mechanism is absent from the handler catalog and the datasheet
|
||||
|
||||
Observed on a plugged, linked, healthy cable: verdicts OK×4 and per-pair lengths of [45, 45, 41, 46] on a ~45 m cable — **meter-accurate with no calibration**, and this ECD reports length for healthy pairs, not just faults, resolving the terminated-far-end concern in [../README.md](../README.md). Caveats:
|
||||
|
||||
- The run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear) — do not run mid-measurement until the disturbance is characterized.
|
||||
- The run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear) — do not run mid-measurement until the disturbance is characterized. cabletest runs the ECD at bringup and on every reset only, and while a measure runs every failure counter is suppressed at its source (worker error adds, loss-window write-offs, NIC-poller accumulation, socket drops, module latch priming) so the blip is never counted anywhere rather than counted and reverted (`phy.go` `cableDiag.measuring`).
|
||||
- **The run can leave clause-45 AN disabled** (7.0.12 cleared — observed live after diags racing an in-progress retrain): with AN off the BCM emits no AN pulses, both ends report no link partner (7.1 LP-AN-able 0), and the link stays down through AN restarts, role changes and PMA resets until the bit is re-enabled. cabletest restarts AN with `7.0 |= 0x1200` (enable + restart) after every diag, and `devRestartAN` always forces the enable bit — a bare bit-9 restart preserves the cleared enable forever.
|
||||
- **An interrupted run can wedge the µC's SMI service** (observed after a process died mid-diag): the bridge ACKs I2C but serves 0x0000 for every register at any delay and silently drops MDIO writes — no in-band recovery, not even PMA reset — while the EEPROM path and the trained link keep working. Recovery: a driver unbind/rebind of the port (the SFP re-initialization resets the µC); a reseat would do the same.
|
||||
- Fault verdicts (open/short/inter-pair) are unexercised — deliberately: the product is a closed-loop tester, both ends always plugged.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# ConnectX-5 (mlx5) — the product NIC candidate
|
||||
# ConnectX-5 (mlx5) — retired
|
||||
|
||||
## Status: retired — the X520 smoothed-bucket plan removes the need for this card
|
||||
|
||||
@@ -17,9 +17,9 @@ The all-packet stamps were this card's only edge, and the smoothed-bucket rate p
|
||||
- Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning.
|
||||
- The MCIA `l` lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock.
|
||||
|
||||
Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set:
|
||||
Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 was the one driver meeting the old requirement set (all-packet stamps + module-I2C writes):
|
||||
|
||||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports; the committed bucketing and `probe.go` run unchanged.
|
||||
- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports.
|
||||
- **One hardware oscillator across both ports, but two PHC devices** — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration.
|
||||
- **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed.
|
||||
- Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable).
|
||||
@@ -85,19 +85,19 @@ Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31
|
||||
|
||||
**The deadlock is module-typed, not port-typed — confirmed by cage swap**: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end).
|
||||
|
||||
Paths forward, in escalation order:
|
||||
Paths forward if this card ever returns, in escalation order:
|
||||
|
||||
1. **Fibergaga + Wiitek as the test pair.** Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — [../../modules/fibergaga/README.md](../../modules/fibergaga/README.md)) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway).
|
||||
2. **EEPROM surgery on the FS** (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent.
|
||||
|
||||
## Arrival notes (resolved)
|
||||
## Arrival notes
|
||||
|
||||
- Ports arrived in Ethernet mode (`enp1s0f*np*` netdevs from first boot); no `mlxconfig` LINK_TYPE change needed.
|
||||
- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs.
|
||||
- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, [../e810/README.md](../e810/README.md)) — re-verify when cabletest's channel setup first runs.
|
||||
- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
|
||||
- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
|
||||
|
||||
## Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`)
|
||||
## Host-setup deltas from the ice/ixgbe path (the stashed implementation's `system.go`/`counters.go`)
|
||||
|
||||
Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
|
||||
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
# Intel E810 (ice)
|
||||
|
||||
The NIC the committed measurement path was built against: its datapath delivers per-packet MAC RX timestamps (`rx_filter=ALL`), which bucketing and `probe.go` require. A patched ice + `sff_i2c` exists for read-side diagnostics work.
|
||||
Out of the box. The measurement path was originally built against it — its datapath delivers per-packet MAC RX timestamps (`rx_filter=ALL`), which the committed rate design no longer needs ([../README.md](../README.md)). A patched ice + `sff_i2c` exists for read-side diagnostics work.
|
||||
|
||||
## Host notes (if it returns)
|
||||
|
||||
- `irdma` autoloads and binds ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active") and failing the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma.
|
||||
- ice refuses channel changes while ntuple rules exist — cabletest clears its own stale rules first.
|
||||
|
||||
## Module I2C: multi-byte framing works, writes are policy-blocked
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Intel X520-DA2 (82599, ixgbe)
|
||||
|
||||
The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate design below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The committed tree implements this design.
|
||||
**The product NIC** — the only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path). Its one gap — a single serial 1588-event latch, no all-packet RX timestamps — doesn't matter under the smoothed-bucket rate design below: the displayed rate is a smoothed throughput headline computed from software-read counts, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. The committed tree implements this design.
|
||||
|
||||
- PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`).
|
||||
- Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
|
||||
@@ -19,12 +19,11 @@ The per-frame hardware RX timestamp requirement existed for exactly one consumer
|
||||
- **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
|
||||
- The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.
|
||||
|
||||
**The audit cleared**: the rate buckets were the *only* consumer of hardware RX stamps in the committed path (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`, nothing else); "late" is and stays a sequence-number notion. The implementation: buckets keyed by one shared host clock read once per drained batch (`rateEpoch`, `rxStats.commit`), and a **settled window** — each completed bucket enters once, donates its excess above line rate backward once (`fillBack`, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window: a recompute shows every excess twice (as the donation, then again unspent when its bucket reaches the display slot) and the headline reads above line rate — observed live as 20.0–20.2 G on a 20 G wire. The whole hardware-timestamp machinery is gone: the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`). The forward pass is not built.
|
||||
"Late" is a sequence-number notion, never a rate one. The implementation: buckets keyed by one shared host clock read once per drained batch (`rateEpoch`), and a **settled window** — each completed bucket enters once, donates its excess above line rate backward once (`fillBack`, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window — the recompute form double-counts moved excess and reads above line rate (measurement.md). The committed path requests no hardware timestamps at all; the forward pass is not built.
|
||||
|
||||
What this enables and retires:
|
||||
What this design carries in practice:
|
||||
|
||||
- **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal.
|
||||
- **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears.
|
||||
- **The FS links and runs full diagnostics here** (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative.
|
||||
- **Built on the committed X520/BCM/ECD implementation**, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
|
||||
|
||||
## `sff_i2c` diagnostics transport (patched driver, validated)
|
||||
|
||||
+10
-39
@@ -2,23 +2,13 @@
|
||||
|
||||
## Where the code is right now
|
||||
|
||||
**The committed tree is the X520 product implementation**: AF_PACKET datapath with read-time rate buckets and the backward excess-fill smear (nics/x520/, "The smoothed-bucket rate" — rate is a headline, never a verdict; `smearWindow` 4 buckets = 64 ms display latency, resize after measuring the stall distribution: open-questions), the mixed FS+Wiitek diagnostics with the BCM ECD length path, `testDriver` = ixgbe. The hardware-timestamp machinery is gone — the `rx_filter=ALL` check, the `SO_TIMESTAMPING` request and the per-frame cmsg parse (formerly `ts.go`) — after the audit confirmed the rate buckets were its sole consumer. The CX-5 is retired (nics/connectx-5/, "Status") and its Wiitek-pair rewrite stays parked in the "CX-5 Wiitek…" stash as a parts bin; the E810 bit-bang investigation is retired with the timestamp requirement (nics/e810/).
|
||||
|
||||
## The stashed Wiitek/CX-5 implementation (parts bin)
|
||||
|
||||
All in on the Wiitek pair behind the ConnectX-5. The BCM/FS handler, the ECD, and all cable-length measurement are dropped in this variant.
|
||||
|
||||
AF_PACKET raw sockets everywhere (`sock.go`); native mlx5 ETHER_FLOW steering (rx-ntuple enabled, explicit slots — nics/connectx-5/); per-packet-MAC-rx-stamped rate buckets (`SO_TIMESTAMPING` cmsg, `rx_filter=ALL`, which the CX-5 satisfies natively); read-time-stamped NIC-counter rates over the mlx5 PHY RMON error set (`counters.go`); test interfaces pinned to MTU 9000 with a 9018-byte jumbo in the size mix.
|
||||
|
||||
Module diagnostics (`phy.go` + `phy_mcia.go`): a transport interface behind the RollBall client, so the same protocol logic runs over either the patched-ixgbe `sff_i2c` debugfs (X520) or MCIA on mlx5 (CX-5). The MCIA transport is the mstflint PCICONF path reimplemented in Go — VSC vendor capability → ICMD → register-access TLV — because `/dev/fwctl` is read-only for MCIA (writes silently no-op, so it cannot run the mailbox; nics/connectx-5/). One loop goroutine per module owns its transport, and a process-wide lock serialises every MCIA transaction: both CX-5 ports are one card sharing one firmware command interface, and interleaved mailbox access reads stale — the single lock is the whole fix, no retries. Every mailbox register is whitelist-guarded, panicking before hardware on anything outside the proven-safe set (modules/wiitek/).
|
||||
|
||||
Bringup is read-only for the Wiitek pair (both auto-resolve master/slave and advertise no EEE — verified, nothing forced), so it never drops the carrier and the run starts immediately with no wait for link. A ~1 Hz poller feeds **per-pair SNR margin from both ends** (each Wiitek's IEEE 1.133–136 addressed by its own card-global MCIA module index — the headline: worst pair across both ends) plus the corrected-error set (PCS 3.33 errored blocks/BER, PMA 1.147 fast-retrain count); green ≥ 3 dB, amber ≥ 1 dB — provisional until the graded-noise run. Framebuffer UI; harness.
|
||||
**The committed tree is the X520 product implementation**: AF_PACKET datapath with read-time rate buckets and the backward excess-fill smear (nics/x520/ — rate is a headline, never a verdict; `smearWindow` 4 buckets = 64 ms display latency, resize after measuring the stall distribution: open-questions), the mixed FS+Wiitek diagnostics with the BCM ECD length path, `testDriver` = ixgbe. No hardware-timestamp machinery anywhere. The CX-5 is retired (nics/connectx-5/, "Status") and the E810 bit-bang investigation with it (nics/e810/).
|
||||
|
||||
## Stashes
|
||||
|
||||
Indices shift as stashes are pushed/popped — match by message, not number.
|
||||
|
||||
- **"CX-5 Wiitek: MCIA transport…"** (currently `stash@{0}`) — the whole Wiitek-pair rewrite: `phy_mcia.go` (VSC/ICMD MCIA transport), transport-interface `phy.go`, dropped BCM/ECD/length, ntuple + mlx5 counters, per-port card-global MCIA module index. Carries live debug instrumentation and `rbCmdPoll` lowered to 2 ms from the mailbox-latency investigation — **not shippable as-is**, clean it before committing. Also carries the pre-existing uncommitted `ui.go`/`ui_test.go` edits that were entangled in the same files.
|
||||
- **"CX-5 Wiitek: MCIA transport…"** (currently `stash@{0}`) — the whole Wiitek-pair rewrite for the CX-5, kept as a parts bin: `phy_mcia.go` (VSC/ICMD MCIA transport), transport-interface `phy.go` (the same RollBall logic over either patched-ixgbe `sff_i2c` or MCIA), dual-end per-pair SNR from both modules (headline: worst pair across both ends), mlx5 ntuple steering + PHY RMON counters, test interfaces at MTU 9000 with a 9018-byte jumbo in the size mix; BCM/ECD/length dropped. Technical record: nics/connectx-5/. Carries live debug instrumentation and `rbCmdPoll` lowered to 2 ms from the mailbox-latency investigation — **not shippable as-is**, clean it before committing — plus the pre-existing uncommitted `ui.go`/`ui_test.go` edits that were entangled in the same files.
|
||||
- **"AF_XDP both directions…"** (currently `stash@{1}`) — AF_XDP conversion (hand-rolled XSK + XDP program, per-frame MAC-stamp buckets via rx metadata). Built for the E810-era datapath. An experiment, not a requirement.
|
||||
- **"phydiag campaign…"** (currently `stash@{2}`) — python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, campaign log. Working copies live in `~/work/phydiag-work/`; the knowledge is in these docs; the code should eventually move into the repo properly.
|
||||
|
||||
@@ -42,38 +32,19 @@ Indices shift as stashes are pushed/popped — match by message, not number.
|
||||
| X520-DA2 | **Installed** in the single PCIe slot — **the product NIC** (nics/x520/) | Unmediated bit-bang I2C (patched-ixgbe `sff_i2c`), <25 ms RollBall, FS links and the ECD length path is proven; the smoothed-bucket rate design removes the need for hardware RX stamps |
|
||||
| ConnectX-5 | Out of the box — retired | Ran the two-Wiitek pair at near line rate with dual-end SNR (stashed `phy_mcia.go` MCIA transport), but the firmware quantizes every module mailbox read to ~150 ms steps and the FS never links behind mlx5 — nics/connectx-5/, "Status" |
|
||||
| FS SFP-10G-T-100 ×2 | One in the X520 test pair, one on the shelf | BCM84891L, documented, robust; ixgbe-only (unlinkable behind mlx5 — host-interface deadlock, confirmed module-typed, nics/connectx-5/ — and rejected by the X710). The FS+Wiitek mixed pair (FS length via ECD + Wiitek SNR) is the product configuration |
|
||||
| Replacement Wiiteks | One in the X520 test pair, two in the X710 noise pair, rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first |
|
||||
| Replacement Wiiteks | One in the X520 test pair, two in the X710 noise pair, rest on the shelf | Originals bricked by register exploration — modules/wiitek/ trap first. IEEE-standard registers only (modules/README.md safe set); VCT templates are single-shot candidates on a sacrificial unit only |
|
||||
| Fibergaga SFP-10G-T-30M | On the shelf | Aquantia, RollBall, the documented oracle; the alternate length path if the pair mix ever changes |
|
||||
| 10Gtek | In hand | Claims SFP-10G-SR, still copper RJ45; filler, not in the test set |
|
||||
| E810 | Out of the box | Patched ice + `sff_i2c` remains useful only if it returns for read-side work |
|
||||
|
||||
## X520 diagnostics path — validated on hardware
|
||||
## Diagnostics artifacts outside the repo
|
||||
|
||||
In `~/work/` alongside the phydiag artifacts, ready to fold into the repo's `kernel/`:
|
||||
In `~/work/`, ready to fold into the repo's `kernel/`:
|
||||
|
||||
- **Patched ixgbe** (`~/work/ixgbe-sff/`): `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` in `ixgbe_phy.c` (arbitrary START…STOP transactions from the existing bit-bang primitives, swfw-bracketed) + `sff_i2c` debugfs file (`w <addr8> <bytes…>` / `r <addr8> <n>` / `x <waddr> <raddr> <delay_us> <n> <wbytes…>` single-hold compound). One real bug found on hardware: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` now releases SDA after each ACK, mirroring stock's own release block. The patch also carries: **ETQF steering** (ETHER_FLOW ntuple inserts mapped onto ETQF/ETQS slots in `ixgbe_ethtool.c`, restore-on-up/clear-on-close hooked, FCoE slot-2 write moved behind its enable guard); **`reg_ops` read-returns-value** and a **`ts_bench`** debugfs command (in-kernel latch poll/re-arm for the timestamp bench pokes). `load-ixgbe` rebuilds + swaps the module, passing `allow_unsupported_sfp=1`.
|
||||
- **Bench tools** (`~/work/phydiag-work/`): `etqfbench/` (Go; proves ETQF steering + the fdir-can't-steer-raw-L2 negative), `x520poke/` (Go; drives `reg_ops`/`ts_bench` for the timestamp/counter pokes), `compound_test.py` (proves the `x` compound op against the FS BCM), `bcm_mcia_emu.py` (proves the BCM bridge tolerates MCIA offset-write-then-read framing — nics/connectx-5/).
|
||||
- **BCM client** (`~/work/phydiag-work/bcm_ixgbe.py`): proven end-to-end on the FS — EEPROM, PHY ID 0x3590:5081, handler STATUS, per-pair SNR ≈ [32, 27.5, 30, 27.6] dB via `CMD_GET_SNR = 0x8030` invoked bare. Code-table trap and stale-DATA1 trap: modules/fs/.
|
||||
- **Exploration probes** (`bcm_explore.py`, `bcm_eee_off.py`): full GET sweep + the AutogrEEEn force-off recipe (results: modules/fs/).
|
||||
- **RollBall client** (`~/work/phydiag-work/rollball_ixgbe.py`): same transport; unlock/page/mailbox, per-pair IEEE SNR. Untested on this card. The `*.0x??64` brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill.
|
||||
- **Patched ixgbe** (`~/work/ixgbe-sff/`) — the `sff_i2c` raw-I2C debugfs transport, ETQF steering, `reg_ops`/`ts_bench`; `./load-ixgbe` rebuilds + swaps the module with `allow_unsupported_sfp=1`. Details: nics/x520/.
|
||||
- **Bench tools and clients** (`~/work/phydiag-work/`) — `etqfbench/`, `x520poke/`, `compound_test.py`, `bcm_mcia_emu.py`, `bcm_ixgbe.py`, `bcm_explore.py`/`bcm_eee_off.py`, `bcm_ecd_probe.py`, `bcm_phaselock_bench.py`, `rollball_ixgbe.py`, `rb_forensic.py`. What each proved lives in modules/fs/, modules/wiitek/ and nics/x520/.
|
||||
|
||||
## Bringup on the swapped box
|
||||
## Every boot
|
||||
|
||||
1. `./load-ixgbe` — until then the FS port has no netdev (stock driver, error -95) and `driverPair("ixgbe")` cannot find 2 interfaces. Verify `sff_i2c` present for both ports and 5 GT/s ×8 trained.
|
||||
2. ~~Run.~~ Done — full pass on hardware: both modules identified and dispatched, ECD at startup (all pairs ok, ~42 m headline), steady 20.00–20.04 G both directions with zero lost/corrupt/link/internal over a 90 s run, SNR +7.4–7.5 dB, noise cycle live. SETTINGS self-reports the ixgbe shape (rx=8160/tx=4096 taken as-is; coalesce tx-shares-rx).
|
||||
3. Measure the host read-stall distribution under load and resize `smearWindow` if 64 ms is mis-sized (open-questions).
|
||||
4. Replacement Wiiteks: IEEE-standard registers only (modules/README.md safe set) unless/until a documented recipe exists for more; VCT templates are single-shot candidates on a sacrificial unit only.
|
||||
|
||||
## RX flow-steering on ixgbe
|
||||
|
||||
**Solved on hardware.** cabletest fans RX across queues by steering raw ethertype 0x88b5+ to distinct queues (`system.go`, ETHTOOL_SRXCLSRLINS, ETHER_FLOW). Stock ixgbe's `ixgbe_flowspec_to_flow_type` accepts only TCP/UDP/SCTP-v4 and IPv4 — no ETHER_FLOW. The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF/ETQS slots (`ixgbe_ethtool.c`), so `system.go` runs unchanged; 7 streams → 7 queues, unfiltered ethertypes → queue 0, verified (`~/work/phydiag-work/etqfbench`). Full analysis in nics/x520/.
|
||||
|
||||
- **ETQF/ETQS L2 EtherType queue filters — proven.** Dedicated ethertype→queue stage ahead of RSS/fdir; 8 slots, index 3 reserved for 1588; the FCoE slot-2 write was moved behind its enable guard so the default config leaves 7 free.
|
||||
- **Flow Director flex-byte match — ruled out on hardware.** fdir classifies IPv4/IPv6 only; a flex-word rule on a raw 0x88b5 stream steers nothing (every frame to queue 0). `FDIRCTRL_FLEX_SHIFT = 0x6` (offset 12 = ethertype) only narrows an IP match.
|
||||
- **Unused fallbacks**: MAC→VMDq pool steering (distinct dest MACs → pools → queues, fully raw); minimal bare-IPv4 framing steered by IP_USER_FLOW; single-queue RX (caps near ~1.6 Mpps single-NAPI). Encap is acceptable but unnecessary — steering never constrained NIC choice.
|
||||
|
||||
## Standing goals
|
||||
|
||||
- **Length is a product goal again** — the FS (ECD length) rides alongside the Wiitek (SNR) in the test pair, which the tree implements. The Fibergaga (Aquantia 1E.C884 ±1 m + 1E.C800 verdicts, documented) remains the alternate length path if the pair mix ever changes.
|
||||
- **BCM ECD works** (recipe recovered from the OpenBCM SDK, validated on the FS — modules/fs/): per-pair lengths meter-accurate against a known ~45 m cable. cabletest runs it at bringup and on every reset; while a measure runs, every failure counter is suppressed at its source (worker error adds, loss-window write-offs, NIC-poller accumulation, socket drops, module latch priming), so the blip is never counted anywhere rather than counted and reverted (`phy.go` `cableDiag.measuring`). Remaining work is characterizing the link blip the run causes (length stays a between-measurements operation until then — open-questions).
|
||||
- **Pre-FEC verification** on the Aquantia — counters documented; needs the graded-noise correlation run (design: modules/fibergaga/).
|
||||
1. `./load-ixgbe` — until then the FS port has no netdev (stock driver, error -95) and `driverPair("ixgbe")` cannot find 2 interfaces. Verify `sff_i2c` present for both ports.
|
||||
2. Re-apply host tuning (hardware.md) before trusting results.
|
||||
|
||||
Reference in New Issue
Block a user