From 06de095d1cf2f448fef2028b29054c7fa40df3da Mon Sep 17 00:00:00 2001 From: flamingcow Date: Sun, 16 Aug 2026 22:02:04 -0700 Subject: [PATCH] X520 returns as product NIC: smoothed-bucket rate plan (read-time buckets, backward excess-fill, rate as headline not verdict) dissolves the per-frame RX timestamp requirement; CX-5 retired with full record (MCIA via VSC/ICMD proven, fwctl write-block, card-global module index, ~150ms mailbox quantum unexplained, FS link deadlock); E810 below-AQ bit-bang idea recorded and retired; FS+Wiitek pair and length return as goals --- docs/modules/fs/README.md | 10 +++ docs/nics/README.md | 20 +++--- docs/nics/connectx-5/README.md | 107 ++++++++++++++++++++++++++------- docs/nics/e810/README.md | 4 ++ docs/nics/x520/README.md | 23 ++++++- docs/state.md | 40 ++++++++---- 6 files changed, 159 insertions(+), 45 deletions(-) diff --git a/docs/modules/fs/README.md b/docs/modules/fs/README.md index 12dfd27..c1def1e 100644 --- a/docs/modules/fs/README.md +++ b/docs/modules/fs/README.md @@ -16,6 +16,16 @@ PHY at I2C 8-bit **0xAC write / 0xAD read**. - Single-byte reads see the bridge as inert (returns 0) — the DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead. Treat 0x0000 reads as retry-with-longer-delay: 0 is also the bridge's not-ready signature. - Recorded for a future multi-byte firmware transport (mlx5 MCIA shape): read = write `[001+devad,RegH,RegL]` offset_size=1, STOP, >1 ms, read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data = `[RegL,DataH,DataL]`. +## Host interface (XFI) bring-up gating + +**The module never enables its XFI TX toward the host until it receives valid host signal** — observed live on the ConnectX-5, where it produces a mutual-wait deadlock ([../../nics/connectx-5/README.md](../../nics/connectx-5/README.md)): + +- While ungated it asserts RX_LOS (the SFP pin and A2h byte 110 bit 1 both) and its host-side TX is electrically dark (NIC RX eye grade 0). The copper side is independent: it trains and holds link normally the whole time, so RX_LOS here means "no host signal", not "no copper link". +- The gate is PCS-qualified, not analog: PRBS31 and electrical-idle patterns at 10.3125G into the module do not clear it — only valid 10GBASE-R does. Hosts that emit SFI idles from driver load (ixgbe) never see any of this. +- With its host side dead and copper trained, the module signals fault over the copper; the far-end module relays it to its own host as remote-fault ordered sets (the far end's tell that this end's host interface is down). +- Not a path to the gate: PMA soft reset (1.0.15, self-clearing — copper retrains, host TX stays dark), KR mode (SET 0x800F PASSes, no host TX with or without AN restart; GET 0x800E is a DATA1-answer readback, unusable), the LED controller (1.0xA83B = 0x8400, 1.0xA83C = 0x00E1; rewriting either does not move the RX_LOS pin — the pin is not LED-sourced). All of it survived un-bricked, values restored. +- A0h byte 36 = 0x1C (10GBASE-T Short Reach) is the honest class declaration; power is not the issue (A0h byte 64 = 0x00 declares no high-power-level mechanism; A2h 92–94 are filler bytes). + ## MDIO command handler Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-commands.md). Traps: diff --git a/docs/nics/README.md b/docs/nics/README.md index 8683ef0..7713664 100644 --- a/docs/nics/README.md +++ b/docs/nics/README.md @@ -1,10 +1,14 @@ # NICs -The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path gets the hardware timestamps it requires. Per-device directories hold the details. +The NIC decides two things: whether the host can reach module-PHY diagnostics at all, and whether the measurement path can report an honest rate. Per-device directories hold the details. ## The transport rule -A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols. +A NIC is usable for module diagnostics only if the **host** can master the module I2C bus with arbitrary multi-byte framing and writes. Firmware-mediated I2C imposes limits that break the module command protocols (E810: writes blocked outright; CX-5: writes work but the firmware's own module traffic quantizes every mailbox read to ~150 ms steps). + +## The timestamp requirement is dissolved + +All-packet hardware RX timestamps existed for one consumer — bucketing received frames by arrival time so the displayed rate stayed honest under host read-jitter. The smoothed-bucket plan ([x520](x520/README.md)) computes the rate from read-time buckets with a backward excess-fill instead, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. With that, the capability that disqualified the X520 — and the only thing the firmware-managed NICs offered over it — is no longer required (pending the audit that the rate buckets were the stamps' sole consumer). Raw-L2 steering was never a hard requirement either (encap acceptable). ## Comparison @@ -12,14 +16,12 @@ Which card is where: [../state.md](../state.md). | NIC (driver) | Module I2C | All-packet RX hw stamps | Verdict | |---|---|---|---| -| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven** | No (single serial latch; 1588-event frames only, EtherType configurable) | Diagnostics transport only — no all-packet exact stamps, so disqualified as product NIC | -| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA; writes allowed; framing open | **Yes** (CQE-stamps everything) | **The product candidate** — the only card meeting the full requirement set | -| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; PTP length-experiment platform | -| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes | - -Requirement weights behind the verdicts: all-packet hardware RX timestamps are non-negotiable (software stamping was tried, not precise enough), raw-L2 steering is not (encap acceptable). Full analyses live in each device directory. +| [X520 / 82599 (ixgbe)](x520/README.md) | Host bit-bangs — full control, **proven**, fast (<25 ms RollBall) | No (single serial latch; 1588-event frames only) — no longer required | **The product NIC** under the smoothed-bucket rate plan: only card with unmediated host I2C, and the FS links here (mixed FS+Wiitek pair proven) | +| [ConnectX-4/5 (mlx5)](connectx-5/README.md) | Firmware MCIA — writes via VSC/ICMD only (fwctl read-only), every mailbox read quantized to ~150 ms steps by firmware module traffic | Yes (CQE-stamps everything) | Retired: its stamps are no longer needed and its I2C mediation is the cost; the FS never links on it either (host-interface deadlock, device page) | +| [X710 (i40e)](x710/README.md) | 1-byte-offset only | No | RollBall oracle path; noise pair | +| [E810 (ice)](e810/README.md) | Multi-byte framing, writes EPERM-blocked | Yes | Dead for diagnostics writes; the below-the-AQ bit-bang idea (device page) is retired with the timestamp requirement | ## Architecture - **Co-location is forced, not chosen.** SNR under load, the pre-FEC correlation, and diagnosing the module that carried the stress all require diagnostics I2C to reach the modules the traffic flows through, and module I2C is only reachable through the hosting NIC. A "diagnostics NIC" would need the modules in *its* cages, off the traffic path — structurally impossible, not merely awkward. The chassis seconds it: one usable slot. -- **The single card is the ConnectX-5.** The requirement weights flip the selection the X520 was chosen under: the 82599 stamps 1588-event frames only, txgbe is likewise PTP-only (verified in upstream `wx_ptp.c`; evaluated on paper, never owned), and mlx5 is the only driver meeting the full set — CQE-stamps every packet, one shared clock across ports, native ETHER_FLOW steering. If MCIA can't reach the BCM, product diagnostics run via the Marvell/Aquantia modules only ([connectx-5](connectx-5/README.md)). +- **The single card is the X520.** With the timestamp requirement dissolved, the selection weights return to where the X520 originally won: unmediated host I2C is the one capability no firmware-managed card provides (E810 blocks writes; mlx5 mediates and quantizes), and it is non-negotiable for the module protocols. Gen2 ×8 (32 Gb/s raw) clears 2×10G full duplex; ETQF steering and loss attribution are proven on hardware. The general lesson, learned twice: NICs that stamp every packet are all firmware-managed, and firmware in the module-I2C path is what breaks or slows the diagnostics — the two headline capabilities appear to be mutually exclusive off the shelf. diff --git a/docs/nics/connectx-5/README.md b/docs/nics/connectx-5/README.md index 0924524..599e58a 100644 --- a/docs/nics/connectx-5/README.md +++ b/docs/nics/connectx-5/README.md @@ -1,11 +1,36 @@ # ConnectX-5 (mlx5) — the product NIC candidate -Dual SFP28, PCIe x8. mlx5 is the one driver meeting the full requirement set: +## Status: retired — the X520 smoothed-bucket plan removes the need for this card -- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — the committed bucketing and `probe.go` run unchanged. -- **One shared clock across both ports** — `probe.go`'s shared-PHC assumption holds. +The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is **parked in a git stash** ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved. + +**Known:** + +- The two capabilities that decide the NIC — all-packet RX stamps and host-mastered module I2C *with writes* — both work. `/dev/fwctl` is read-only for MCIA (writes silently no-op under its restricted uctx); the mstflint PCICONF path (VSC → ICMD → register-access TLV, reimplemented in Go) does writes. Details below. +- The MCIA `module` field is **card-global** (module 0 = port 0, module 1 = port 1, per the PCI function digit) and must be set per port; unset it defaults to 0 and *both* ports read port 0 — a silent fake-dual-end bug the identical Wiitek vendor name hides and the serial numbers expose. +- **The RollBall mailbox is slow here.** Each register read waits for the module µC to flip the mailbox DONE bit, and that completion is **quantized**: ~50 ms (fast) or ~200 ms or ~350 ms — base + 0/1/2 steps of ~150 ms, never in between. A 7-register poll of both modules runs ~4–5 s, so SNR first appears ~5 s in and refreshes slowly. On the X520 the *same* µC answered in <25 ms, so this is imposed by the CX-5 side, not the module. The value trace is clean (uniform reads, µC holds `0x02` then flips to `0x04`), so it is not a transport stall, page clobber, or stale read. +- Slightly under line rate: ~0.03 Gb/s below the E810 on the same traffic. Host tuning (coalescing/rings) was never re-derived for mlx5, so this is probably untuned, not a ceiling. + +**Not known:** + +- **The source of the ~150 ms quantum.** Something periodic on the CX-5 side (firmware module monitoring? a driver poll? — no mlx5 thermal zone and hwmon module temp is on-demand, so a firmware task is the suspect) gates when the µC's result becomes committable; the mailbox completion phase-locks to it. Not confirmed, and not yet known whether it can be slowed/disabled (may be thermal-safety machinery). +- Whether that quantum is fixable at all, and whether the 0.03 Gb/s gap closes with proper coalescing/ring tuning. +- The MCIA `l` lock bit's real semantics (did nothing useful in testing); the two-master interleaving story is moot given the process-wide lock. + +Dual SFP28, PCIe x8 (MCX512A-ACU, firmware 16.35.4506; trains Gen3 ×8 in the box's slot, 63 Gb/s). mlx5 is the one driver meeting the full requirement set: + +- **CQE-stamps every RX packet** (`HWTSTAMP_FILTER_ALL`) — `ethtool -T` reports rx filter `all` on both ports; the committed bucketing and `probe.go` run unchanged. +- **One hardware oscillator across both ports, but two PHC devices** — see the measured PHC topology below; cross-port stamp comparison needs a startup offset calibration. - **Native ETHER_FLOW ntuple steering** — no driver patch, no encap needed. -- Mature driver; SFP+ modules drop into SFP28 cages at 10G. +- Mature driver; SFP+ modules drop into SFP28 cages at 10G (proven — the Wiitek pair links at 10G over the long cable). + +## PHC topology — measured + +Each port is its own PCI function and exposes its own PHC (`mlx5_ptp` ×2), fed by **one shared oscillator**: both clocks drift against CLOCK_REALTIME at an identical rate (~59 ppm uncorrected), and their mutual offset moves only ~10 ppb (590 ns over 60 s — timecounter-update jitter, not oscillator difference). The offset itself is **quasi-static but nonzero** (~33 µs observed; set by each function's timecounter init). Consequences: + +- Per-port rate bucketing is unaffected. +- Cross-port stamp comparison (one-way latency) needs the differential measured at startup — one `PTP_SYS_OFFSET` (`phc_ctl cmp`) per clock and a subtraction, good to sub-µs. +- This firmware (16.35.4506) has no real-time-clock NV option to collapse the two functions onto one hardware time. ## MCIA register format (verified in kernel headers) @@ -16,27 +41,67 @@ The public PRM documents only the data path plus the `ACCESS_REGISTER` envelope - **Every read is offset-addressed**: 16-bit `device_address` + 16-bit `size`, 12 data dwords (48 B max per transaction). There is **no raw-read encoding** — a read with no preceding offset cannot even be requested. - **`l` lock bit** — locks the module across consecutive transactions; MCIA's own answer to the two-master interleaving problem. -## BCM SMI through MCIA framing — module side proven +## BCM SMI through MCIA — proven end-to-end on hardware -MCIA has no raw-read encoding (every read emits an offset write first), so SMI reads work only if the BCM bridge tolerates that framing. **It does — proven on hardware** (X520 bit-bang emulation of MCIA's write-offset-then-read, `~/work/phydiag-work/bcm_mcia_emu.py` against the FS BCM): +Every MCIA open question resolves in favor of the transport, live against the FS BCM in the CX-5's own cage (mstflint's `mstreg`; mstflint comes from the AUR, not the Arch repos): -- A 1-byte offset write (0x00 and 0xFF tried) injected between SMI command and read phase leaves the pending read data intact — immediately and after 3 ms; the expected register value (PHY ID 0x3590/0x5081) reads back correctly every time. -- A cold offset-write+read returns the last-latched SMI data — the offset byte is ignored, not parsed as a new command frame. -- Normal SMI operation recovers cleanly afterward; no lasting disturbance. -- Caveat: emulated with STOP-separated transactions; a repeated-START combined transfer (if that's what firmware emits) is untested. +- **Firmware accepts arbitrary I2C device addresses.** `i2c_device_address=0x56` (the BCM bridge) transacts with status 0 in both directions; kernel paths only ever use 0x50/0x51 but the firmware doesn't care. +- **MCIA writes emit exactly one offset byte**, so `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly. +- **The bridge tolerates MCIA's write-offset-then-read framing live** (the offset byte injected before the read phase is ignored, as the X520 bit-bang emulation predicted — `~/work/phydiag-work/bcm_mcia_emu.py`); PHY ID 0x3590 reads back correctly. +- **Handler mailbox ops work**: DATA1–5 writes, CMD write, STATUS poll (PASS observed), all through the same framing. +- RollBall is plain offset addressing at 0x51 — fits MCIA by construction. -SMI *writes* fit MCIA's model directly if firmware emits a 1-byte offset: an MCIA write emits `[offset][data…]` as one I2C write, so choosing `device_address` = first SMI frame byte synthesizes the 3–5 byte command frame exactly. +The recipe (one SMI transaction = one MCIA SET, plus one MCIA GET for reads): -## Open questions (decide on arrival) +- SMI read of `devad.reg`: MCIA SET with `i2c_device_address=0x56, device_address=(0x20|devad), size=2, dword[0]=reg<<16` → MCIA GET with `device_address=0, size=2` → value in dword[0] bits 31:16. Respect the >1 ms bridge delay between the two. +- SMI write of `devad.reg=val`: MCIA SET with `device_address=devad, size=4, dword[0]=reg<<16|val`. +- Live example: `mstreg -d 01:00.1 --reg_name MCIA --set "dword[0]=0x00020000" --indexes "module=1,i2c_device_address=0x56,page_number=0,device_address=0x21,size=2" --yes` then the GET → 0x3590. -- Whether firmware MCIA accepts **arbitrary I2C device addresses** — the BCM bridge sits at 0x56; kernel paths only ever use 0x50/0x51 (`MLX5_I2C_ADDR_LOW/HIGH`). The register format allows it and the per-access status byte makes rejection observable. -- Whether MCIA writes emit exactly **one offset byte** (needed for the SMI-frame synthesis above), and whether reads use STOP-separated or repeated-START framing. -- RollBall (0x51-resident, offset-model) fits MCIA — low risk. -- If MCIA can't reach the BCM: product diagnostics via the Marvell/Aquantia modules only. The X520 is **not** a fallback product NIC — it has no all-packet exact-timestamp path, which a cable tester's rate buckets require ([../x520/README.md](../x520/README.md)). +## How cabletest reaches MCIA in production (`phy_mcia.go`) -## Arrival notes +There are two host paths to MCIA, and the split is decisive: -- Ports may ship in InfiniBand mode: `mlxconfig -d set LINK_TYPE_P1=2 LINK_TYPE_P2=2`, reboot. -- Check whether auto-loading `mlx5_ib` interferes with channel changes (the irdma lesson, hardware.md). -- Force 10G if SFP28 autoneg is stubborn: `ethtool -s speed 10000 autoneg off`. -- Mellanox is permissive with third-party modules (no Intel-style qualification lockout) — verify with the lying-EEPROM clones anyway. +- **`/dev/fwctl` (the kernel's sanctioned firmware back-channel) is read-only for MCIA.** ACCESS_REG reads work at scope `DEBUG_READ_ONLY`; **writes are silently dropped** — the RPC returns success at every layer (`errno=0`, command status 0, MCIA status 0) yet the register never changes, at every scope including `DEBUG_WRITE_FULL`. The firmware honours the ACCESS_REG opcode under fwctl's restricted user context (uid 2, caps `0x04000000`) but no-ops the actual I2C write. Proven in `~/work/phydiag-work/fwctl_mcia/diag`. Since a RollBall register **read** requires **writes** (set page/devad/command), fwctl cannot run RollBall at all. +- **The mstflint PCICONF path (VSC vendor capability → ICMD → FLASH_REG_ACCESS) can write**, so cabletest uses it. Per-dword gateway ops through the PCI config-space vendor capability reach ICMD space; a register-access command wrapped in operation+reg TLVs carries the MCIA transaction. Reimplemented in Go (`phy_mcia.go`), no mstflint dependency at runtime. + +Two things that are not optional and one that is not needed: + +- **The MCIA `module` field is card-global and must be set per port** (module 0 = port 0, module 1 = port 1, i.e. the PCI function digit), *regardless of which function issues the command* — function 1's ICMD with `module=0` reads port 0's module. It defaults to 0, so leaving it unset makes **both** ports read port 0: identical to a working dual-end read except the second end is a phantom of the first. It hid for a while because the vendor name is identical on both Wiiteks; the serial numbers (`…X256U` vs `…X039U`) exposed it. This was also the real cause of the SNR "coming and going": two functions hammering **one** module's I2C serialised into multi-second reads that blew the freshness window. Routed to their own modules, reads are prompt and the margin holds steady. +- **A process-wide lock holds each MCIA transaction whole.** The two ports drive independent per-function gateways but one shared card firmware; the lock keeps a command and its result from interleaving with the other port's. Not a retry, not a workaround — a plain mutex. +- The gateway cap9 semaphore is taken in one shot (this run is its only user; the driver reaches the firmware by its own BAR interface, not this config-space back-channel) and dropped at open if a `SIGKILL`ed prior run left it held. The ICMD-level semaphore mstflint also takes is **not** needed — the mutex plus the ICMD busy-bit check cover it, and taking it raced the driver. The MCIA `l` lock bit did nothing useful in testing. + +There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox. + +## The FS cannot link on the CX-5 — host-interface deadlock + +The FS's XFI never comes up against this firmware; **diagnostics to the FS work regardless** (all of the above ran with the link down). The mechanism, each side observed directly: + +- **The firmware never enables the port's serdes TX before seeing module signal.** The port FSM sits in `ETH_AN_FSM_ENABLE`/Polling and SLTP reads all-zero TX parameters (the linked Wiitek port shows real ones), in force-10G mission mode and every other reachable mode. +- **The FS never enables its XFI TX before seeing valid host signal.** It asserts RX_LOS (the pin, mirrored in MCION, and A2h byte 110 bit 1) and its TX is electrically dark (RX eye grade 0 on the NIC) — while its copper side trains and stays up normally (7.33 shows master resolution, both receivers OK). PRBS31 and electrical-idle test patterns at 10.3125G do not clear the LOS: the gate wants valid 10GBASE-R, which only an active mission-mode port emits. +- Each side waits for the other; the link never starts. The Wiitek escapes because it transmits unconditionally (its port reaches Physical LinkUp immediately, then relays remote-fault from the FS's dead host side over the trained copper — so its netdev also stays down until the FS end resolves). +- ixgbe masks the FS behavior entirely: SFI idles are on the wire from driver load, so the FS side always has host signal first. + +Attempts that do not break it: force 10G / autoneg toggles, port bounces, PRBS31/IDLE test-mode TX, PH and EX loopback, PMMP `eeprom_override` (accepted but reads back zero — not implemented for SFP on 16.35.4506), PMAOS module toggle (ICMD error 0x3), FS LED-controller rewrites (1.0xA83B/0xA83C do not source the LOS pin), FS PMA soft reset, FS KR mode enable + AN restarts ([../../modules/fs/README.md](../../modules/fs/README.md)). + +**The deadlock is module-typed, not port-typed — confirmed by cage swap**: the Wiitek reaches Physical LinkUp in either cage, the FS in neither (same Polling FSM, same remote-fault relay from the far end). + +Paths forward, in escalation order: + +1. **Fibergaga + Wiitek as the test pair.** Both are lying-EEPROM/TX-always modules; both populate IEEE per-pair SNR margins; the Fibergaga carries the length path in documented registers (1E.C884 length ±1 m, 1E.C800 TDR verdicts — [../../modules/fibergaga/README.md](../../modules/fibergaga/README.md)) with no ECD-style link blip. Costs the BCM ECD; frees the FS to the shelf (it is ixgbe-only anyway). +2. **EEPROM surgery on the FS** (byte 36: 0x1C→0x00, plus checksum) to make it lie like the Wiitek — only worth it if the firmware's gate keys on the module's declared class rather than the LOS pin, which is unproven; writability unassessed; semi-permanent. + +## Arrival notes (resolved) + +- Ports arrived in Ethernet mode (`enp1s0f*np*` netdevs from first boot); no `mlxconfig` LINK_TYPE change needed. +- `mlx5_ib` autoloads; no channel-change interference observed so far (the irdma lesson, hardware.md) — re-verify when cabletest's channel setup first runs. +- SFP28 autoneg does not settle with these modules; `ethtool -s speed 10000 autoneg off` is the working configuration on the module ports. +- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above). + +## Host-setup deltas from the ice/ixgbe path (`system.go`, `counters.go`) + +Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks: + +- **RX flow steering needs `rx-ntuple-filter` enabled first** (`ETHTOOL_SFEATURES`); mlx5 ships it off and reports filter capacity 0 and rejects inserts until it is on. It then advertises **no fixed filter table** (capacity 0) and rejects `RX_CLS_LOC_ANY` over the legacy `SIOCETHTOOL` ioctl, but accepts any **explicit** slot — so ethertype rules are placed at explicit locations allocated upward from 0 (ice/ixgbe allocate downward into their advertised table; the split is keyed on the advertised capacity). +- **Error counters**: the NIC-side rx error set is the PHY RMON categories `rx_{crc_errors,in_range_len_errors,out_of_range_len,undersize_pkts,oversize_pkts,fragments,jabbers}_phy`, mutually exclusive per frame. This is the NIC's view of the module↔NIC SFI link; the copper's own pre-FEC errors come from the module over RollBall, not here. Do **not** fold `tx_dropped` into cable-health — mlx5 counts millions of tx drops while the link is down at bringup. +- **Bringup forces a retrain** (AN restart) that drops the link for several seconds, so the run waits for carrier before baselining counters and starting traffic; otherwise the settling flap and tx-into-dark-link drops are charged to the fresh run. +- **PHC**: one oscillator, two PHC devices, ~33 µs quasi-static cross-port offset (see the PHC-topology section) — calibrate once at startup for cross-port latency. diff --git a/docs/nics/e810/README.md b/docs/nics/e810/README.md index 2806248..bcf3ace 100644 --- a/docs/nics/e810/README.md +++ b/docs/nics/e810/README.md @@ -7,3 +7,7 @@ The NIC the committed measurement path was built against: its datapath delivers - topo-I2C (0x06E2/E3) has offset-size control: params bit[7] repeated-start, [6:5] address length, [3:0] data size. **Reads work perfectly.** - Every write returns aq 1 (EPERM) — tested identically against EEPROM 0x50 and BCM 0x56, so the block is write-vs-read, not address-scoped. Intel policy, survives NVM update; NVM 5.01 hardened the 3.10 silent-drop into an explicit EPERM. - Dead for anything needing writes: RollBall unlock needs a write; BCM SMI reads *start* with a write. + +## The bit-bang-below-the-AQ idea — retired with the timestamp requirement + +The write block is at the **Admin Queue** (firmware) level — the `0x06E2/E3` topo-I2C commands are refused with EPERM. That is not the same as the module's SDA/SCL being unreachable: the 82599 reaches the module I2C **without firmware** by toggling SDA/SCL as GPIO bits in `I2CCTL` (the `~/work/ixgbe-sff/` patch bit-bangs arbitrary START…STOP from those primitives), and whether the E810 exposes an equivalent register-level / SDP-GPIO path below the AQ policy was never tested. The idea existed to marry all-packet stamps to raw I2C in one card; the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves the stamps requirement, so the X520 covers everything and this investigation is not needed. It stays recorded as the E810's one open door if per-frame arrival times are ever wanted again — the investigation would start at the datasheet's SDP/GPIO and low-level I2C controller registers, asking whether the cage's SDA/SCL are host-addressable. diff --git a/docs/nics/x520/README.md b/docs/nics/x520/README.md index 656496a..af80c57 100644 --- a/docs/nics/x520/README.md +++ b/docs/nics/x520/README.md @@ -1,11 +1,30 @@ # Intel X520-DA2 (82599, ixgbe) -The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — the diagnostics workhorse, and **only** that. It cannot run the measurement path and is not a fallback product NIC: a cable tester needs exact per-frame RX timestamps, the 82599 has one serial 1588-event latch and no all-packet path, and aggregate counters/rates are honest only when the receive is well-behaved — useless for catching the fault. The ConnectX-5 is the product NIC. +The only *certain* arbitrary-framing module-I2C transport (bit-banged, no firmware in the path) — and, under the smoothed-bucket rate plan below, **the product NIC**. The disqualification that exiled it (a cable tester needs exact per-frame RX timestamps; the 82599 has one serial 1588-event latch and no all-packet path) dissolves once the displayed rate is a smoothed throughput headline computed from software-read counts, with every fault verdict coming from per-frame sequence/CRC accounting rather than the rate. The plan, its prerequisite audit, and what it retires are in the section below. - PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (`lspci -vv`). - Loss attribution survives here: missed-packet (RXMPC → `rx_missed_errors`) and per-queue drop (QPRDC) counters — "prove host-side zero" works. - **`allow_unsupported_sfp=1` is mandatory** (`ixgbe_main.c:165`): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). `load-ixgbe` passes it. +## The product plan: smoothed-bucket rate, no hardware stamps + +The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by *arrival* time so the displayed rate stays honest while the host's reads jitter. The plan replaces arrival-time bucketing with read-time bucketing plus a deconvolution pass, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed. + +- **Rate is a headline, never a verdict.** Lost/late/corrupt come from per-frame sequence numbers and CRC checks, independent of the rate; no cable-health decision keys off the rate number. This is the license for everything below. +- **Buckets are keyed by read time; excess above line rate moves backward.** Read jitter is a purely backward smear — a frame is read at or after its arrival, never before — so a bucket's excess above line rate is always frames that arrived earlier and were read late, and moving that excess backward to fill earlier deficits is the physically correct deconvolution, not an estimate. The dominant jitter shape (host stalls, then drains the backlog) is deficit-then-burst, which is exactly what the backward pass repairs. Line rate is known, so over/under is well-defined. +- **A forward pass is optional edge polish.** A stall/burst pair entirely inside the window balances under the backward pass alone; only pairs cut by the window boundary leave orphan excess or deficit. Pairing those up forward is cosmetic, is the pass most likely to paper over a genuine dip at the window edge, and is droppable if it ever lies. +- **Window > worst host read-stall; window = display latency.** Sized comfortably past the worst stall, pairs stop straddling the edge (measure the actual stall distribution before choosing). The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs. +- **Faults stay sharp.** The pass only moves *real* excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur. +- The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is. + +**Prerequisite audit before building**: confirm the rate buckets are the *only* consumer of hardware RX stamps in the committed path. The buckets are currently keyed by the MAC's clock (`SO_TIMESTAMPING` cmsg → `rxStats` epochs → `readRateBucket`), and `probe.go` leaned on a shared-PHC assumption; anything else that needs true arrival times (one-way latency, cross-port stamp comparison) does not survive the change. "Late" must remain a sequence-number notion, not a timestamp one. + +What this enables and retires: + +- **The configuration menu returns.** The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative. Length returns as a goal. +- **The ConnectX-5 and the E810 bit-bang hunt retire.** The CX-5's only edge was all-packet stamps, paid for with firmware-mediated module I2C (the mailbox quantum — [../connectx-5/README.md](../connectx-5/README.md)); the E810 investigation existed only to marry stamps to raw I2C ([../e810/README.md](../e810/README.md)). Neither is needed once the audit clears. +- **Build on HEAD** — the committed X520/BCM/ECD implementation, not the CX-5 stash. The work: re-key the receive buckets from MAC-stamp time to read time, add the backward-fill pass, and remove the `rx_filter=ALL` hard host check (`ts.go`) whose requirement this dissolves. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted. + ## `sff_i2c` diagnostics transport (patched driver, validated) Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe//sff_i2c` (`w ` / `r `). @@ -21,7 +40,7 @@ For the timestamp/counter bench pokes the driver also carries `reg_ops` `read