Appliance image builds and boots for the locked-in hardware: kernel/ixgbe-sff.patch carries the product driver subset (raw I2C + sff_i2c, ETQF steering with rule round-trip, FCoE slot guard — bench-only reg_ops/ts_bench excluded), ported to v7.2 and compile-verified; mkimage applies it or dies, builds IXGBE=y (ICE/I40E out with the ice.pkg blob) and puts ixgbe.allow_unsupported_sfp=1 on the cmdline; load-ixgbe joins the repo anchored to its own path; module bringup ordered after the host checks because a down port's TX_DISABLE powers the PHY core down while the mailbox keeps answering (proven by flipping admin state: down reads 0xffff everywhere, core wakes 0.07-0.24s after the raise) with identify settling 3s bounded; cold-boot condition reproduced on the bench (ports downed first) and passing, confirmed on the appliance

This commit is contained in:
flamingcow
2026-08-18 09:41:38 -07:00
parent bb99695b47
commit d5ed4bfbd9
8 changed files with 626 additions and 17 deletions
+2
View File
@@ -40,6 +40,8 @@ cabletest drives this module through its own Go RollBall client (`phy.go`): modu
Not available in safe space: pre-FEC counter, cable length — those live in the µC vendor space that killed the modules.
**A down port hides the PHY: TX_DISABLE powers the core down, not the µC.** ixgbe qualifies this module as fiber (the fake-SR EEPROM) and asserts TX_DISABLE while the netdev is admin-down; the module then answers every PHY register read with 0xFFFF through a fully working, echo-matched mailbox — EEPROM reads are unaffected. Proven by flipping admin state on the bench: down = PHY ID 0xFFFFFFFF, and the core wakes within 0.25 s of the raise (measured 0.07–0.24 s across cycles; the BCM answers instantly). The dev host masks it (its unmanaged config leaves the ports up from boot); a fresh appliance boot hits it on a never-raised port. cabletest therefore raises the links before first touching a module, and settles identify through a short bounded wait while the core wakes.
**Mailbox latency is regime-dependent** — <25 ms per read linked (X520 host), ~220 ms steady with no link (bench-measured, `~/work/phydiag-work/rb_mbox_latency.py`), and beyond 3 s while 10GBASE-T training runs (observed live when a cable plug landed mid-read at bringup). cabletest's mailbox deadline (`rbCmdTimeout`) is sized past a full ~6 s training so a plug event reads busy, not dead.
**Mailbox trap: an orphaned command's completion answers as yours.** If a session dies mid-mailbox-op (killed process), the µC finishes the orphaned command on its own time, leaving CMD=DONE and that command's value in the block. A next session that issues a command and samples status immediately reads the orphan's DONE and value as its own — seen live: a PHY ID 1.2 read answered 0x0006, the orphaned EEE-advert (7.60) value of the killed session, while the very next read was correct. The client therefore never samples status before a full poll gap, reads the value from the same 6-byte block read as the status, and accepts a completion only when the block echoes the issued devad/reg (`phy.go` mbox). The split-transaction python clients always had the pre-sample gap, which is why the field campaign never hit this.
+3 -3
View File
@@ -28,16 +28,16 @@ What this design carries in practice:
## `sff_i2c` diagnostics transport (patched driver, validated)
Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`).
Patched ixgbe (`~/work/ixgbe-sff/`) adds `ixgbe_i2c_raw_write`/`ixgbe_i2c_raw_read` (`ixgbe_phy.c`) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind `/sys/kernel/debug/ixgbe/<pci>/sff_i2c` (`w <addr8> <bytes…>` / `r <addr8> <n>`). The product subset of the patch (this transport, the ETQF steering below, the FCoE slot guard) is versioned as `kernel/ixgbe-sff.patch` and applied by `mkimage` to the appliance kernel.
- **SDA-release fix (required)**: multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). `raw_read` releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
- **Two-master windows closed by the compound op**: each simple `w`/`r` takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, `ethtool -m`) can interleave — traffic that fires exactly around cable swaps, when diagnostics also run, so collisions are rare but correlated with the interesting moments. Serialization itself holds (driver SFP traffic uses the same byte primitives under the same `phy_semaphore_mask`), and AT24-style word pointers do persist across STOP — the hazard is the other master *moving* the pointer or consuming the BCM 0x56 bridge's pending read data. The `x <waddr> <raddr> <delay_us> <n> <wbytes…>` command (`ixgbe_i2c_raw_xfer`) runs write→STOP→in-kernel delay→read under a single swfw hold, closing the window by construction; the write/read primitives are split into locked/unlocked halves so `x` reuses them under one acquire.
- **Compound op proven on the FS BCM** (`~/work/phydiag-work/compound_test.py`): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004); a ~3 ms hold is shorter than the driver's own SFP-identify holds. The delay sweep corroborates the hazard the op closes: at delay 0 the bridged read returns stale data (the prior read's 0x0004), at ≥200 µs it reads correctly. The >1 ms BCM delay is validated at 3 ms; treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.
- **RollBall split transactions are field-proven** — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks; the kernel's combined form is not required by the modules.
## Register/timestamp bench transport (patched driver)
## Register/timestamp bench transport (bench build only)
For the timestamp/counter bench pokes the driver also carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. Poke driver: `~/work/phydiag-work/x520poke`.
The out-of-tree bench build additionally carries `reg_ops` `read <reg>` returning the value through the file (not just dmesg) and a `ts_bench` command running the latch poll/re-arm loop in-kernel (`arm <ctl>`, `control`, `latchrate <ms>`) — needed because the BAR is unmappable from userspace under `IO_STRICT_DEVMEM`. These serve the concluded timestamp investigation and are deliberately not in the appliance patch. Poke driver: `~/work/phydiag-work/x520poke`.
## Timestamping facts
+1 -3
View File
@@ -39,9 +39,7 @@ Indices shift as stashes are pushed/popped — match by message, not number.
## Diagnostics artifacts outside the repo
In `~/work/`, ready to fold into the repo's `kernel/`:
- **Patched ixgbe** (`~/work/ixgbe-sff/`) — the `sff_i2c` raw-I2C debugfs transport, ETQF steering, `reg_ops`/`ts_bench`; `./load-ixgbe` rebuilds + swaps the module with `allow_unsupported_sfp=1`. Details: nics/x520/.
- **Patched ixgbe, bench build** (`~/work/ixgbe-sff/`) — out-of-tree module for the dev host: the product patch plus the bench-only extras (`reg_ops` value readback, `ts_bench`); `./load-ixgbe` rebuilds + swaps the module with `allow_unsupported_sfp=1`. The product subset (raw I2C + `sff_i2c`, ETQF steering, FCoE slot guard) is versioned at `kernel/ixgbe-sff.patch` and applied by `./mkimage` to every appliance kernel build — apply fails loudly when stable moves under it; port the patch then. Details: nics/x520/.
- **Bench tools and clients** (`~/work/phydiag-work/`) — `etqfbench/`, `x520poke/`, `compound_test.py`, `bcm_mcia_emu.py`, `bcm_ixgbe.py`, `bcm_explore.py`/`bcm_eee_off.py`, `bcm_ecd_probe.py`, `bcm_phaselock_bench.py`, `rollball_ixgbe.py`, `rb_forensic.py`. What each proved lives in modules/fs/, modules/wiitek/ and nics/x520/.
## Every boot