From 6f787c08731ba7b8a0f18b53b19e0681cb626626 Mon Sep 17 00:00:00 2001 From: flamingcow Date: Wed, 12 Aug 2026 17:35:25 -0700 Subject: [PATCH] Record X520 install: FS+Wiitek seated and linked, FS's honest EEPROM is what trips ixgbe qualification, uplink moved to enp88s0 --- docs/hardware.md | 15 +++++++++------ docs/state.md | 6 +++--- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index eb72f75..acec9c5 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -1,16 +1,19 @@ # Hardware and host ## The box -Single usable PCIe slot (Gen4 x8, currently the E810 — being replaced by the X520). The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome. +Single usable PCIe slot (Gen4 x8) holding the X520-DA2, which trains at its own Gen2 ceiling (5 GT/s ×8, 32 Gb/s); the ConnectX-5 replaces it when it arrives. The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome. ## Interfaces -- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order). Currently the X710 pair `enp4s0f0np0` / `enp4s0f1np1` (i40e); becomes the X520 pair when it arrives. -- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. Was the i40e pair during the E810-test-path era. -- **`enp89s0` (igc)**: this box's LAN uplink with the default route. Never repurpose or down it. +- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order — the X710 pair has already been enp4s0f\* and enp3s0f\* across reboots). Currently the X520 pair `enp1s0f0` (new Wiitek) / `enp1s0f1` (FS), ixgbe. +- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. The X710 pair `enp3s0f0np0` / `enp3s0f1np1` (i40e). +- **`enp88s0` (igc)**: this box's LAN uplink with the default route (its sibling port `enp89s0` is dark). Never repurpose or down it; identify by default route, not name. - Both test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4. -## The media lies -`ethtool` reports `Port: FIBRE` / `10000baseSR` on the test ports, and the modules' own EEPROMs claim fiber identities (SR, 850 nm, LC connector, multimode fiber lengths, even fake optical DOM). All false. **Every test module is a copper RJ45 10GBASE-T module with a cloned/lying EEPROM** — none are fiber. The cheap RJ45 SFP+ modules clone a real optical module's EEPROM to pass NIC compatibility checks, so a module whose part number reads `SFP-10G-SR` is still 10GBASE-T copper. The real media is a 10GBASE-T PHY inside each module. +## The media lies (mostly) +**Every test module is a copper RJ45 10GBASE-T module** — none are fiber, whatever the EEPROM claims; the real media is a 10GBASE-T PHY inside each module. But EEPROM honesty varies by vendor, with driver consequences: +- **The Wiitek lies**: LC connector, 850 nm SR, multimode lengths, fake optical DOM (the temperature is real, the "laser" powers are theater). The clone exists to pass NIC compatibility checks — and it works: stock ixgbe qualifies it without complaint. +- **The FS SFP-10G-T-100 is honest**: connector RJ45 (0x22), extended transceiver code 10GBASE-T Short Reach, 100 m copper, no fake DOM. Stock ixgbe rejects exactly that honesty as an unqualified type and **fails the whole port's probe** (error -95, no netdev at all) — the driver must load with `allow_unsupported_sfp=1`, which is therefore mandatory on the X520, and it's the truthful module that requires it. +- The 10Gtek claims `SFP-10G-SR`; a module whose part number reads SR can still be 10GBASE-T copper. Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE. diff --git a/docs/state.md b/docs/state.md index 4f487e0..07d36f5 100644 --- a/docs/state.md +++ b/docs/state.md @@ -6,10 +6,10 @@ AF_PACKET raw sockets everywhere (`sock.go`), flow-director steering, per-packet `stash@{1}` holds the phydiag campaign: python transport clients (`i40e_aq.py`, `rollball_i40e.py`, `sff.py`), scanners, and a STATUS.md campaign log. Working copies extracted to `~/work/phydiag-work/` during the module work. The knowledge from that campaign is now in these docs; the code should eventually move into the repo properly. ## Hardware in flight -- **Intel X520-DA2 ordered** — replaces the E810 in the box's single PCIe slot. Chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md. +- **Intel X520-DA2 installed** in the box's single PCIe slot (E810 out; PCIe trained 5 GT/s ×8). Port 0 `enp1s0f0` holds a new Wiitek (SN WAMZ012606X039U), port 1 `enp1s0f1` the FS (SN S2433774168), cable between them linked at 10G. Stock ixgbe runs with `allow_unsupported_sfp=1` — mandatory, and it's the *FS* that needs it (honest 10GBASE-T EEPROM fails Intel qualification; see hardware.md). The patched `sff_i2c` driver is not yet loaded. The X520 was chosen because ixgbe bit-bangs module I2C from the host (no firmware policy layer), enabling in-product module-PHY diagnostics; see transports.md. - **ConnectX-5 ordered (dual SFP28, PCIe x8) — the product NIC candidate.** Rate bucketing requires all-packet hardware RX timestamps (software stamping was tried and isn't precise enough — hard requirement); the 82599 stamps PTP frames only, and with raw-L2 steering relaxed (encap acceptable), mlx5 is the one driver that meets the full set: stamps every packet, shared PHC across ports, native ETHER_FLOW steering. Open: the MCIA diagnostics questions in open-questions.md §6 (BCM at I2C 0x56, SMI read data phase). On arrival: ports may ship in InfiniBand mode (`mlxconfig set LINK_TYPE_P1=2 LINK_TYPE_P2=2`), SFP+ modules drop into SFP28 cages at 10G, and check whether `mlx5_ib` autoloading interferes with channel changes (the irdma lesson). - **The X520 arrives first and carries two campaigns**: the diagnostics bring-up (certain arbitrary I2C framing for proving the BCM/RollBall clients), and the 82599 bucket-timestamping exploration in open-questions.md §2 — per-queue hardware counters as bucket content plus latch-anchor sampling. If that validates, the X520 is the fallback product NIC should MCIA fail on the ConnectX. -- **Replacement Wiitek (Marvell CUX3610) modules ordered.** Both original units were bricked by register exploration — see modules.md for the trap before ever touching one. +- **Replacement Wiitek (Marvell CUX3610) modules arrived** — one is in X520 port 0. Both original units were bricked by register exploration — see modules.md for the trap before ever touching one. - **In hand** (all copper RJ45 10GBASE-T modules with cloned/lying fiber EEPROMs — see hardware.md; none are actually fiber): - 2× FS SFP-10G-T-100 (Broadcom BCM84891L, fully documented diagnostics). - 1× Fibergaga SFP-10G-T-30M (Aquantia, RollBall, the proven oracle module). @@ -22,7 +22,7 @@ In `~/work/` alongside the ice-sff/phydiag-work artifacts, ready to fold into th - **RollBall client for the Wiiteks** (`~/work/phydiag-work/rollball_ixgbe.py`): the same `sff_i2c` transport, RollBall unlock/page/mailbox, per-pair IEEE SNR (PMA 1.133–1.136). The `*.0x??64` (high-byte ≥ 0x80) brick blacklist is a hard guard that raises *before* touching hardware — the client structurally cannot repeat the kill. ## Bringup plan when the X520 arrives -1. Install card, move FS modules in, `./load-ixgbe` (with `allow_unsupported_sfp=1` — ixgbe refuses SFPs without Intel qualification bytes, and the cloned EEPROMs aren't Intel-coded; `ixgbe_main.c:165`), confirm `sff_i2c` appears. +1. Card installed and modules seated (done). Next: `./load-ixgbe` with `allow_unsupported_sfp=1` (mandatory — the FS's honest 10GBASE-T EEPROM fails Intel qualification and kills the port probe; `ixgbe_main.c:165`), confirm `sff_i2c` appears. 2. FS/BCM modules first: `bcm_ixgbe.py` sanity reads (EEPROM, PHY ID, STATUS) to prove the SMI transport, then the command handler (CMD_GET_SNR vs the IEEE registers) on the test cable set. 3. Replacement Wiiteks: IEEE-standard registers only (SNR, latches) unless/until a documented recipe exists for more; the VCT templates in modules.md are candidates for single-shot targeted probes on a sacrificial unit only. 4. Re-derive host tuning on ixgbe (coalescing/ring syntax differs).