theater/cabletest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record X520 install: FS+Wiitek seated and linked, FS's honest EEPROM is what trips ixgbe qualification, uplink moved to enp88s0

Browse Source
This commit is contained in:
flamingcow
2026-08-12 17:35:25 -07:00
parent 5f4a8ceb57
commit 6f787c0873
2 changed files with 12 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/hardware.md
+9 -6
View File
@@ -1,16 +1,19 @@
# Hardware and host
## The box
Single usable PCIe slot (Gen4 x8, currently the E810 — being replaced by the X520). The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.
Single usable PCIe slot (Gen4 x8) holding the X520-DA2, which trains at its own Gen2 ceiling (5 GT/s ×8, 32 Gb/s); the ConnectX-5 replaces it when it arrives. The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.
## Interfaces
- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order). Currently the X710 pair `enp4s0f0np0` / `enp4s0f1np1` (i40e); becomes the X520 pair when it arrives.
- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. Was the i40e pair during the E810-test-path era.
- **`enp89s0` (igc)**: this box's LAN uplink with the default route. Never repurpose or down it.
- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order — the X710 pair has already been enp4s0f\* and enp3s0f\* across reboots). Currently the X520 pair `enp1s0f0` (new Wiitek) / `enp1s0f1` (FS), ixgbe.
- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. The X710 pair `enp3s0f0np0` / `enp3s0f1np1` (i40e).
- **`enp88s0` (igc)**: this box's LAN uplink with the default route (its sibling port `enp89s0` is dark). Never repurpose or down it; identify by default route, not name.
- Both test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.
## The media lies
`ethtool` reports `Port: FIBRE` / `10000baseSR` on the test ports, and the modules' own EEPROMs claim fiber identities (SR, 850 nm, LC connector, multimode fiber lengths, even fake optical DOM). All false. **Every test module is a copper RJ45 10GBASE-T module with a cloned/lying EEPROM** — none are fiber. The cheap RJ45 SFP+ modules clone a real optical module's EEPROM to pass NIC compatibility checks, so a module whose part number reads `SFP-10G-SR` is still 10GBASE-T copper. The real media is a 10GBASE-T PHY inside each module.
## The media lies (mostly)
**Every test module is a copper RJ45 10GBASE-T module** — none are fiber, whatever the EEPROM claims; the real media is a 10GBASE-T PHY inside each module. But EEPROM honesty varies by vendor, with driver consequences:
- **The Wiitek lies**: LC connector, 850 nm SR, multimode lengths, fake optical DOM (the temperature is real, the "laser" powers are theater). The clone exists to pass NIC compatibility checks — and it works: stock ixgbe qualifies it without complaint.
- **The FS SFP-10G-T-100 is honest**: connector RJ45 (0x22), extended transceiver code 10GBASE-T Short Reach, 100 m copper, no fake DOM. Stock ixgbe rejects exactly that honesty as an unqualified type and **fails the whole port's probe** (error -95, no netdev at all) — the driver must load with `allow_unsupported_sfp=1`, which is therefore mandatory on the X520, and it's the truthful module that requires it.
- The 10Gtek claims `SFP-10G-SR`; a module whose part number reads SR can still be 10GBASE-T copper.
Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE.