43 lines
4.9 KiB
Markdown
43 lines
4.9 KiB
Markdown
# Hardware and host
|
||
|
||
## The box
|
||
Single usable PCIe slot (Gen4 x8) holding the X520-DA2, which trains at its own Gen2 ceiling (5 GT/s ×8, 32 Gb/s); the ConnectX-5 replaces it when it arrives. The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.
|
||
|
||
## Interfaces
|
||
- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order — the X710 pair has already been enp4s0f\* and enp3s0f\* across reboots). Currently the X520 pair `enp1s0f0` (new Wiitek) / `enp1s0f1` (FS), ixgbe.
|
||
- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. The X710 pair `enp3s0f0np0` / `enp3s0f1np1` (i40e).
|
||
- **`enp88s0` (igc)**: this box's LAN uplink with the default route (its sibling port `enp89s0` is dark). Never repurpose or down it; identify by default route, not name.
|
||
- Both test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.
|
||
|
||
## The media lies (mostly)
|
||
**Every test module is a copper RJ45 10GBASE-T module** — none are fiber, whatever the EEPROM claims; the real media is a 10GBASE-T PHY inside each module. But EEPROM honesty varies by vendor, with driver consequences:
|
||
- **The Wiitek lies**: LC connector, 850 nm SR, multimode lengths, fake optical DOM (the temperature is real, the "laser" powers are theater). The clone exists to pass NIC compatibility checks — and it works: stock ixgbe qualifies it without complaint.
|
||
- **The FS SFP-10G-T-100 is honest**: connector RJ45 (0x22), extended transceiver code 10GBASE-T Short Reach, 100 m copper, no fake DOM. Stock ixgbe rejects exactly that honesty as an unqualified type and **fails the whole port's probe** (error -95, no netdev at all) — the driver must load with `allow_unsupported_sfp=1`, which is therefore mandatory on the X520, and it's the truthful module that requires it.
|
||
- The 10Gtek claims `SFP-10G-SR`; a module whose part number reads SR can still be 10GBASE-T copper.
|
||
|
||
Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE.
|
||
|
||
## Host tuning (resets every reboot; re-apply before trusting results)
|
||
Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults.
|
||
|
||
1. **CPU governor → `performance`** on all cores.
|
||
- Biggest single contributor.
|
||
- Under powersave, cores idle at 400 MHz and ramp too slowly, dropping frames at startup.
|
||
2. **Disable adaptive coalescing.**
|
||
- `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`.
|
||
- Syntax/values differ on ixgbe — re-derive for the X520.
|
||
3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair).
|
||
- Effect was not isolated in testing; don't claim it as a proven win.
|
||
4. **Socket buffers** need no sysctl change.
|
||
- The tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) to bypass the 4 MB `rmem_max` clamp.
|
||
|
||
Verify a clean run shows `tx frames == rx frames` exactly before a run counts.
|
||
|
||
## irdma (dev host only)
|
||
`irdma` autoloads and binds the ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active"), which fails the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.
|
||
|
||
## Running
|
||
Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a enp1s0f0np0 -b enp1s0f1np1`. It starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1) and do not follow every run with a pgrep orphan check — the group kill is the guarantee. The tell for two competing instances (an orphan, or the user's own hand-run) is `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs — ask rather than killing something that might be theirs. `./harness` also dumps the panel to `shots/<offset>.png` for remote assessment.
|
||
|
||
Test runs use the **default config only** — no `-sizes`/`-streams` overrides, and no editing the hardcoded `frameSizes`/`numStreams`/`batchSize` (that's the same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.
|