40 lines
4.2 KiB
Markdown
40 lines
4.2 KiB
Markdown
# Hardware and host
|
||
|
||
## The box
|
||
Single usable PCIe slot (Gen4 x8, currently the E810 — being replaced by the X520). The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.
|
||
|
||
## Interfaces
|
||
- **Test pair**: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order). Currently the X710 pair `enp4s0f0np0` / `enp4s0f1np1` (i40e); becomes the X520 pair when it arrives.
|
||
- **Noise pair**: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. Was the i40e pair during the E810-test-path era.
|
||
- **`enp89s0` (igc)**: this box's LAN uplink with the default route. Never repurpose or down it.
|
||
- Both test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.
|
||
|
||
## The media lies
|
||
`ethtool` reports `Port: FIBRE` / `10000baseSR` on the test ports, and the modules' own EEPROMs claim fiber identities (SR, 850 nm, LC connector, multimode fiber lengths, even fake optical DOM). All false. **Every test module is a copper RJ45 10GBASE-T module with a cloned/lying EEPROM** — none are fiber. The cheap RJ45 SFP+ modules clone a real optical module's EEPROM to pass NIC compatibility checks, so a module whose part number reads `SFP-10G-SR` is still 10GBASE-T copper. The real media is a 10GBASE-T PHY inside each module.
|
||
|
||
Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE.
|
||
|
||
## Host tuning (resets every reboot; re-apply before trusting results)
|
||
Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults.
|
||
|
||
1. **CPU governor → `performance`** on all cores.
|
||
- Biggest single contributor.
|
||
- Under powersave, cores idle at 400 MHz and ramp too slowly, dropping frames at startup.
|
||
2. **Disable adaptive coalescing.**
|
||
- `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`.
|
||
- Syntax/values differ on ixgbe — re-derive for the X520.
|
||
3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair).
|
||
- Effect was not isolated in testing; don't claim it as a proven win.
|
||
4. **Socket buffers** need no sysctl change.
|
||
- The tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) to bypass the 4 MB `rmem_max` clamp.
|
||
|
||
Verify a clean run shows `tx frames == rx frames` exactly before a run counts.
|
||
|
||
## irdma (dev host only)
|
||
`irdma` autoloads and binds the ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active"), which fails the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.
|
||
|
||
## Running
|
||
Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a enp1s0f0np0 -b enp1s0f1np1`. It starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1) and do not follow every run with a pgrep orphan check — the group kill is the guarantee. The tell for two competing instances (an orphan, or the user's own hand-run) is `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs — ask rather than killing something that might be theirs. `./harness` also dumps the panel to `shots/<offset>.png` for remote assessment.
|
||
|
||
Test runs use the **default config only** — no `-sizes`/`-streams` overrides, and no editing the hardcoded `frameSizes`/`numStreams`/`batchSize` (that's the same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.
|