38 lines
3.2 KiB
Markdown
38 lines
3.2 KiB
Markdown
# Host rules
|
||
|
||
Box contents, card placement, and interface assignments live in [state.md](state.md). This file carries the rules that stay true regardless of what's plugged in.
|
||
|
||
## Interfaces
|
||
|
||
- **Test pair**: the two ports whose modules hold the cable under test. Find by driver name, never by ethN — names shift with kernel link order.
|
||
- **Noise pair**: driven bad on purpose, link up/down cycled, intertwined with the test cable to inject alien crosstalk.
|
||
- **LAN uplink**: carries the default route. **Never repurpose or down it**; identify by default route, not name.
|
||
- Test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.
|
||
|
||
Module identities, EEPROM honesty (and why the X520 needs `allow_unsupported_sfp=1`), and the physical-layer model live in [modules/](modules/README.md).
|
||
|
||
## Host tuning (resets every reboot; re-apply before trusting results)
|
||
|
||
Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults.
|
||
|
||
1. **CPU governor → `performance`** on all cores. Biggest single contributor: under powersave, cores idle at 400 MHz, ramp too slowly, and drop frames at startup.
|
||
2. **Disable adaptive coalescing**: `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC.
|
||
3. **Rings deep**, raised toward the max via `ethtool -G` (RX 8160 / TX 4096 on the ice pair). Effect was not isolated in testing; don't claim it as a proven win.
|
||
4. **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp.
|
||
|
||
Verify a clean run shows `tx frames == rx frames` exactly before a run counts.
|
||
|
||
## irdma (dev host only)
|
||
|
||
`irdma` autoloads and binds ice ports, making `ETHTOOL_SCHANNELS` fail EBUSY ("Cannot change channels when RDMA is active") and failing the channels host check. `sudo rmmod irdma` (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.
|
||
|
||
## Running
|
||
|
||
Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a <portA> -b <portB>`.
|
||
|
||
- The harness starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1); do not follow runs with pgrep orphan checks — the group kill is the guarantee.
|
||
- It dumps the panel to `shots/<offset>.png` for remote assessment.
|
||
- The tell for two competing instances (an orphan, or the user's own hand-run): `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs. Ask rather than killing something that might be theirs.
|
||
- **Default config only** — no `-sizes`/`-streams` overrides, no editing the hardcoded `frameSizes`/`numStreams`/`batchSize` (same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.
|
||
- The first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) — not residual error.
|