cabletest/docs/hardware.md

Hardware and host

The box

Single usable PCIe slot (Gen4 x8, currently the E810 — being replaced by the X520). The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.

Interfaces

• Test pair: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order). Currently the X710 pair enp4s0f0np0 / enp4s0f1np1 (i40e); becomes the X520 pair when it arrives.
• Noise pair: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. Was the i40e pair during the E810-test-path era.
• enp89s0 (igc): this box's LAN uplink with the default route. Never repurpose or down it.
• Both test-path ports live in NetworkManager's unmanaged list (/etc/NetworkManager/conf.d/99-unmanaged-10g.conf), up with no IPv4.

The media lies

ethtool reports Port: FIBRE / 10000baseSR on the test ports, and the modules' own EEPROMs claim fiber identities (SR, 850 nm, LC connector, multimode fiber lengths, even fake optical DOM). All false. Every test module is a copper RJ45 10GBASE-T module with a cloned/lying EEPROM — none are fiber. The cheap RJ45 SFP+ modules clone a real optical module's EEPROM to pass NIC compatibility checks, so a module whose part number reads SFP-10G-SR is still 10GBASE-T copper. The real media is a 10GBASE-T PHY inside each module.

Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin ip link set down does NOT drop the wire unless the i40e link-down-on-close priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE.

Host tuning (resets every reboot; re-apply before trusting results)

Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults.

1. CPU governor → performance on all cores.
   • Biggest single contributor.
   • Under powersave, cores idle at 400 MHz and ramp too slowly, dropping frames at startup.
2. Disable adaptive coalescing.
   • ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25.
   • Syntax/values differ on ixgbe — re-derive for the X520.
3. Rings deep, raised toward the max via ethtool -G (RX 8160 / TX 4096 on the ice pair).
   • Effect was not isolated in testing; don't claim it as a proven win.
4. Socket buffers need no sysctl change.
   • The tool forces SO_RCVBUFFORCE/SO_SNDBUFFORCE (needs root) to bypass the 4 MB rmem_max clamp.

Verify a clean run shows tx frames == rx frames exactly before a run counts.

irdma (dev host only)

irdma autoloads and binds the ice ports, making ETHTOOL_SCHANNELS fail EBUSY ("Cannot change channels when RDMA is active"), which fails the channels host check. sudo rmmod irdma (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.

Running

Through ./harness: sudo go run ./harness -for 8s -at 4s -- -a enp1s0f0np0 -b enp1s0f1np1. It starts go run . in its own process group and kills the whole group on exit, confirming with kill(-pgid,0)==ESRCH. Do not wrap in timeout (kills go run but orphans the compiled binary to PID 1) and do not follow every run with a pgrep orphan check — the group kill is the guarantee. The tell for two competing instances (an orphan, or the user's own hand-run) is RX pps > TX pps from cross-received frames on shared ethertypes/MACs — ask rather than killing something that might be theirs. ./harness also dumps the panel to shots/<offset>.png for remote assessment.

Test runs use the default config only — no -sizes/-streams overrides, and no editing the hardcoded frameSizes/numStreams/batchSize (that's the same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.