cabletest/docs/hardware.md

Hardware and host

The box

Single usable PCIe slot (Gen4 x8) holding the X520-DA2, which trains at its own Gen2 ceiling (5 GT/s ×8, 32 Gb/s); the ConnectX-5 replaces it when it arrives. The X710 is not in that slot; it hangs off a CPU x4 port (Gen3 x4, ~31.5 Gbps/dir — enough for 2×10G full duplex despite the driver's worst-case "insufficient bandwidth" warning). Many CPU cores; goroutine-heavy designs welcome.

Interfaces

• Test pair: the two ports whose modules hold the cable under test. Found by driver name, never by ethN (ethN shifts with kernel link order — the X710 pair has already been enp4s0f* and enp3s0f* across reboots). Currently the X520 pair enp1s0f0 (new Wiitek) / enp1s0f1 (FS), ixgbe.
• Noise pair: driven bad on purpose, intertwined with the test cable to inject crosstalk. Cycled link up/down. The X710 pair enp3s0f0np0 / enp3s0f1np1 (i40e).
• enp88s0 (igc): this box's LAN uplink with the default route (its sibling port enp89s0 is dark). Never repurpose or down it; identify by default route, not name.
• Both test-path ports live in NetworkManager's unmanaged list (/etc/NetworkManager/conf.d/99-unmanaged-10g.conf), up with no IPv4.

The media lies (mostly)

Every test module is a copper RJ45 10GBASE-T module — none are fiber, whatever the EEPROM claims; the real media is a 10GBASE-T PHY inside each module. But EEPROM honesty varies by vendor, with driver consequences:

• The Wiitek lies: LC connector, 850 nm SR, multimode lengths, fake optical DOM (the temperature is real, the "laser" powers are theater). The clone exists to pass NIC compatibility checks — and it works: stock ixgbe qualifies it without complaint.
• The FS SFP-10G-T-100 is honest: connector RJ45 (0x22), extended transceiver code 10GBASE-T Short Reach, 100 m copper, no fake DOM. Stock ixgbe rejects exactly that honesty as an unqualified type and fails the whole port's probe (error -95, no netdev at all) — the driver must load with allow_unsupported_sfp=1, which is therefore mandatory on the X520, and it's the truthful module that requires it.
• The 10Gtek claims SFP-10G-SR; a module whose part number reads SR can still be 10GBASE-T copper.

Any physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — not any optical model. The module PHYs keep the copper link trained on their own — an admin ip link set down does NOT drop the wire unless the i40e link-down-on-close priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s). i40e/X710 has no EEE.

Host tuning (resets every reboot; re-apply before trusting results)

Required for zero baseline loss; without them baseline loss is 0.02–0.3% and masks cable faults.

1. CPU governor → performance on all cores.
   • Biggest single contributor.
   • Under powersave, cores idle at 400 MHz and ramp too slowly, dropping frames at startup.
2. Disable adaptive coalescing.
   • ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25.
   • Syntax/values differ on ixgbe — re-derive for the X520.
3. Rings deep, raised toward the max via ethtool -G (RX 8160 / TX 4096 on the ice pair).
   • Effect was not isolated in testing; don't claim it as a proven win.
4. Socket buffers need no sysctl change.
   • The tool forces SO_RCVBUFFORCE/SO_SNDBUFFORCE (needs root) to bypass the 4 MB rmem_max clamp.

Verify a clean run shows tx frames == rx frames exactly before a run counts.

irdma (dev host only)

irdma autoloads and binds the ice ports, making ETHTOOL_SCHANNELS fail EBUSY ("Cannot change channels when RDMA is active"), which fails the channels host check. sudo rmmod irdma (usage count 0; returns on reboot). The appliance kernel has no irdma. ice also refuses channel changes while ntuple rules exist — cabletest clears its own first.

Running

Through ./harness: sudo go run ./harness -for 8s -at 4s -- -a enp1s0f0np0 -b enp1s0f1np1. It starts go run . in its own process group and kills the whole group on exit, confirming with kill(-pgid,0)==ESRCH. Do not wrap in timeout (kills go run but orphans the compiled binary to PID 1) and do not follow every run with a pgrep orphan check — the group kill is the guarantee. The tell for two competing instances (an orphan, or the user's own hand-run) is RX pps > TX pps from cross-received frames on shared ethertypes/MACs — ask rather than killing something that might be theirs. ./harness also dumps the panel to shots/<offset>.png for remote assessment.

Test runs use the default config only — no -sizes/-streams overrides, and no editing the hardcoded frameSizes/numStreams/batchSize (that's the same violation). The default is the config that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.