docs/hardware.md

# Host rules

Box contents, card placement, and interface assignments live in [state.md](state.md). This file carries the rules that stay true regardless of what's plugged in.

## Interfaces

- **Test pair**: the two ports whose modules hold the cable under test. Find by driver name, never by ethN — names shift with kernel link order.
- **LAN uplink**: carries the default route. **Never repurpose or down it**; identify by default route, not name.
- Test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.

Module identities, EEPROM honesty (and why the X520 needs `allow_unsupported_sfp=1`), and the physical-layer model live in [modules/](modules/README.md).

## Host tuning (resets every reboot; re-apply before trusting results)

Required for zero baseline loss (without them baseline loss is 0.02–0.3% and masks cable faults) and for the power budget.

1. **CPU governor → `performance`** on all cores. Biggest single contributor: under powersave, cores idle at 400 MHz, ramp too slowly, and drop frames at startup.
2. **CPU max frequency → 2.0 GHz** (`scaling_max_freq`, all cores). Under the performance governor this is a pinned clock, not a ceiling, and it is the power budget: ~30 W package at line rate against ~80 W unbounded (measurement.md, "Power"). The value is host-silicon-bound — 1.6 GHz still holds line rate with thinner margin, 1.2 GHz cannot fill the wire — so re-derive it on new host silicon and re-qualify zero-loss.
3. **Disable adaptive coalescing**: `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC.
4. **Rings deep**, raised toward the max via `ethtool -G` (the tool asks RX 8160 / TX 4096; ixgbe takes it as-is). Effect was not isolated in testing; don't claim it as a proven win.
5. **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp.

Verify a clean run shows `tx frames == rx frames` exactly before a run counts.

## Running

Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a <portA> -b <portB>`.

- The harness starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1); do not follow runs with pgrep orphan checks — the group kill is the guarantee.
- It dumps the panel to `shots/<offset>.png` for remote assessment.
- The tell for two competing instances (an orphan, or the user's own hand-run): `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs. Ask rather than killing something that might be theirs.
- **Default config only** — never edit the hardcoded `frameSizes`/`numStreams`/`batchSize` for a run. The only flags are `-a`/`-b`; the compiled-in config is the one that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.
- The first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) — not residual error.
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+								# Host rules
-											Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons
										
										
											2026-08-10 10:34:26 -07:00
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+								Box contents, card placement, and interface assignments live in [state.md](state.md). This file carries the rules that stay true regardless of what's plugged in.
-											Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons
										
										
											2026-08-10 10:34:26 -07:00
 								## Interfaces
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+								- **Test pair**: the two ports whose modules hold the cable under test. Find by driver name, never by ethN — names shift with kernel link order.
 								- **LAN uplink**: carries the default route. **Never repurpose or down it**; identify by default route, not name.
 								- Test-path ports live in NetworkManager's unmanaged list (`/etc/NetworkManager/conf.d/99-unmanaged-10g.conf`), up with no IPv4.
-											Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons
										
										
											2026-08-10 10:34:26 -07:00
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+								Module identities, EEPROM honesty (and why the X520 needs `allow_unsupported_sfp=1`), and the physical-layer model live in [modules/](modules/README.md).
-											Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons
										
										
											2026-08-10 10:34:26 -07:00
 								## Host tuning (resets every reboot; re-apply before trusting results)
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
-											Power: 2 streams and a 2.0 GHz clock pinned by a new checkMaxFreq SETTINGS check (performance governor + scaling_max_freq = fixed clock) take line rate from 80 W to 29.7 W package, qualified 96 s zero-loss; measured on the bench: streams buy scheduling headroom not watts (C1-parked threads, lonelier spinners boost harder), relaxed cpu_dma_latency never reaches deep idle (scheduler spreads wakeups, ~2% residency), offlining E-cores saves 1.7 W and is wire-invisible mid-run, the cliff is TX starvation below 1.6 GHz (1.2 GHz sags to ~18 G, zero errors)
										
										
											2026-08-19 14:11:19 -07:00
+								Required for zero baseline loss (without them baseline loss is 0.02–0.3% and masks cable faults) and for the power budget.
-											Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons
										
										
											2026-08-10 10:34:26 -07:00
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+. **CPU governor → `performance`** on all cores. Biggest single contributor: under powersave, cores idle at 400 MHz, ramp too slowly, and drop frames at startup.
-											Power: 2 streams and a 2.0 GHz clock pinned by a new checkMaxFreq SETTINGS check (performance governor + scaling_max_freq = fixed clock) take line rate from 80 W to 29.7 W package, qualified 96 s zero-loss; measured on the bench: streams buy scheduling headroom not watts (C1-parked threads, lonelier spinners boost harder), relaxed cpu_dma_latency never reaches deep idle (scheduler spreads wakeups, ~2% residency), offlining E-cores saves 1.7 W and is wire-invisible mid-run, the cliff is TX starvation below 1.6 GHz (1.2 GHz sags to ~18 G, zero errors)
										
										
											2026-08-19 14:11:19 -07:00
+. **CPU max frequency → 2.0 GHz** (`scaling_max_freq`, all cores). Under the performance governor this is a pinned clock, not a ceiling, and it is the power budget: ~30 W package at line rate against ~80 W unbounded (measurement.md, "Power"). The value is host-silicon-bound — 1.6 GHz still holds line rate with thinner margin, 1.2 GHz cannot fill the wire — so re-derive it on new host silicon and re-qualify zero-loss.
 . **Disable adaptive coalescing**: `ethtool -C <dev> adaptive-rx off adaptive-tx off rx-usecs 25 tx-usecs 25`. Syntax/values differ per driver — re-derive per NIC.
 . **Rings deep**, raised toward the max via `ethtool -G` (the tool asks RX 8160 / TX 4096; ixgbe takes it as-is). Effect was not isolated in testing; don't claim it as a proven win.
 . **Socket buffers**: no sysctl change needed — the tool forces `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` (needs root) past the 4 MB `rmem_max` clamp.
-											Add in-repo project docs: goals, state, hardware, module-PHY transports, module diagnostics, and measurement lessons
										
										
											2026-08-10 10:34:26 -07:00
 								Verify a clean run shows `tx frames == rx frames` exactly before a run counts.
 								## Running
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+								Through `./harness`: `sudo go run ./harness -for 8s -at 4s -- -a <portA> -b <portB>`.
 								- The harness starts `go run .` in its own process group and kills the whole group on exit, confirming with `kill(-pgid,0)==ESRCH`. Do not wrap in `timeout` (kills `go run` but orphans the compiled binary to PID 1); do not follow runs with pgrep orphan checks — the group kill is the guarantee.
 								- It dumps the panel to `shots/<offset>.png` for remote assessment.
 								- The tell for two competing instances (an orphan, or the user's own hand-run): `RX pps > TX pps` from cross-received frames on shared ethertypes/MACs. Ask rather than killing something that might be theirs.
-											Drop goals and open-questions docs, cut PTP from the appliance kernel
										
										
											2026-08-28 21:59:41 -07:00
+								- **Default config only** — never edit the hardcoded `frameSizes`/`numStreams`/`batchSize` for a run. The only flags are `-a`/`-b`; the compiled-in config is the one that must work and the only one comparable across runs; small-frame runs measure the host, not the cable. Ask before running a specific non-default case.
-											Restructure docs: per-device directories under modules/ and nics/ with co-located datasheets, state centralized in state.md, prose tightened into lists and tables
										
										
											2026-08-12 19:23:02 -07:00
+								- The first ~5 s of any run is a settling transient (flow rules, rings, workers coming up) — not residual error.