cabletest/docs/nics/x520

Intel X520-DA2 (82599, ixgbe)

The product NIC — the only certain arbitrary-framing module-I2C transport (bit-banged, no firmware in the path). Its one gap — a single serial 1588-event latch, no all-packet RX timestamps — doesn't matter under the smoothed-bucket rate design below: the displayed rate is a smoothed throughput headline computed from software-read counts, and every fault verdict comes from per-frame sequence/CRC accounting, never the rate. The committed tree implements this design.

• PCIe Gen2 ×8 (5 GT/s, 32 Gb/s raw, ~25–26 Gb/s/dir effective vs 20 needed) — enough for 2×10G full duplex at the default mix; the 64 B case was host-bound already on the E810. Verify 5 GT/s ×8 trained (lspci -vv).
• Loss attribution survives here: missed-packet (RXMPC → rx_missed_errors) and per-queue drop (QPRDC) counters — "prove host-side zero" works.
• The RX error counter set is the 82599's slimmer one (counters.go): no jabber, fragment, illegal-byte or MAC-fault counters exist — crc/missed/length errors are what this link reports; the ice set remains the richer reference.
• ixgbe's mixed rx/tx interrupt vectors reject a tx-specific coalesce value; checkCoalesce falls back to rx-shared-with-tx on EINVAL, which is the normal path here. The full rx=8160/tx=4096 ring ask is taken as-is.
• allow_unsupported_sfp=1 is mandatory (ixgbe_main.c:165): the FS module's honest 10GBASE-T EEPROM fails Intel qualification and kills the whole port probe (error -95, no netdev). load-ixgbe passes it.

The smoothed-bucket rate: read-time buckets, backward smear, no hardware stamps

The per-frame hardware RX timestamp requirement existed for exactly one consumer: bucketing received frames by arrival time so the displayed rate stays honest while the host's reads jitter. Read-time bucketing plus a deconvolution pass replaces arrival-time bucketing, dissolving the requirement — and with it, the reason any firmware-managed NIC was ever needed.

• Rate is a headline, never a verdict. Lost/late/corrupt come from per-frame sequence numbers and CRC checks, independent of the rate; no cable-health decision keys off the rate number. This is the license for everything below.
• Buckets are keyed by read time; excess above line rate moves backward. Read jitter is a purely backward smear — a frame is read at or after its arrival, never before — so a bucket's excess above line rate is always frames that arrived earlier and were read late, and moving that excess backward to fill earlier deficits is the physically correct deconvolution, not an estimate. The dominant jitter shape (host stalls, then drains the backlog) is deficit-then-burst, which is exactly what the backward pass repairs. Line rate is known, so over/under is well-defined.
• A forward pass is optional edge polish. A stall/burst pair entirely inside the window balances under the backward pass alone; only pairs cut by the window boundary leave orphan excess or deficit. Pairing those up forward is cosmetic, is the pass most likely to paper over a genuine dip at the window edge, and is droppable if it ever lies.
• Window > worst host read-stall; window = display latency. Sized comfortably past the worst stall, pairs stop straddling the edge. The window is also the bound: no credit pool, no saturation risk — excess travels no farther than the window — and its length is how far behind real time the displayed rate runs. Currently smearWindow = 4 buckets = 64 ms at the 16 ms bucket; resize after measuring the actual stall distribution.
• Faults stay sharp. The pass only moves real excess and never invents frames. A genuine wire loss is a deficit with no matching excess anywhere in the window and displays at full magnitude in its own bucket — unlike a moving average, which would smear it thin across the window. Host jitter flattens; faults do not blur.
• The two honest cases: truly at line rate with lumpy reads displays flat line rate; truly below line rate has nothing to move and displays as-is.

"Late" is a sequence-number notion, never a rate one. The implementation: buckets keyed by one shared host clock read once per drained batch (rateEpoch), and a settled window — each completed bucket enters once, donates its excess above line rate backward once (fillBack, wire-byte capacity, frames riding in the donor's proportion, mutation persisted), and pops for display once no later bucket can still refill it. Settlement must be once-per-bucket, not a per-sample recompute over the sliding window — the recompute form double-counts moved excess and reads above line rate (measurement.md). The committed path requests no hardware timestamps at all; the forward pass is not built.

What this design carries in practice:

• The FS links and runs full diagnostics here (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative.
• Built on the committed X520/BCM/ECD implementation, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.

sff_i2c diagnostics transport (patched driver, validated)

Patched ixgbe (~/work/ixgbe-sff/) adds ixgbe_i2c_raw_write/ixgbe_i2c_raw_read (ixgbe_phy.c) — arbitrary START…STOP transactions from the driver's bit-bang primitives, swfw-semaphore bracketed — behind /sys/kernel/debug/ixgbe/<pci>/sff_i2c (w <addr8> <bytes…> / r <addr8> <n>).

• SDA-release fix (required): multi-byte reads returned only byte 0 — the master's ACK left SDA driven low and nothing released it (stock 82599 paths never clock in more than one byte, so the missing release was invisible). raw_read releases SDA after each ACK, mirroring stock's release block after clocking out a byte.
• Two-master windows closed by the compound op: each simple w/r takes the semaphore separately, so multi-op protocol sequences have unlocked gaps where the driver's event-driven I2C (SFP identify, link setup, ethtool -m) can interleave — traffic that fires exactly around cable swaps, when diagnostics also run, so collisions are rare but correlated with the interesting moments. Serialization itself holds (driver SFP traffic uses the same byte primitives under the same phy_semaphore_mask), and AT24-style word pointers do persist across STOP — the hazard is the other master moving the pointer or consuming the BCM 0x56 bridge's pending read data. The x <waddr> <raddr> <delay_us> <n> <wbytes…> command (ixgbe_i2c_raw_xfer) runs write→STOP→in-kernel delay→read under a single swfw hold, closing the window by construction; the write/read primitives are split into locked/unlocked halves so x reuses them under one acquire.
• Compound op proven on the FS BCM (~/work/phydiag-work/compound_test.py): PHY ID and cmd STATUS read identically through the single-hold op and the old split sequence (0x3590 / 0x5081 / 0x0004); a ~3 ms hold is shorter than the driver's own SFP-identify holds. The delay sweep corroborates the hazard the op closes: at delay 0 the bridged read returns stale data (the prior read's 0x0004), at ≥200 µs it reads correctly. The >1 ms BCM delay is validated at 3 ms; treat 0x0000 reads as retry-with-longer-delay — 0 is also the bridge's not-ready signature.
• RollBall split transactions are field-proven — the i40e oracle path drove RollBall entirely with single-byte, separate-STOP transactions on both the Fibergaga and the original Wiiteks; the kernel's combined form is not required by the modules.

Register/timestamp bench transport (patched driver)

For the timestamp/counter bench pokes the driver also carries reg_ops read <reg> returning the value through the file (not just dmesg) and a ts_bench command running the latch poll/re-arm loop in-kernel (arm <ctl>, control, latchrate <ms>) — needed because the BAR is unmappable from userspace under IO_STRICT_DEVMEM. Poke driver: ~/work/phydiag-work/x520poke.

Timestamping facts

No all-packet exact RX timestamping, and no path to it — measured, settled, and no longer disqualifying. The 82599 cannot stamp the 0x88b5 test traffic per-frame; the smoothed-bucket plan above removes the need. The facts stand for the record:

• RX latches 1588-recognized frames only — confirmed on hardware. Writing TYPE_ALL (0x08 = Type 100b) into TSYNCRXCTL sticks in the register but never arms the latch under raw-frame flood — and the datasheet marks 100b reserved on the 82599 (it's an X550 define); TSIP_UT_EN (0x00800000) will not even hold (X550+; HWTSTAMP_FILTER_ALL → -ERANGE, ixgbe_ptp.c:1043). One serial latch pair (RXSTMPL/H, locked until RXSTMPH read), attribute capture (RXSATRL/H — stream seq lands in RXSATRH[31:16] byte-swapped), no RX-descriptor timestamp field.
• RX latch eligibility is configurable, not hardwired to 0x88F7 (datasheet §7.9.3.2, §8.2.3.26.1, ETQF bit 30): a frame qualifies when its EtherType sits in an ETQF slot with the 1588 bit set and TSYNCRXCTL.Type admits its message class — valid encodings 000b (L2 sync/delay_req + msgID 2,3), 001b (V1/L4), 010b (V2 L2+L4), 101b (every V2 event frame: payload byte 0 low nibble with bit 3 clear). Declaring 0x88b5 as 1588 and shaping byte 0 would latch our own raw frames — datasheet-derived, untested on hardware, and irrelevant to the verdict: the serial ceiling is unchanged.
• TX latch has no parser (§8.2.3.26.7): the advanced TX data descriptor's 1588 bit (MAC field, bit 1) stamps any marked frame into TXSTMPL/H — serial, TXTT cleared by TXSTMPH read. Both directions therefore stamp arbitrary frames one-at-a-time: a ping-pong probe gets PHY-adjacent hardware stamps on both ends, paced by the MMIO read loop.
• The single latch services ~383 k stamps/s in a dedicated in-kernel poll+re-arm loop (measured) against ~1.78 M pps/dir line rate — serial, so it stamps a fraction of frames at best and none of the line-rate traffic. NIC-timestamp length measurement is discarded (length strategy: ../../modules/README.md), so the latch has no committed use.
• Per-port free-running SYSTIME (~6.4 ns at 10G), one shared crystal; the two SYSTIMEs start at driver load and can sit seconds apart.
• SDP auxiliary elements (§7.9.4): an edge on SDP2/3 latches SYSTIME into AUXSTMP0/1; target-time registers assert an SDP pin when SYSTIME reaches a programmed value; SDP6/7 carry configurable clock outputs. No product use.
• Per-queue counters are not a substitute: they're a rate, honest only in zero-loss steady state, and a cable tester measures the receive precisely when it is not — it cannot place a single frame in time, so it cannot see the fault. Measured facts retained for the record: RQSMR maps queue→counter 1:1; QPRC/QBRC exact, read-clear; the driver's 2 s stats watchdog reads-and-clears the same registers (ixgbe_main.c:7780,7789,7822). Also rejected: wire-determinism reconstruction (per-frame times from the size schedule assume saturation regularity — blind exactly at loss/retrain events).

RX steering facts

• Stock ntuple ETHER_FLOW is rejected (ixgbe_flowspec_to_flow_type: TCP/UDP/SCTP-v4 and IPv4 only). The patched driver maps exact-ethertype ETHER_FLOW inserts onto ETQF slots, so the tool runs unchanged; ETHTOOL_GRXCLSRULE/GRXCLSRLALL round-trip the rules so stale-rule cleanup sees and clears them.
• ETQF/ETQS L2 EtherType steering is proven on hardware (~/work/phydiag-work/etqfbench): a dedicated ethertype→queue stage checked at L2, ahead of RSS/fdir — ETQF holds the 16-bit ethertype + FILTER_EN, ETQS holds RX_QUEUE + QUEUE_EN, and the RX descriptor reports the matched index (RXDADV_PKTTYPE_ETQF). 7 rules 0x88b5–0x88bb → queues 0–6 land exactly 1000 frames each; unfiltered ethertypes fall to queue 0 (non-IP frames hash to RSS 0); delete restores the fall-through.
• Slot budget fits: 8 filters; the patch reserves index 3 (1588) and refuses ethertype 0x88F7 → 7 free = exactly the 7 streams. Stock FCoE writes moved behind the FCoE-enabled guard (without that, FCoE CRC pre-claims slot 2). Restore-on-link-up (ixgbe_etqf_filter_restore) and clear-on-close (ixgbe_etqf_filter_exit) are hooked so a ring-resize reset re-applies the rules.
• Flow Director is a dead end for raw L2 — confirmed on hardware: 82599 fdir classifies IPv4/IPv6 only; a flex-word rule at the ethertype offset (FDIRCTRL_FLEX_SHIFT = 0x6) on a raw 0x88b5 stream steers nothing (etqfbench -mode fdirneg) — the flex word only narrows an IP-typed match.
• Unused fallbacks: MAC→VMDq pool steering (distinct dest MACs → queues, fully raw; ETQF even has a pool field); UDP encap acceptable — raw-L2 steering was never a hard NIC requirement. ETQF stays preferred (no framing change, no header overhead in the size mix).