All three stashes distilled into docs and dropped: xdp.md carries the AF_XDP record (ixgbe's 3 KB single-buffer XDP cap excludes the locked jumbo mix; XSK need_wakeup TX as the spin-free flow control, zero-copy-or-die, the stray-RX fill-ring trap, BTF-resolved stamp kfunc, wire-weighted size schedule as an unadopted option); connectx-5 README gains the PCICONF/ICMD/MCIA gateway constants so a rebuild needs no mstflint re-mine; e810 keeps the read-side transport map (cage-cache, bus=port, bus-5 phantom, 16-bit-addr hazard, latency as wire-truth) and NVM rollback pointers; x710 the >=30 ms EEPROM write pacing; fibergaga the empirical vendor-space finds; fs the inert-I2C-surface map; state.md stash section now empty

This commit is contained in:
flamingcow
2026-08-19 14:42:13 -07:00
parent 4c6de3b618
commit 2d3369af85
10 changed files with 88 additions and 12 deletions
+11 -2
View File
@@ -2,7 +2,7 @@
## Status: retired — the X520 smoothed-bucket plan removes the need for this card
The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR, BCM+length dropped) is **parked in a git stash** ("CX-5 Wiitek…"), not on the tree; it runs (line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules) and stays available as a parts bin. Everything below is the record of what was proven and what was never resolved.
The all-packet stamps were this card's only edge, and the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves that requirement; what remains is the cost — firmware-mediated module I2C with the ~150 ms mailbox quantum below, plus the FS link deadlock. The full Wiitek-pair implementation (MCIA transport, dual-end SNR with the headline as worst pair across both ends, BCM+length dropped) ran on hardware — line-rate-ish traffic, all-packet stamps, real dual-end SNR from both physical modules — and its code is dropped; this file is the complete record (including the gateway constants below), with mstflint as the reimplementation reference. Everything below is what was proven and what was never resolved.
**Known:**
@@ -72,6 +72,15 @@ Two things that are not optional and one that is not needed:
There are no retry loops in the path: the gateway flag settles within its own on-chip access (faster than the PCI round-trip that reads it back, so a single read either sees it or faults), and the only waits left are genuine completion polls for the firmware command and the module's own mailbox.
### PCICONF gateway constants (verified live; rebuildable without re-mining mstflint)
- Vendor-specific capability ID **0x09**; offsets from the cap base: **ctrl 0x4** (low 16 bits select the address space; a readback with bits 31:29 zero means the space is unsupported), **counter 0x8**, **semaphore 0xC**, **addr 0x10**, **data 0x14**.
- The semaphore is a ticket lock: free at 0; taken by writing the counter register's current value and reading it back unchanged. Address spaces: **CR 0x2**, **ICMD 0x3**.
- One dword op: for a write, data register then addr with bit 31 set; for a read, addr with bit 31 clear, then read data. The flag bit toggles inside the device's own access (clear = write done, set = read done) — always settled before the PCI round-trip that reads it back, so an unsettled flag is a fault, not a wait.
- ICMD space: **ctrl at 0x0** (opcode bits 31:16, status bits 15:8, go bit 0 — set to fire, cleared by firmware on completion, observed < 5 ms), **mailbox size at 0x1000**, **mailbox at 0x100000**. CR-space **0xB5E04 bit 31** = firmware static config not done; check before any command.
- Command **FLASH_REG_ACCESS 0x9001**: operation TLV (type 1, 4 dwords — register ID at bytes 4–5, method byte 6 with GET=1/SET=2, class byte 7 = 1 for register access; completion status returns in byte 2 & 0x7F) + reg TLV (type 3, length in dwords) + register payload, all big-endian dwords in the mailbox.
- MCIA register ID **0x9014**, 16-byte header + 48 data bytes: module at byte 1, per-access status at byte 3, 8-bit I2C device address at byte 4, device offset at byte 7, size at bytes 10–11, data from byte 16.
## The FS cannot link on the CX-5 — host-interface deadlock
The FS's XFI never comes up against this firmware; **diagnostics to the FS work regardless** (all of the above ran with the link down). The mechanism, each side observed directly:
@@ -97,7 +106,7 @@ Paths forward if this card ever returns, in escalation order:
- SFP28 autoneg does not settle with these modules; `ethtool -s <dev> speed 10000 autoneg off` is the working configuration on the module ports.
- Mellanox is permissive with third-party modules, confirmed: the honest FS EEPROM that i40e rejects and stock ixgbe qualifies out probes and transacts cleanly ("Cable plugged", full EEPROM/DOM access). Link bring-up is a separate story (above).
## Host-setup deltas from the ice/ixgbe path (the stashed implementation's `system.go`/`counters.go`)
## Host-setup deltas from the ice/ixgbe path
Runs at line rate on the Wiitek pair. What differs from the earlier NICs, all handled in the checks:
+11
View File
@@ -17,6 +17,17 @@ The richest per-frame RX error breakdown of the NICs used — `illegal_bytes` an
- Every write returns aq 1 (EPERM) — tested identically against EEPROM 0x50 and BCM 0x56, so the block is write-vs-read, not address-scoped. Intel policy, survives NVM update; NVM 5.01 hardened the 3.10 silent-drop into an explicit EPERM.
- Dead for anything needing writes: RollBall unlock needs a write; BCM SMI reads *start* with a write.
Read-side transport details (the patched ice exists for read-side work, so these stay load-bearing):
- CTX_OVERRIDE (ntc 0x50, handle 0) reaches the real wire; `index` is the raw I2C bus number and equals the port number (verified by serial), so one PF0 debugfs file reaches both cages. NACK reads report `err -5 aq 1`.
- Cage-node addressing (ntc 0x26/0x46, handle from the netlist) returns a **firmware cache** of the module EEPROM, not the wire: unknown regions read zeros, and the cache serves stale bytes after the wire changes. Don't use it.
- Override bus 5 ACKs every address with zeros — a firmware phantom.
- 16-bit-address mode (params bit4) turns the offset phase into a 2-byte write an SFF EEPROM can latch as data — never use it on 0x50/0x51.
- No offset-less read mode exists, so the plain mdio-i2c clause-45 read phase (bcm84881-style at 0x56) cannot be formed. RollBall fits fully (≤4 B native multi-byte writes).
- Latency is the wire-truth judge on this transport: on 3.10 firmware, silently-dropped 1-byte writes completed in ~63 µs while real wire traffic ran ~103 µs/byte (reads) and multi-ms ack-polling (writes), and `ok` returns even for writes at an empty address — protocol-level readback is the only delivery confirmation.
Firmware on the card: NVM 5.01 (release 31.2.2, EETRACK 0x80021FA2). The update package and logs live in `~/work/e810nvm/`; `nvmupdate64e` needs `-AllowMissingCert` on this platform (harmless — nothing boots from the NIC), writes no file backup, and rollback lives in the card's inactive flash bank (`nvmupdate64e -r -m <MAC>`).
## The bit-bang-below-the-AQ idea — retired with the timestamp requirement
The write block is at the **Admin Queue** (firmware) level — the `0x06E2/E3` topo-I2C commands are refused with EPERM. That is not the same as the module's SDA/SCL being unreachable: the 82599 reaches the module I2C **without firmware** by toggling SDA/SCL as GPIO bits in `I2CCTL` (the `~/work/ixgbe-sff/` patch bit-bangs arbitrary START…STOP from those primitives), and whether the E810 exposes an equivalent register-level / SDP-GPIO path below the AQ policy was never tested. The idea existed to marry all-packet stamps to raw I2C in one card; the smoothed-bucket rate plan ([../x520/README.md](../x520/README.md)) dissolves the stamps requirement, so the X520 covers everything and this investigation is not needed. It stays recorded as the E810's one open door if per-frame arrival times are ever wanted again — the investigation would start at the datasheet's SDP/GPIO and low-level I2C controller registers, asking whether the cage's SDA/SCL are host-addressable.
+1 -1
View File
@@ -24,7 +24,7 @@ The per-frame hardware RX timestamp requirement existed for exactly one consumer
What this design carries in practice:
- **The FS links and runs full diagnostics here** (ixgbe drives SFI idles from driver load — none of the mlx5 wait-for-module deadlock — and the ECD length path is proven), and the Wiitek's RollBall answers in <25 ms (the host is the sole I2C master; none of the CX-5's ~150 ms firmware quantum). The committed FS+Wiitek mixed pair — length from the FS ECD, IEEE per-pair SNR from the Wiitek — is the product configuration, with Wiitek+Wiitek (dual-end SNR, no length) as the selectable alternative.
- **Built on the committed X520/BCM/ECD implementation**, not the CX-5 stash. The CX-5 stash stays as a parts bin (dual-end SNR display, the transport interface) if wanted.
- **Built on the committed X520/BCM/ECD implementation.** The CX-5 implementation is dropped; its record, including the MCIA gateway constants, is [../connectx-5/](../connectx-5/README.md).
## `sff_i2c` diagnostics transport (patched driver, validated)
+1
View File
@@ -13,6 +13,7 @@
- Opcode 0x0629 get / 0x0628 set PHY register. param0 = phy_interface | dev_addr<<8 | cmd_flags<<16 (phy_interface 2 = EXTERNAL_MODULE, dev_addr 0xA2); param1 = reg offset; param2 = value.
- Result comes back in the dmesg "AQ desc WB" line, 9th field.
- Proven on the Fibergaga, link up: per-pair SNR (PMA 1.133–1.136), PCS 3.32/3.33, the 3.E820 counter ([../../modules/fibergaga/](../../modules/fibergaga/README.md)).
- **Write pacing**: single-byte writes issued ~7 ms apart can land inside the EEPROM's internal write cycle and be silently dropped — no AQ error, stale readback. Space writes ≥30 ms when framing a protocol; a client whose per-op cost is ≥11 ms (page-select + poll sleeps) never trips it, which is why the RollBall path did not.
## Timestamping