Mixed FS+Wiitek test pair: modules dispatched by EEPROM PN to a BCM or Go RollBall client (whitelist panics outside the wiitek proven-safe set), SNR now solely the wiitek's IEEE margins (BCM handler SNR dropped as unactionable), ECD-clears-AN-enable trap found and fixed (7.0 |= 0x1200 on every restart, explicit restart after each diag), mixed roles FS-slave/wiitek-auto-master, window re-arms at taken edge so the one CMD write per window sits far from the poll; X710 rejects the honest FS EEPROM so the noise pair runs Fibergaga+Wiitek
This commit is contained in:
@@ -26,6 +26,7 @@ Counts and current placement: [../state.md](../state.md).
|
||||
| 10Gtek | `SFP-10G-SR` part number | (not in the test set) |
|
||||
|
||||
- `allow_unsupported_sfp=1` is mandatory on the X520 — and it's the *truthful* module that requires it ([../nics/x520/](../nics/x520/README.md)).
|
||||
- The X710 rejects the FS outright: i40e firmware disables Rx/Tx ("unsupported SFP module type", logged only at driver bind) and i40e has no override parameter — the FS can only live behind ixgbe. The rejection is silent at insertion time; a dead port with `ethtool` reporting no supported link modes is the symptom.
|
||||
- Physical-layer reasoning must use 10GBASE-T: PAM16, LDPC FEC, self-synchronizing scrambler, 4 twisted pairs, distance/temperature sensitive — never an optical model.
|
||||
- The module PHYs keep the copper link trained on their own: an admin `ip link set down` does NOT drop the wire unless the i40e `link-down-on-close` priv flag is set (peer sees the drop in ~200 ms, relinks in ~0.9 s).
|
||||
|
||||
@@ -39,7 +40,7 @@ Proven on the CUX3610 (before the originals died) and the Fibergaga; the BCM not
|
||||
| PCS 3.32/3.33 | Block-lock loss, BER saturate, errored blocks | Latched, clear-on-read |
|
||||
| AN 7.33 | Link-partner status / master-slave | |
|
||||
| PMA 1.147 | Fast retrain status/count | Per-run quality metric |
|
||||
| AN 7.0 bit 9 | Restart AN — forces retrain, refreshes SNR minimums | Proven safe everywhere. An `ip link` bounce does not drop the copper line, so it won't retrain |
|
||||
| AN 7.0 | Restart AN (`\|= 0x1200`, enable + restart) — forces retrain, refreshes SNR minimums | Proven safe everywhere. Always force the enable bit too: the BCM ECD can leave 7.0.12 cleared ([fs/](fs/README.md)), and a bare restart preserves it. An `ip link` bounce does not drop the copper line, so it won't retrain |
|
||||
|
||||
## RollBall protocol (Marvell and Aquantia modules)
|
||||
|
||||
@@ -49,7 +50,7 @@ Clause-45 access to the internal PHY over I2C A2h, all messages ≤ 4 B, offset-
|
||||
2. Page-select: A2h 0x7F → 3.
|
||||
3. Mailbox: A2h 0x80 (cmd) / 0x81 (data), poll for DONE (0x04).
|
||||
|
||||
Split transactions (separate STOP per byte) are field-proven on both the Fibergaga and the original Wiiteks; the kernel `mdio-i2c` combined form is not required by the modules.
|
||||
The full mailbox map (0x80 CMD, 0x81 devad, 0x82/0x83 reg, 0x84/0x85 value) is vendor-confirmed in [wiitek/10gbase-t-phy-register-operation.md](wiitek/10gbase-t-phy-register-operation.md). Split transactions (separate STOP per byte) are field-proven on both the Fibergaga and the original Wiiteks; the kernel `mdio-i2c` combined form is not required by the modules. cabletest carries a Go client (`phy.go`): split ≤4-byte offset writes, single-hold compound reads, whitelist-guarded against the Wiitek brick space — proven live over the patched-ixgbe `sff_i2c` on the X520.
|
||||
|
||||
## Cable-length physics (applies to all vendors)
|
||||
|
||||
|
||||
@@ -33,11 +33,11 @@ Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-
|
||||
|
||||
| Capability | How | Observed |
|
||||
|---|---|---|
|
||||
| Per-pair SNR | `CMD_GET_SNR` 0x8030, **invoked bare** (writing the documented DATA1 display flag returns zeros; IEEE 1.133–1.140 never populate — constant 0x8080) | DATA2–5 = SNR A–D ×0.1 dB; ≈ 27–32 dB absolute on bench, 0.1–0.4 dB jitter. 10GBASE-T operating point ≈ 26.5 dB, so margin ≈ value − 26.5. cabletest shows the margin classified green ≥ 3 dB / amber ≥ 1 dB / red below — provisional thresholds until the graded-noise correlation run. **Each training re-converges differently**: back-to-back retrains with identical roles, temperature and noise moved one pair by 3.6 dB (roles pinned, launch power pinned — the convergence itself is the variable, and the two bench modules differ in stability: ≤1.5 dB spread on one, 3.6 dB on the other). A single training's absolute SNR is a weak qualification number; qualification wants median-of-N trainings, live SNR is honest as a within-session tracker |
|
||||
| Per-pair SNR | `CMD_GET_SNR` 0x8030, **invoked bare** (writing the documented DATA1 display flag returns zeros; IEEE 1.133–1.140 never populate — constant 0x8080) | DATA2–5 = SNR A–D ×0.1 dB; ≈ 27–32 dB absolute on bench, 0.1–0.4 dB jitter. 10GBASE-T operating point ≈ 26.5 dB, so margin ≈ value − 26.5 — a derived number that never proved actionable, so **cabletest does not poll it** (the panel's SNR comes from an IEEE-margin end — modules/wiitek/ — and each GET costs handler traffic on this µC). **Each training re-converges differently**: back-to-back retrains with identical roles, temperature and noise moved one pair by 3.6 dB (roles pinned, launch power pinned — the convergence itself is the variable, and the two bench modules differ in stability: ≤1.5 dB spread on one, 3.6 dB on the other). A single training's absolute SNR is a weak qualification number; qualification wants median-of-N trainings |
|
||||
| Die temperature | `CMD_GET_CURRENT_TEMP` 0x8031 | ~68–70 °C on bench |
|
||||
| Supply rails | `GET_CURRENT_VOLTAGE` 0x802F | 0.8 V and 1.88 V rails, tenths of mV |
|
||||
| Error counters | IEEE PCS 3.32/3.33 — block lock, latched errored-block/BER, clear-on-read | The noise-stress error proxy |
|
||||
| Retrain on demand | IEEE AN restart `7.0 \|= bit9` | Relinks in ~5–6 s |
|
||||
| Retrain on demand | IEEE AN restart `7.0 \|= 0x1200` (enable + restart — the ECD can leave the enable bit cleared, see below) | Relinks in ~5–6 s |
|
||||
|
||||
## Probed configuration states
|
||||
|
||||
@@ -52,7 +52,7 @@ Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-
|
||||
| Jumbo | 0x801C/0x801D | GET 0x801D answers in DATA1 — scratch-prone, unusable — and bringup cannot assume a cable to probe through, so cabletest forces enable every boot (SET 0x801C + AN restart); the running mix's 9018-byte frames are the standing wire truth (`phy.go`). The FS at its 9K option passes them at line rate with zero loss/corrupt |
|
||||
| 1588 | 0x8004 | Disabled (engine registers undocumented — see asks below) |
|
||||
| Limited reach | 0x8006 | Disabled — its DATA3 "linked in LR mode" bit, the only handler-visible trace of the DSP length estimate, is unavailable without a config change |
|
||||
| Master/slave role | IEEE 7.32 bits 15/14 | Left to AN the resolution is a per-training lottery; manual config writes stick exactly and the resolution follows deterministically (7.33.14, no config fault with complementary ends). cabletest forces A master / B slave every boot (`phy.go`) so sessions are measured under identical conditions |
|
||||
| Master/slave role | IEEE 7.32 bits 15/14 | Left to AN the resolution is a per-training lottery; manual config writes stick exactly and the resolution follows deterministically (7.33.14, no config fault with complementary ends). cabletest forces A master / B slave every boot when both ends are BCM; with a non-BCM partner (whose manual config is unreachable) the BCM is forced slave and the partner auto-resolves master — proven linking in ~3 s against the Wiitek (`phy.go`) |
|
||||
| Launch power | IEEE 1.130, 1.0.11 | 1.130 reads 0x0003 and is firmware-pinned: host writes to the short-reach bit revert instantly and the value never changes across retrains, so launch power is constant and not host-influenceable. PMA low-power bit clear. The handler catalog has no TX-power command |
|
||||
|
||||
## Firmware-reliability notes
|
||||
@@ -61,7 +61,7 @@ Protocol and full verified catalog: [bcm84891l-mdio-commands.md](bcm84891l-mdio-
|
||||
|
||||
**The internal poll's clock, measured on hardware.** One poll every 3.49–4.22 s, regime-dependent but rock-stable within a regime (±15 ms): ≈3.50 s warm and idle, ≈3.70 s, ≈4.01 s cool and idle, ≈4.21 s under continuous host mailbox load — host traffic *defers* the poll, never advances it, and nothing observed brings two polls closer than 3.49 s. The poll is link-independent: it runs straight through an AN restart and the whole retrain without missing a beat. Around each poll's service the bridge serves stale data for ~50 ms: every read returns the last value successfully fetched before the µC went busy (stuck-at-last-fetch, not one-behind), which is exactly how 3.33 inherits 1.147's 0x0011. The window closes *before* the poll is detectable — a CMD read returning the fresh resident 0x0031 is itself proof the µC serviced the fetch, so flip detection doubles as the all-clear.
|
||||
|
||||
**The windowed protocol (`phy.go`) — contention resolved by time-division.** Since the poll cannot be silenced, the host schedules around it. One goroutine per module owns every transport touch (requests execute one at a time on the loop — nothing else can reach the wire, by construction), and each request is admitted only inside a fixed 3.4 s window following an observed poll (CMD flipping to resident 0x0031, polled at 10 ms); at the cutoff the host goes quiet until the next flip. 3.4 s sits below every observed cadence, so a regime switch mid-run lands harmlessly in vacated time — no prediction, no period tracking, just the last observed flip plus two constants. **Re-lock — no window at assumed phase**: edge detection needs CMD ≠ 0x0031, and normal work leaves it armed (every handler command rewrites CMD), but after a gap with no commands — a diag plus its relink wait, an idle stretch, a poll that skipped its GET because the link bit read down — the resident is 0x0031 and the phase is unknown. Admission then arms first (a bare GET_PAIR_SWAP, resident 0x0000) and anchors only on the true edge that follows, so every window is edge-locked from first contact at boot; bringup pays one phase acquisition per module (~2–4 s). A missing heartbeat free-runs one window rather than stall (never observed — the heartbeat survives retrains). Before re-lock the post-diag window anchored blind on the stale resident, and every ghost observed under windowing sat in exactly that window; with re-lock, ghost-free across all bench diags to date. A poisoned batch is self-consistent and forensically visible: the stuck value clears the link bit (SNR skipped, panel shows "-", the ghost-SNR panic sidestepped) while 3.33 charges the same value as errored blocks (0x0011 → the "+17"); every corrected charge and link-down reading prints its raw 1.1/3.33/1.147 on the console, so events self-attribute — a ghost names its neighbor register, real correction decodes as nothing but itself. Validated on hardware in `~/work/phydiag-work/bcm_phaselock_bench.py`: aiming read bursts at the poll reproduces the poisoning on 100% of polls (the "+17" manufactured on demand); windowed operation ran 2,881 back-to-back batches — 16× the production rate, riding the cutoff — with zero stale values, and the production 1 Hz stream is unperturbed (blackout hides inside natural gaps, worst sample gap ~1.6 s).
|
||||
**The windowed protocol (`phy.go`) — contention resolved by time-division.** Since the poll cannot be silenced, the host schedules around it. One goroutine per module owns every transport touch (requests execute one at a time on the loop — nothing else can reach the wire, by construction), and each request is admitted only inside a fixed 3.4 s window following an observed poll (CMD flipping to resident 0x0031, polled at 10 ms); at the cutoff the host goes quiet until the next flip. 3.4 s sits below every observed cadence, so a regime switch mid-run lands harmlessly in vacated time — no prediction, no period tracking, just the last observed flip plus two constants. **Re-lock — no window at assumed phase**: edge detection needs CMD ≠ 0x0031, and handler commands leave it armed (every one rewrites CMD); whenever CMD sits at the resident 0x0031 at expiry the phase is unknown and admission arms first, then anchors on the true edge that follows. Steady-state polling issues no handler commands of its own (the SNR GET is dropped), so each taken edge immediately re-arms with a bare GET_PAIR_SWAP: the single CMD write per window lands at the start of the quiet period, maximally far from the next poll — the wedge risk is writes near the poll — and every later expiry reads the phase without writing. One command per ~4 s window replaces a handler GET per second. Admission then arms first (a bare GET_PAIR_SWAP, resident 0x0000) and anchors only on the true edge that follows, so every window is edge-locked from first contact at boot; bringup pays one phase acquisition per module (~2–4 s). A missing heartbeat free-runs one window rather than stall (never observed — the heartbeat survives retrains). Before re-lock the post-diag window anchored blind on the stale resident, and every ghost observed under windowing sat in exactly that window; with re-lock, ghost-free across all bench diags to date. A poisoned batch is self-consistent and forensically visible: the stuck value clears the link bit (SNR skipped, panel shows "-", the ghost-SNR panic sidestepped) while 3.33 charges the same value as errored blocks (0x0011 → the "+17"); every corrected charge and link-down reading prints its raw 1.1/3.33/1.147 on the console, so events self-attribute — a ghost names its neighbor register, real correction decodes as nothing but itself. Validated on hardware in `~/work/phydiag-work/bcm_phaselock_bench.py`: aiming read bursts at the poll reproduces the poisoning on 100% of polls (the "+17" manufactured on demand); windowed operation ran 2,881 back-to-back batches — 16× the production rate, riding the cutoff — with zero stale values, and the production 1 Hz stream is unperturbed (blackout hides inside natural gaps, worst sample gap ~1.6 s).
|
||||
|
||||
## ECD — recovered from the OpenBCM SDK, proven on hardware
|
||||
|
||||
@@ -76,6 +76,7 @@ The ECD register mechanism is absent from the handler catalog and the datasheet
|
||||
Observed on a plugged, linked, healthy cable: verdicts OK×4 and per-pair lengths of [45, 45, 41, 46] on a ~45 m cable — **meter-accurate with no calibration**, and this ECD reports length for healthy pairs, not just faults, resolving the terminated-far-end concern in [../README.md](../README.md). Caveats:
|
||||
|
||||
- The run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear) — do not run mid-measurement until the disturbance is characterized.
|
||||
- **The run can leave clause-45 AN disabled** (7.0.12 cleared — observed live after diags racing an in-progress retrain): with AN off the BCM emits no AN pulses, both ends report no link partner (7.1 LP-AN-able 0), and the link stays down through AN restarts, role changes and PMA resets until the bit is re-enabled. cabletest restarts AN with `7.0 |= 0x1200` (enable + restart) after every diag, and `devRestartAN` always forces the enable bit — a bare bit-9 restart preserves the cleared enable forever.
|
||||
- **An interrupted run can wedge the µC's SMI service** (observed after a process died mid-diag): the bridge ACKs I2C but serves 0x0000 for every register at any delay and silently drops MDIO writes — no in-band recovery, not even PMA reset — while the EEPROM path and the trained link keep working. Recovery: a driver unbind/rebind of the port (the SFP re-initialization resets the µC); a reseat would do the same.
|
||||
- Fault verdicts (open/short/inter-pair) are unexercised — deliberately: the product is a closed-loop tester, both ends always plugged.
|
||||
- Family constraints from the SDK: port must be enabled; unsupported at forced 100M.
|
||||
|
||||
@@ -21,14 +21,23 @@ The shared IEEE 802.3an set ([../README.md](../README.md)) plus two safe vendor
|
||||
|
||||
| Registers | What | Reading |
|
||||
|---|---|---|
|
||||
| PMA 1.133–1.136 / 1.137–1.140 | Per-pair SNR margin (offset-0x8000, 0.1 dB) | [8.8, 7.0, 7.3, 8.9] dB |
|
||||
| PMA 1.133–1.136 / 1.137–1.140 | Per-pair SNR margin (offset-0x8000, 0.1 dB) | [8.8, 7.0, 7.3, 8.9] dB (as auto-master vs FS manual-slave: [9.2, 7.2, 7.0, 8.7]) |
|
||||
| PCS 3.32 / 3.33 | Errored-block / block-lock latches | 0x1001 / 0x7f00 |
|
||||
| AN 7.33 | Master/slave, link partner | 0x3c62 |
|
||||
| AN 7.33 | Master/slave, link partner | 0x3c62 (as master against the FS: 0x7c62) |
|
||||
| 1.0xC011 / 12 | Firmware version | 0x0c01 / 0x0500 (12.1.5.0) |
|
||||
| 1.0xC050 | Boot status | 0x087e |
|
||||
| PMA 1.0 | Speed selection | 0x2040 (10G) — see the reset trap below |
|
||||
| AN 7.0 / 7.1 / 7.32 / 7.60 | AN control/status, 10GBASE-T control, EEE advert | 0x3000 / 0x0008 idle · 0x00ad linked / 0x11e1 (auto role, 10G advert) / 0x0000 (no EEE advertised) |
|
||||
| A2.250 (Table 3) | PHY part number ([10gbase-t-phy-register-operation.md](10gbase-t-phy-register-operation.md)) | 9 on the CUX3610 (1 is documented as 88X3310) |
|
||||
|
||||
Proven-safe writes, all IEEE standard space: AN restart `7.0 |= 0x1200`, and PMA reset `1.0 = 0x8000` — **the PMA reset clears the speed-selection bits** (1.0 reads 0x0000 after, no speed selected, 10G training impossible) and does not touch 7.x config; write 1.0 = 0x2040 back after any reset or the link never returns.
|
||||
|
||||
`rb_forensic.py` in phydiag-work performs this baseline and logs every sff_i2c op (PRE line fsync'd before each op, POST after) to `wiitek-ops.log` — a PRE with no POST names whatever killed the µC. `rollball_ixgbe.py` hard-guards the `*.0x??64` blacklist and raises before touching hardware.
|
||||
|
||||
## In the test set — mixed pair with the FS
|
||||
|
||||
cabletest drives this module through its own Go RollBall client (`phy.go`): module type dispatched from the EEPROM vendor PN before any protocol touch, mailbox ops as split ≤4-byte offset writes plus single-hold compound reads over the patched-ixgbe `sff_i2c`, and a **whitelist guard** — every MDIO read/write is checked against the proven-safe set above and panics before touching hardware on anything else, so the client structurally cannot repeat the kill. The 1 Hz poller reads link (1.1), per-pair IEEE margins (1.133–136, used directly — no operating-point subtraction), corrected latches (3.33) and fast-retrain count (1.147); margins ride ghost bounds of −10…+25 dB. **This module's IEEE margins are the panel's only SNR source** — the FS's handler SNR is not polled at all (the derived number was never actionable and each GET costs BCM handler traffic), so an all-BCM pair shows no SNR. Roles in the mixed pair: the FS is forced manual-slave and this module auto-resolves master (its 7.32 reads auto and cannot be safely forced) — links in ~3 s. EEE stays down one-sided (this unit advertises none; the FS forces none). Module-level jumbo cannot be forced here; the running mix's 9018-byte frames pass at line rate regardless. Length comes from the FS end's ECD with this module as the far end: healthy per-pair verdicts and lengths within a few meters of the FS↔FS baseline on the same cable.
|
||||
|
||||
Not available in safe space: pre-FEC counter, cable length — those live in the µC vendor space that killed the modules.
|
||||
|
||||
## VCT/cable length: no confirmed-safe path
|
||||
|
||||
Reference in New Issue
Block a user