FS SFP-10G-T-100 (Broadcom BCM84891L)
Documented, robust — survived the whole poking campaign un-bricked. Honest EEPROM (RJ45/10GBASE-T codes), which is what trips stock ixgbe qualification (../README.md).
References in this directory: bcm84891l-mdio-commands.md (command handler, transcribed), bcm84891l-sfp-registers.md (transport reference, transcribed), and their source PDFs.
SMI transport (I2C→MDIO bridge)
PHY at I2C 8-bit 0xAC write / 0xAD read.
| Op | Frames |
|---|---|
| Clause-45 write | one I2C write to 0xAC: [000+DevAD, RegH, RegL, DataH, DataL] |
| Clause-45 read | I2C write to 0xAC: [001+DevAD, RegH, RegL] → delay >1 ms (3 ms proven) → I2C read 2 B from 0xAD |
- Single-byte reads see the bridge as inert (returns 0) — the DevAD-prefixed frame + delay is mandatory. This is why early probing wrongly declared 0x56 dead. Treat 0x0000 reads as retry-with-longer-delay: 0 is also the bridge's not-ready signature.
- Recorded for a future multi-byte firmware transport (mlx5 MCIA shape): read = write
[001+devad,RegH,RegL]offset_size=1, STOP, >1 ms, read 2 B offset_size=0; write = offset_size=2, addr = devad<<8|RegH, data =[RegL,DataH,DataL].
MDIO command handler
Protocol and full verified catalog: bcm84891l-mdio-commands.md. Traps:
- SET commands execute stale DATA — the handler never clears DATA registers; write every parameter register explicitly before any SET.
- GETs must be invoked bare — pre-writing any DATA register, not just the documented DATA1 display flag, leaves the handler executing as a no-op with results never written (sentinel pre-fills survived GET_SNR untouched, proven on hardware).
- STATUS must never be written — any user value (0x0000 and 0x0008 both tried) closes the mailbox: the next command is silently ignored until firmware restores it.
- PASS cannot prove completion. The previous command's PASS stays latched while a slow firmware still executes, and until then the DATA registers are firmware scratch — the die temperature turns up in them (0x004E–0x0051 observed as "SNR" and "EEE mode"; the campaign's "anomalous 0x0047" was this same race).
- Handler writes during firmware-busy windows wedge the µC permanently — DATA/CMD writes landing while firmware does post-AN provisioning or training work (a window stretching ~10 s past relink) killed the SMI service four times; reads alone never once. The windowed protocol keeps steady-state writes clear of the temp poll's busy windows; bringup-era writes run unwindowed (they fire pre-AN, where writes have never wedged) and post-AN provisioning exposure remains as the diag path's accepted risk.
- Bridge reads carry no fetch identity — a fetch that outruns the delay leaves the previous transaction's data at 0xAD with no error. A killed host can also leave the pipeline latched a response behind persistently (PHY ID served a stuck stale value across processes until driver re-init); every appliance boot clears it by loading the driver, so only bench runs reusing a loaded driver see it, and the identity check at startup dies loudly on it.
- Stale reads poisoned the corrected-error channel — caught with per-register instrumentation: 3.33 reads served 1.147's resident 0x0011 (decoding as exactly 17 errored blocks — the recurring "+17"), SNR DATA values (+21/22), and 1.147 reads served 3.33's 0x8000 (count 16, charged twice by the rollover delta) and the die temperature. Every such burst decodes as a neighbor register. The source is the firmware's own temp poll and the fix is the windowed protocol (firmware-reliability notes): work confined to a fixed window after each observed poll ends the poisoning — reproduced at will by aiming reads at the poll, eliminated by avoiding it.
- Poll STATUS ~100 ms; frozen up to 2 s during 10GBASE-T training — only run after link-up.
Proven diagnostics
| Capability | How | Observed |
|---|---|---|
| Per-pair SNR | CMD_GET_SNR 0x8030, invoked bare (writing the documented DATA1 display flag returns zeros; IEEE 1.133–1.140 never populate — constant 0x8080) |
DATA2–5 = SNR A–D ×0.1 dB; ≈ 27–32 dB absolute on bench, 0.1–0.4 dB jitter. 10GBASE-T operating point ≈ 26.5 dB, so margin ≈ value − 26.5 — a derived number that never proved actionable, so cabletest does not poll it (the panel's SNR comes from an IEEE-margin end — modules/wiitek/ — and each GET costs handler traffic on this µC). Each training re-converges differently: back-to-back retrains with identical roles, temperature and noise moved one pair by 3.6 dB (roles pinned, launch power pinned — the convergence itself is the variable, and the two bench modules differ in stability: ≤1.5 dB spread on one, 3.6 dB on the other). A single training's absolute SNR is a weak qualification number; qualification wants median-of-N trainings |
| Die temperature | CMD_GET_CURRENT_TEMP 0x8031 |
~68–70 °C on bench |
| Supply rails | GET_CURRENT_VOLTAGE 0x802F |
0.8 V and 1.88 V rails, tenths of mV |
| Error counters | IEEE PCS 3.32/3.33 — block lock, latched errored-block/BER, clear-on-read | The noise-stress error proxy |
| Retrain on demand | IEEE AN restart 7.0 |= 0x1200 (enable + restart — the ECD can leave the enable bit cleared, see below) |
Relinks in ~5–6 s |
Probed configuration states
| Item | Command | State |
|---|---|---|
| EEE / AutogrEEEn | 0x8008/0x8009 | The famous 0x0047 GET reading was the die temperature in DATA1 (firmware scratch — see reliability notes), not an AutogrEEEn mode; GET 0x8008 is unusable. cabletest forces all-off every boot (SET with explicit params (0, 0, 0x7A12, 0x480, 0) + AN restart, proven). The SET programs the advert register directly, so 7.60 reads 0 immediately — bringup verifies it before any AN restart, both µCs quiet; the modules link to each other, so one restart puts both into training, and bringup configures and verifies both before either restart fires (phy.go) |
| EEE wire-truth | arm 0x801A after link-up, read 0x801B | Zero LPI events/duration on idle link; repeat under traffic |
| Fast retrain | 0x800A (datasheet titles it EMI_MODE; description is fast retrain) | Enabled 10G/5G/2.5G; IEEE 1.147 = 0x0019, count bits zero. Keep enabled; read the 1.147 count per run — a marginal cable that fast-retrains still gets counted |
| Pair map | 0x8000 | DATA2 = 0x00E4 = identity (A/B/C/D straight through) — MDI wiring verification works |
| Fault pass-through | 0x800D | 0 = LF/RF pass to the line (0x8011/0x8010 could mask them as IDLE — leave off) |
| Host interface | 0x802B/0x802D/0x8027 | XFI polarity normal; TX FIR = main tap 0x3C only; MAC/PHY frequency locked; KR/SyncE/USXGMII/WOL disabled; temp warnings off |
| Jumbo | 0x801C/0x801D | GET 0x801D answers in DATA1 — scratch-prone, unusable — and bringup cannot assume a cable to probe through, so cabletest forces enable every boot (SET 0x801C + AN restart); the running mix's 9018-byte frames are the standing wire truth (phy.go). The FS at its 9K option passes them at line rate with zero loss/corrupt |
| 1588 | 0x8004 | Disabled (engine registers undocumented — see asks below) |
| Limited reach | 0x8006 | Disabled — its DATA3 "linked in LR mode" bit, the only handler-visible trace of the DSP length estimate, is unavailable without a config change |
| Master/slave role | IEEE 7.32 bits 15/14 | Left to AN the resolution is a per-training lottery; manual config writes stick exactly and the resolution follows deterministically (7.33.14, no config fault with complementary ends). cabletest forces A master / B slave every boot when both ends are BCM; with a non-BCM partner (whose manual config is unreachable) the BCM is forced slave and the partner auto-resolves master — proven linking in ~3 s against the Wiitek (phy.go) |
| Launch power | IEEE 1.130, 1.0.11 | 1.130 reads 0x0003 and is firmware-pinned: host writes to the short-reach bit revert instantly and the value never changes across retrains, so launch power is constant and not host-influenceable. PMA low-power bit clear. The handler catalog has no TX-power command |
Firmware-reliability notes
The firmware is its own mailbox client — the handler is a shared, unarbitrated resource. Watched read-only on an idle module: CMD sits at 0x0031 (CMD_GET_CURRENT_TEMP, bit 15 consumed) and DATA1 tracks the live die temperature (0x4F–0x52 ≈ 79–82 °C under load) — the firmware issues its own temperature command through the same CMD/STATUS/DATA registers the host uses, with no arbitration. Every mystery this explains: temperature appearing in DATA1 (it is that command's output — the campaign's "anomalous 0x0047" included), PASS satisfying a host poll when it belongs to the internal command, and the µC wedges (two writers colliding on CMD/DATA, likeliest when internal management activity spikes after AN events). Any GET whose answer lives in DATA1 is unusable; corroborate through IEEE registers or the wire. The hard rule that stands: write every DATA register explicitly before any SET — the handler executes stale DATA. The internal poll is not silenceable through the handler: disabling the temperature warning (SET 0x8032 (0x007D, 5, 0, 0, 0) and 0x8034 (0xFFD8, 5, 0, 0, 0) — DATA3 = 0 = warning off — both PASS) leaves CMD resident at 0x0031 with DATA1 tracking the die within seconds of the host going quiet; the poll is independent of the warning machinery. GET 0x8033/0x8035 read DATA1–3 = 0 before and after those SETs (DATA4/5 leftover scratch) — DATA1-answer readbacks, unusable as verification either way.
The internal poll's clock, measured on hardware. One poll every 3.49–4.22 s, regime-dependent but rock-stable within a regime (±15 ms): ≈3.50 s warm and idle, ≈3.70 s, ≈4.01 s cool and idle, ≈4.21 s under continuous host mailbox load — host traffic defers the poll, never advances it, and nothing observed brings two polls closer than 3.49 s. The poll is link-independent: it runs straight through an AN restart and the whole retrain without missing a beat. Around each poll's service the bridge serves stale data for ~50 ms: every read returns the last value successfully fetched before the µC went busy (stuck-at-last-fetch, not one-behind), which is exactly how 3.33 inherits 1.147's 0x0011. The window closes before the poll is detectable — a CMD read returning the fresh resident 0x0031 is itself proof the µC serviced the fetch, so flip detection doubles as the all-clear.
The windowed protocol (phy.go) — contention resolved by time-division. Since the poll cannot be silenced, the host schedules around it. One goroutine per module owns every transport touch (requests execute one at a time on the loop — nothing else can reach the wire, by construction), and each request is admitted only inside a fixed 3.4 s window following an observed poll (CMD flipping to resident 0x0031, polled at 10 ms); at the cutoff the host goes quiet until the next flip. 3.4 s sits below every observed cadence, so a regime switch mid-run lands harmlessly in vacated time — no prediction, no period tracking, just the last observed flip plus two constants. Re-lock — no window at assumed phase: edge detection needs CMD ≠ 0x0031, and handler commands leave it armed (every one rewrites CMD); whenever CMD sits at the resident 0x0031 at expiry the phase is unknown and admission arms first, then anchors on the true edge that follows. Steady-state polling issues no handler commands of its own (the SNR GET is dropped), so each taken edge immediately re-arms with a bare GET_PAIR_SWAP: the single CMD write per window lands at the start of the quiet period, maximally far from the next poll — the wedge risk is writes near the poll — and every later expiry reads the phase without writing. One command per ~4 s window replaces a handler GET per second. Admission then arms first (a bare GET_PAIR_SWAP, resident 0x0000) and anchors only on the true edge that follows, so every window is edge-locked from first contact at boot; bringup pays one phase acquisition per module (~2–4 s). A missing heartbeat free-runs one window rather than stall (never observed — the heartbeat survives retrains). Before re-lock the post-diag window anchored blind on the stale resident, and every ghost observed under windowing sat in exactly that window; with re-lock, ghost-free across all bench diags to date. A poisoned batch is self-consistent and forensically visible: the stuck value clears the link bit (SNR skipped, panel shows "-", the ghost-SNR panic sidestepped) while 3.33 charges the same value as errored blocks (0x0011 → the "+17"); every corrected charge and link-down reading prints its raw 1.1/3.33/1.147 on the console, so events self-attribute — a ghost names its neighbor register, real correction decodes as nothing but itself. Validated on hardware in ~/work/phydiag-work/bcm_phaselock_bench.py: aiming read bursts at the poll reproduces the poisoning on 100% of polls (the "+17" manufactured on demand); windowed operation ran 2,881 back-to-back batches — 16× the production rate, riding the cutoff — with zero stale values, and the production 1 Hz stream is unperturbed (blackout hides inside natural gaps, worst sample gap ~1.6 s).
ECD — recovered from the OpenBCM SDK, proven on hardware
The ECD register mechanism is absent from the handler catalog and the datasheet excerpts here, but the OpenBCM SDK's copper-XGPHY driver (sdk-6.5.27/src/soc/phy/phy8481.c phy_8481_cable_diag + phy8481.h) carries it for the 8483x/8485x/8488x family — and the same SDK drives the identical command-handler registers (1E.0x4005/0x4037/0x4038–3C) as the BCM84891L datasheet, confirming the shared map. Validated on the FS:
| Register | Role |
|---|---|
1E.0x4006 |
Control/status. Write under mask {15,14,13,12,10}: bit 15 = run now, bit 14 = run at AN, bit 12 = break link, bit 10 = length in meters (SDK writes value 0x8400 = run now + meters). Bit 11 = busy — poll until clear (SDK allows up to 50 s; observed < 0.5 s) |
1.0xA896 |
Verdicts, 4 bits per pair: 1 = OK, 2 = open, 3 = short, 4 = inter-pair short |
1.0xA897–0xA89A |
Per-pair lengths (pairs 1/2, 3/4, 5/6, 7/8), meters |
Observed on a plugged, linked, healthy cable: verdicts OK×4 and per-pair lengths of [45, 45, 41, 46] on a ~45 m cable — meter-accurate with no calibration, and this ECD reports length for healthy pairs, not just faults, resolving the terminated-far-end concern in ../README.md. Caveats:
- The run blips the link (PMA 1.1 latch-low catches a drop even with the break-link bit clear) — do not run mid-measurement until the disturbance is characterized.
- The run can leave clause-45 AN disabled (7.0.12 cleared — observed live after diags racing an in-progress retrain): with AN off the BCM emits no AN pulses, both ends report no link partner (7.1 LP-AN-able 0), and the link stays down through AN restarts, role changes and PMA resets until the bit is re-enabled. cabletest restarts AN with
7.0 |= 0x1200(enable + restart) after every diag, anddevRestartANalways forces the enable bit — a bare bit-9 restart preserves the cleared enable forever. - An interrupted run can wedge the µC's SMI service (observed after a process died mid-diag): the bridge ACKs I2C but serves 0x0000 for every register at any delay and silently drops MDIO writes — no in-band recovery, not even PMA reset — while the EEPROM path and the trained link keep working. Recovery: a driver unbind/rebind of the port (the SFP re-initialization resets the µC); a reseat would do the same.
- Fault verdicts (open/short/inter-pair) are unexercised — deliberately: the product is a closed-loop tester, both ends always plugged.
- Family constraints from the SDK: port must be enabled; unsupported at forced 100M.
bcm_ecd_probe.pyin phydiag-work implements the recipe.
Missing datasheet chapters
Chapters the datasheet TOC lists but the excerpts here omit — to source elsewhere:
- The ECD chapter — now for confirming bit meanings rather than unblocking.
- The 1588 operation chapter.
- Datasheet §1.20 loopback (copper line loopback) and §1.17 EEE/fast-retrain monitoring.
- Chapter 2 register summary. The excerpt's TOC names them all.