ai/byre
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Show agent liveness, stuck/zombie detection and parse errors in admin

Browse Source
This commit is contained in:
Ian Gulliver
2026-10-14 22:35:53 -07:00
parent 33e682a9d5
commit 1dc09c77e0
21 changed files with 1072 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations.md
+9 -2
View File
@@ -81,9 +81,16 @@ When Claude Code reports a rate-limit status of `rejected` and the turn then fai
## Admin view
`GET /api/admin/status` reports `running_turns` (sessions now `running`) and `queued_turns` (rows of `turn_queue`, in the order they will start), each as session, task and vendor; `caps`, the global and per-vendor limits from `DefaultLimits`; `rate_limits`, the latest `rate_limit_info` each vendor sent since byre started, unchanged; and server counts (start time, agent processes, projects, active tasks, SSE clients, database size including WAL, last event seq). A turn's queue row is written before its session's `queued` state event, so a refetch triggered by that event sees it.
`GET /api/admin/status` reports `running_turns` (sessions now `running`) and `queued_turns` (rows of `turn_queue`, in the order they will start), each as session, task and vendor; `caps`, the global and per-vendor limits from `DefaultLimits`; `rate_limits`, the latest `rate_limit_info` each vendor sent since byre started, unchanged; `agent_processes`, `zombies`, `stuck_after_seconds` and `protocol_errors` (below); and server counts (start time, agent processes, projects, active tasks, SSE clients, database size including WAL, last event seq). A turn's queue row is written before its session's `queued` state event, so a refetch triggered by that event sees it.
The Admin view (a pane tab, also opened in the right pane by the navbar's **Admin** button) shows all of it: running turns against each cap, queued turns in order, both linking to their tasks, each vendor's rate-limit status with its reset time and per-window utilization, and the server counts. It refetches the status when a `session.created`, `session.state`, `task.*` or `project.created` event arrives, when an `agent.vendor` event carries a `rate_limit_event`, and when the SSE stream reconnects; other events only advance the last-event seq it shows. There is no timer, so the agent-process count can lag until one of those events, and the "in 2h 10m" beside a reset time is as of the last render.
Agent health is `internal/agents/health.go`:
- **Processes.** `agent_processes` lists every process byre is tracking (idle sessions keep theirs), with session, task, vendor, pid, the session's state, `last_event_at` (when the process last wrote any line to stdout, or when it started; `claude.Session.LastLine`) and `health`. `health` is `exited` when the stream has closed or `kill(pid, 0)` says the pid is gone, `stuck` when the session is `running` and nothing has arrived for `StuckAfter`, otherwise `ok`. The stream does mark long tool calls: while a tool runs, Claude Code sends a `tool_progress` line with `"heartbeat": true` every 30 seconds, so a long `go test` keeps `last_event_at` fresh and a silent stream means nothing is happening.
- **Zombies.** `running_without_process`: a session marked `running` whose process byre isn't tracking or has `exited`. `process_without_session`: a `claude` child of the byre process (from `ps -A -o pid=,ppid=,comm=`) whose pid belongs to no tracked process, or a tracked process still alive for a `completed` session. A live process for an `idle` session is normal and not a zombie.
- **Watching.** `Manager.Watch` recomputes health every `healthInterval`. When the set of problems (non-`ok` processes and zombies) changes, it logs each new one as `agent health problem` and broadcasts an unstored `admin.health` event (`seq` 0). Nothing is killed or repaired automatically.
- **Protocol errors.** `internal/claude` reports a stdout line that isn't a JSON object, doesn't parse, or has a `type` outside `knownTypes` (the line is still passed on), and the manager adds lines whose content it fails to decode. `protocol_errors.counts` counts them per vendor since byre started; `recent` holds the last `keptBadLines`, newest first, each with session, time, reason, the raw line cut to `badLineBytes` and its full `size`. They live in memory only, and each one also broadcasts `admin.health`.
The Admin view (a pane tab, also opened in the right pane by the navbar's **Admin** button) shows, from the top: a Problems section only when there are zombies or protocol errors (one line per zombie, and per vendor the error count with the latest line folded under it); running turns against each cap, each running turn with the time since its process's last event and `stuck` or `exited` when it is; queued turns in order, both linking to their tasks; each vendor's rate-limit status with its reset time and per-window utilization, and the server counts. It refetches the status when a `session.created`, `session.state`, `task.*`, `project.created` or `admin.health` event arrives, when an `agent.vendor` event carries a `rate_limit_event`, and when the SSE stream reconnects. Other events advance the last-event seq it shows, and any event carrying a process's `agent_session_id` (stored or not, such as `agent.delta`) moves that process's last-event time forward, re-rendering at most once a second per process. A re-render keeps the scroll position and which folded lines are open. There is no timer, so "time since last event" and the "in 2h 10m" beside a reset time are as of the last render; a process going quiet long enough to be stuck shows up when the next health check broadcasts `admin.health`.
Agent activity becomes events with `agent_session_id` set: `user.message`, `agent.message`, `agent.thinking`, `agent.tool_call`, `agent.tool_result`, `turn.ended`, `session.state`, `session.model`, and `session.context`. Anything else Claude Code emits (rate-limit status, task progress, state changes) is recorded unchanged as `agent.vendor`. The one exception is partial output: each top-level `text_delta` is broadcast to SSE clients as an `agent.delta` event (`payload.text`, `seq` 0) that is never stored, and the transcript grows a live message from them until the finished `agent.message` replaces it. A `turn.ended` event carries `inbox_item_id` when its final message was filed as a `report` item. `session.context` records the context size whenever it changes: `used` is the last assistant message's input, cache-read, cache-creation and output tokens, and `window` is the model's context window from the turn's `result` (`modelUsage[model].contextWindow`), so it is 0 until the session's first turn after a server start ends. The task header shows the latest one as a percentage of the window, coloured green, amber from 70%, red from 90%.