Show agent liveness, stuck/zombie detection and parse errors in admin
This commit is contained in:
@@ -29,6 +29,8 @@ NDJSON both ways: one JSON object per line on stdin and stdout. Non-JSON stdout
|
||||
- Stdout events: `system` (subtype `init` carries `session_id`, `model`, tools, MCP server status), `assistant` and `user` (Anthropic message content blocks: `text`, `tool_use`, `tool_result`), `result` (ends a turn; `subtype` is `success` or `error_during_execution` after an interrupt, with cost, duration, turn count).
|
||||
- `--include-partial-messages` adds `stream_event` lines wrapping the raw API stream (`message_start`, `content_block_start`, `content_block_delta` with `text_delta`, `content_block_stop`, `message_delta`, `message_stop`), each with `parent_tool_use_id` (set inside subagents). The complete `assistant` event for a block still arrives, before that block's `content_block_stop`. Transcripts leave `stream_event` lines out.
|
||||
- The `init` event arrives with the first turn, not at launch, so the session ID is unknown until the first message is sent.
|
||||
- While a tool runs, `tool_progress` lines with `"heartbeat": true`, the tool's `elapsed_time_seconds`, and a `parent_tool_use_id` naming the tool call arrive every 30 seconds. A Bash call also brings `system` `task_started` and `task_notification` lines around it.
|
||||
- Lines that aren't a JSON object, don't parse, or have a `type` not in `knownTypes` (`internal/claude/session.go`) go to `Options.OnBadLine`; lines that parse are still delivered as events whatever their type.
|
||||
|
||||
## Control channel
|
||||
|
||||
|
||||
+9
-2
@@ -81,9 +81,16 @@ When Claude Code reports a rate-limit status of `rejected` and the turn then fai
|
||||
|
||||
## Admin view
|
||||
|
||||
`GET /api/admin/status` reports `running_turns` (sessions now `running`) and `queued_turns` (rows of `turn_queue`, in the order they will start), each as session, task and vendor; `caps`, the global and per-vendor limits from `DefaultLimits`; `rate_limits`, the latest `rate_limit_info` each vendor sent since byre started, unchanged; and server counts (start time, agent processes, projects, active tasks, SSE clients, database size including WAL, last event seq). A turn's queue row is written before its session's `queued` state event, so a refetch triggered by that event sees it.
|
||||
`GET /api/admin/status` reports `running_turns` (sessions now `running`) and `queued_turns` (rows of `turn_queue`, in the order they will start), each as session, task and vendor; `caps`, the global and per-vendor limits from `DefaultLimits`; `rate_limits`, the latest `rate_limit_info` each vendor sent since byre started, unchanged; `agent_processes`, `zombies`, `stuck_after_seconds` and `protocol_errors` (below); and server counts (start time, agent processes, projects, active tasks, SSE clients, database size including WAL, last event seq). A turn's queue row is written before its session's `queued` state event, so a refetch triggered by that event sees it.
|
||||
|
||||
The Admin view (a pane tab, also opened in the right pane by the navbar's **Admin** button) shows all of it: running turns against each cap, queued turns in order, both linking to their tasks, each vendor's rate-limit status with its reset time and per-window utilization, and the server counts. It refetches the status when a `session.created`, `session.state`, `task.*` or `project.created` event arrives, when an `agent.vendor` event carries a `rate_limit_event`, and when the SSE stream reconnects; other events only advance the last-event seq it shows. There is no timer, so the agent-process count can lag until one of those events, and the "in 2h 10m" beside a reset time is as of the last render.
|
||||
Agent health is `internal/agents/health.go`:
|
||||
|
||||
- **Processes.** `agent_processes` lists every process byre is tracking (idle sessions keep theirs), with session, task, vendor, pid, the session's state, `last_event_at` (when the process last wrote any line to stdout, or when it started; `claude.Session.LastLine`) and `health`. `health` is `exited` when the stream has closed or `kill(pid, 0)` says the pid is gone, `stuck` when the session is `running` and nothing has arrived for `StuckAfter`, otherwise `ok`. The stream does mark long tool calls: while a tool runs, Claude Code sends a `tool_progress` line with `"heartbeat": true` every 30 seconds, so a long `go test` keeps `last_event_at` fresh and a silent stream means nothing is happening.
|
||||
- **Zombies.** `running_without_process`: a session marked `running` whose process byre isn't tracking or has `exited`. `process_without_session`: a `claude` child of the byre process (from `ps -A -o pid=,ppid=,comm=`) whose pid belongs to no tracked process, or a tracked process still alive for a `completed` session. A live process for an `idle` session is normal and not a zombie.
|
||||
- **Watching.** `Manager.Watch` recomputes health every `healthInterval`. When the set of problems (non-`ok` processes and zombies) changes, it logs each new one as `agent health problem` and broadcasts an unstored `admin.health` event (`seq` 0). Nothing is killed or repaired automatically.
|
||||
- **Protocol errors.** `internal/claude` reports a stdout line that isn't a JSON object, doesn't parse, or has a `type` outside `knownTypes` (the line is still passed on), and the manager adds lines whose content it fails to decode. `protocol_errors.counts` counts them per vendor since byre started; `recent` holds the last `keptBadLines`, newest first, each with session, time, reason, the raw line cut to `badLineBytes` and its full `size`. They live in memory only, and each one also broadcasts `admin.health`.
|
||||
|
||||
The Admin view (a pane tab, also opened in the right pane by the navbar's **Admin** button) shows, from the top: a Problems section only when there are zombies or protocol errors (one line per zombie, and per vendor the error count with the latest line folded under it); running turns against each cap, each running turn with the time since its process's last event and `stuck` or `exited` when it is; queued turns in order, both linking to their tasks; each vendor's rate-limit status with its reset time and per-window utilization, and the server counts. It refetches the status when a `session.created`, `session.state`, `task.*`, `project.created` or `admin.health` event arrives, when an `agent.vendor` event carries a `rate_limit_event`, and when the SSE stream reconnects. Other events advance the last-event seq it shows, and any event carrying a process's `agent_session_id` (stored or not, such as `agent.delta`) moves that process's last-event time forward, re-rendering at most once a second per process. A re-render keeps the scroll position and which folded lines are open. There is no timer, so "time since last event" and the "in 2h 10m" beside a reset time are as of the last render; a process going quiet long enough to be stuck shows up when the next health check broadcasts `admin.health`.
|
||||
|
||||
Agent activity becomes events with `agent_session_id` set: `user.message`, `agent.message`, `agent.thinking`, `agent.tool_call`, `agent.tool_result`, `turn.ended`, `session.state`, `session.model`, and `session.context`. Anything else Claude Code emits (rate-limit status, task progress, state changes) is recorded unchanged as `agent.vendor`. The one exception is partial output: each top-level `text_delta` is broadcast to SSE clients as an `agent.delta` event (`payload.text`, `seq` 0) that is never stored, and the transcript grows a live message from them until the finished `agent.message` replaces it. A `turn.ended` event carries `inbox_item_id` when its final message was filed as a `report` item. `session.context` records the context size whenever it changes: `used` is the last assistant message's input, cache-read, cache-creation and output tokens, and `window` is the model's context window from the turn's `result` (`modelUsage[model].contextWindow`), so it is 0 until the session's first turn after a server start ends. The task header shows the latest one as a percentage of the window, coloured green, amber from 70%, red from 90%.
|
||||
|
||||
|
||||
+1
-1
@@ -475,7 +475,7 @@ Record transcripts from the spike as the seed of the contract test suite.
|
||||
### Milestones
|
||||
|
||||
- **M0 — Skeleton.** launchd service, tailscale serve auth, SQLite schema, OpenAPI spec, projects, tasks, worktree create/delete, event log, SSE stream with resume, embedded SPA shell, basic server observability.
|
||||
- **Status: done**, and running as a service on the Mac Studio. Observability covers queue state and vendor rate-limit status, through `GET /api/admin/status` and the SPA's Admin view; process liveness and stuck-process detection, adapter parse errors, and per-task snapshot counts and disk usage are not done.
|
||||
- **Status: done**, and running as a service on the Mac Studio. Observability covers queue state, vendor rate-limit status, agent process liveness, stuck and zombie detection, and adapter and protocol-parse errors, through `GET /api/admin/status` and the SPA's Admin view; per-task snapshot counts and disk usage are not done.
|
||||
- **M1 — One agent end to end (Claude).** Sessions, streaming, messaging, the interface coverage for Claude, the `ask_human` MCP tool, turn summaries, inbox with all views and response types, web push notifications, restart-and-resume, concurrency caps.
|
||||
- **Done:** sessions, streaming, messaging, interrupt; `ask_human` with validation; each turn's final message filed to the inbox as its summary; inbox with global, project, and task views and every response type, answered as one batch and delivered to the agent as one message; restart-and-resume; global and per-vendor caps with queueing; rate limits as their own inbox state; the web UI (two panes, task switcher, transcript, inbox) with Markdown rendering; the session instructions from "Session configuration", given as an appended system prompt and a per-turn reminder; per-turn commits by byre; bouncing questions asked in plain prose; web push notifications through a service worker, with per-user preferences (blocking questions by default, other kinds opt-in) set in a Settings tab, bursts collapsed into one notification, and a click opening the task. Compaction is done, as a handoff into a fresh session; model switching is done, per message; image input is done, pasted or dropped into the composer and stored as blobs.
|
||||
- **Later:** subagent views and Mermaid rendering.
|
||||
|
||||
Reference in New Issue
Block a user