Run workflow stages with finish_stage handoffs and lead signals

docs/operations.md

@@ -100,7 +100,7 @@ Agents spawn parallel work with the `create_task` MCP tool (`internal/agents/tas
 
 Agents don't commit; byre does. Each `user.message` event that starts a turn records `head`, the worktree's `HEAD` when the turn started. When a turn ends, byre runs `git add --all` and commits if anything changed (`git.CommitAll`), as the session's creator, with the first sentence of the turn's final message as the message (`commitMessage` in `internal/agents/manager.go`); `turn.ended` then carries the new sha as `commit`. A failed commit is logged and the turn still ends.
 
-A session runs on one of two vendors, picked by the model it is created with (`vendorOf` in `internal/agents/process.go`): `gpt-*` models run on Codex (`codex app-server`), every other model on Claude Code. The UI's model list is `models` in `web/src/task.ts`; a live session's picker offers only its own vendor's models, and the API refuses (400) a switch to the other vendor's model. Both vendors sit behind `agentProcess` (`internal/agents/process.go`), and both end a turn through the same `turnEnded`, so commits, turn summaries, snapshots, handoffs, merges and deliveries don't depend on the vendor.
+A session runs on one of two vendors, picked by the model it is created with (`workflow.VendorOf` in `internal/workflow/workflow.go`): `gpt-*` models run on Codex (`codex app-server`), every other model on Claude Code. The UI's model list is `models` in `web/src/task.ts`; a live session's picker offers only its own vendor's models, and the API refuses (400) a switch to the other vendor's model. Both vendors sit behind `agentProcess` (`internal/agents/process.go`), and both end a turn through the same `turnEnded`, so commits, turn summaries, snapshots, handoffs, merges and deliveries don't depend on the vendor.
 
 For Claude, byre chooses the session ID (`vendor_session_id`) up front. Codex picks its own thread id: a Codex session is created with an empty `vendor_session_id`, and byre stores the thread id when its first process starts (`startCodex`); a session with an id resumes it with `thread/resume`. Processes start on demand: the first message after a restart resumes the session (`--resume` for Claude). Sessions are `running` from the moment a turn is sent until it ends (Claude's `result`, Codex's `turn/completed`); on startup, every session still marked `running` was cut off mid-turn, and byre resumes it and sends a message telling the agent to re-check the working tree. Stopping the server kills all agent processes. Merging or discarding a task kills its sessions and marks them `completed`.
 
@@ -145,8 +145,8 @@ Each user has one long-running lead agent, shown in the left pane. It is not tie
 
 - **Sessions.** A lead session is an `agent_sessions` row with `owner` set to the user's login, `role` `lead` and no `task_id`. Its events carry `owner` and no project or task. It runs in `leads/<login>` with `leadGuidance` (passed and re-attached the same way as a task agent's guidance) and the lead tools instead of the task tools. It has no Stop-hook bounce, no snapshots, no end-of-turn commit and no turn summary; a turn that ends in error records the error on its `session.state` event, which the pane shows as a note.
 - **Messages.** `POST /api/me/lead/messages` sends the caller's message to their lead, starting a lead session when there is none. While the lead is mid-turn the message is held in `lead_inputs`; the lead's PostToolUse hook (Codex: `turn/steer` after each completed tool item) hands held messages over after its next tool call, like a task's queued messages (see "Agent sessions"), and any still held become its next turn. Signals are never handed over mid-turn. The composer stays enabled. `GET /api/me/lead/sessions` lists the caller's lead sessions; interrupt and compact use the usual session endpoints.
-- **Signals.** Every published event goes through `Manager.Observe`. These become signals in `lead_inputs` for every user who has a lead session: `inbox.created` (a question with its choices and recommendation, an error or rate limit), `inbox.updated` when a human answered a question or error, `task.created` (naming the TODO item it was started from, if any), `task.merged`, `task.discarded`, and `turn.ended` with `summary: true` ("<task> finished a turn:" with its `result` cut to 1500 characters). The human switching the right pane is not a signal. A signal caused by the lead itself (actor `<login> (lead)`) is skipped, as is the update from an answer it gave. A task's `task.created` event carries the actor that created it, separately from its `created_by` login: tasks the lead creates with `create_task` or `start_todo` keep the human's login as `created_by` but have actor `<login> (lead)`. Whenever the lead is idle, byre sends the oldest held message from the human as its own turn, or else every held signal as one `byre` message headed "Signals since your last turn (n)". Its guidance tells it not to respond to the human's own actions, and to make its final message exactly `NO_REPLY` when a batch needs nothing from it. A transcript (`quietReply` in `web/src/task.ts`) never draws an `agent.message` whose last line is `NO_REPLY` (the model sometimes writes a sentence before it), and hides the live partial message while its text is still a prefix of it; the partial message starts hidden and is first shown by its 100ms render, so an empty one never takes up space. A lead turn that byre starts without a handoff (a signal batch or a handoff continuation) is held (`ctx.held`): its `byre` message isn't drawn and doesn't open the turn, so the spinner stays off, and its partial message stays hidden, until the lead draws something else, such as a tool call, thinking or a real message, which draws the `byre` message ahead of it. When such a turn ends in `NO_REPLY`, everything it drew is removed with it (its `byre` message, tool calls, thinking), so the transcript keeps nothing of it; a turn that ends in a real message keeps all of it.
-- **Tools.** `list_tasks` (each project's active tasks, then a line listing the titles of its `TODO.md` items on the default branch that no active task was started from), `read_transcript` (a task's events in brief, the last `limit` or those after `after_seq`), `read_inbox`, `answer_question` (an `option`, `other` or `dismiss` response recorded as the user, through the same checks as `POST /api/inbox/responses`), `send_message` (starts the task's agent if it has none; refused while it is mid-turn), `create_task` and `start_todo` (each with an optional `workflow` to freeze), `interrupt`, `compact_task`, `merge_task`, `discard_task`, `show`, `compact_self`, and the workflow tools (see "Workflows"). The lead also has Claude Code's own tools and reads worktrees by path. Messages it sends task agents have actor `<login> (lead)`, so they don't make the user a participant; tasks, merges, discards and answers are done as the user. Its guidance says to answer a question only when the answer is clear or the human gave it, and to merge or discard only when the human asks.
+- **Signals.** Every published event goes through `Manager.Observe`. These become signals in `lead_inputs` for every user who has a lead session: `inbox.created` (a question with its choices and recommendation, an error or rate limit), `inbox.updated` when a human answered a question or error, `task.created` (naming the TODO item it was started from, if any), `task.merged`, `task.discarded`, `turn.ended` with `summary: true` ("<task> finished a turn:" with its `result` cut to 1500 characters), `task.workflow_stage` ("<task> started stage n/N of "<workflow>": <stage>.") and `task.workflow_finished` ("<task> finished workflow "<workflow>":" with its output cut to 1500 characters). The human switching the right pane is not a signal. A signal caused by the lead itself (actor `<login> (lead)`) is skipped, as is the update from an answer it gave. A task's `task.created` event carries the actor that created it, separately from its `created_by` login: tasks the lead creates with `create_task` or `start_todo` keep the human's login as `created_by` but have actor `<login> (lead)`. Whenever the lead is idle, byre sends the oldest held message from the human as its own turn, or else every held signal as one `byre` message headed "Signals since your last turn (n)". Its guidance tells it not to respond to the human's own actions, and to make its final message exactly `NO_REPLY` when a batch needs nothing from it. A transcript (`quietReply` in `web/src/task.ts`) never draws an `agent.message` whose last line is `NO_REPLY` (the model sometimes writes a sentence before it), and hides the live partial message while its text is still a prefix of it; the partial message starts hidden and is first shown by its 100ms render, so an empty one never takes up space. A lead turn that byre starts without a handoff (a signal batch or a handoff continuation) is held (`ctx.held`): its `byre` message isn't drawn and doesn't open the turn, so the spinner stays off, and its partial message stays hidden, until the lead draws something else, such as a tool call, thinking or a real message, which draws the `byre` message ahead of it. When such a turn ends in `NO_REPLY`, everything it drew is removed with it (its `byre` message, tool calls, thinking), so the transcript keeps nothing of it; a turn that ends in a real message keeps all of it.
+- **Tools.** `list_tasks` (each project's active tasks, then a line listing the titles of its `TODO.md` items on the default branch that no active task was started from), `read_transcript` (a task's events in brief, the last `limit` or those after `after_seq`), `read_inbox`, `answer_question` (an `option`, `other` or `dismiss` response recorded as the user, through the same checks as `POST /api/inbox/responses`), `send_message` (starts the task's agent if it has none; refused while it is mid-turn), `create_task` and `start_todo` (each with an optional `workflow` to freeze and run), `interrupt`, `compact_task`, `merge_task`, `discard_task`, `show`, `compact_self`, and the workflow tools (see "Workflows"). The lead also has Claude Code's own tools and reads worktrees by path. Messages it sends task agents have actor `<login> (lead)`, so they don't make the user a participant; tasks, merges, discards and answers are done as the user. Its guidance says to answer a question only when the answer is clear or the human gave it, and to merge or discard only when the human asks.
 - **Focus.** `users.focus` holds what the user's right pane shows: `project_id`, `task_id`, `tabs` (the tabs put first, in order) and `tab` (the selected one). With a `task_id`, `project_id` is always that task's project, filled in by the server; with only a `project_id`, the pane shows that project with no task, which only the Project tab has anything for. `GET /api/me/focus` reads it; `PUT /api/me/focus` and the lead's `show` set it, refusing an unknown tab, a task that isn't active or an unknown project, and record a `focus.updated` event owned by the user. The page applies the focus on load and on every `focus.updated` it didn't cause itself; the human's own changes (⌘K, the right pane's task picker, clicking a tab, a task link, a notification) are sent with `PUT /api/me/focus`. The URL follows the focus: `#/task/<id>`, `#/project/<id>`, or `#/` for neither.
 - **Pane lock.** A `show` of what the pane already shows (the same task, or the same project with no task, on the same selected tab; `sameView`) tells the lead so and applies only its `tabs` order, whether or not the human is in the pane, so the human's selected tab stays selected. Otherwise, while the human is working in the right pane, the lead's `show` is refused with an error telling it to say in chat what it would have shown, unless its selected tab is `project` (the fallback the lead shows when nothing needs the human, so the human being there doesn't hold it) or the pane is finished (`Manager.paneFinished`): its task is merged or discarded, or its selected tab is `inbox` and the task has no open items. When an `inbox.updated`, `task.merged` or `task.discarded` event leaves a user's focused task finished that way, that user's signal for it gets a line "Your right pane is free: <task> has nothing left in its inbox." (or "is merged."/"is discarded."), even when the event itself makes no signal, such as a dismissed error; `inbox.updated` on a task that is no longer active adds none. The lead's guidance tells it to show the next thing that needs the human on that line, or, when nothing does, the project's Project tab with a short status in chat: one line per active task, then up to three TODO items it would start next, each with a reason, starting one only when the human picks it. It writes each item as a link to `#/todo/<project_id>/<URL-encoded title>`; Markdown views draw that (`web/src/todolink.ts`) as the title with a **Start** button that calls `POST /api/projects/{id}/todo/tasks` with the first model, as the Project tab's does, or, once an active task was started from that item, the task's colour, dot and state, linking to it. A chat view refills them whenever its transcript grows or tasks or sessions change. When that status says nothing new since its last one, it replies `NO_REPLY` instead. The page sends `PUT /api/me/pane-focus` with `focused` whenever that changes: true while the browser window has focus and the focused element is inside the right pane. The pane itself is focusable (`tabIndex` -1), so a click anywhere in it counts, not only on inputs and buttons; a re-render that removes the focused element hands focus back to the pane rather than counting as leaving. The page sends false on `pagehide` (with `keepalive`) and resends true when its event stream reconnects. byre keeps the flag in memory per user (`Manager.paneFocused`), so it starts false after a restart until the page resends it.
 - **Start hold.** Start on a TODO item in the Project tab leaves the pane where it is. When the user starts one themselves (`StartTodo` with the user as actor, not the lead's `start_todo`), byre sets `Manager.startedTodo` for them, and while it is set and the pane's selected tab is `project`, every `show` that would change the view is refused, focused or not, with an error telling the lead the human is likely to start more. The user's own `PUT /api/me/focus` (switching task or tab) or a message to the lead clears it. It is in memory, so a restart clears it.
@@ -237,7 +237,7 @@ The transcript is the timeline. The UI learns a task's snapshots from the `snaps
 
 The cut point comes from `vendor_uuid`: every top-level `agent.message`, `agent.thinking`, `agent.tool_call` and `agent.tool_result` event (not a subagent's) carries the `uuid` of the Claude Code stream-json line it came from, which is the id of that entry in the CLI's own session transcript. `forkPoint` takes the last such event of the snapshot's session at or before the snapshot's `event_seq`, then extends the cut through the following tool calls and results until every tool call it has kept has its result, so the fork never resumes on an unanswered tool call. For a `turn_end` snapshot the cut is the turn's last message. A Codex session's events carry no `vendor_uuid` except its `turn.ended`, which carries the turn id, so for Codex `forkPoint` picks the last turn that ended at or before the snapshot. Snapshots whose session recorded no `vendor_uuid` by then, such as those from before byre recorded it, can't be forked.
 
-A fork of a task that froze a workflow gets a copy of the source's frozen workflow, not the library's current one (see "Workflows").
+A fork of a task that froze a workflow gets a copy of the source's frozen workflow and brief, not the library's current one, and its session carries the snapshot session's stage, so the fork goes on through the workflow from there in its own worktree (see "Workflows").
 
 The fork's session guidance gets one more line (`forkGuidance` in `internal/agents/guidance.go`) naming the source task, the snapshot, the fork's branch and worktree and the source's worktree, telling the agent that everything before the fork happened in the other worktree and to work only in its own. Claude Code can reuse the system prompt recorded for the source conversation on a fork, the source's cwd included, so the `UserPromptSubmit` hook's copy is what reliably tells it. The fork's transcript starts with a note from its `task.created` event, "forked this task from <source> at snapshot #n", linking the source while it is active; the lead's signal for it reads "<task> was forked by <user> from <source> at snapshot n", and `list_tasks` notes the source. The fork's transcript shows only its own events, not the source's history.
 
@@ -254,14 +254,17 @@ Every new snapshot of a task gets a review: a separate agent session on the vend
 
 ## Workflows
 
-A workflow is a named sequence of stages, each a stage name, a vendor, a model and the stage agent's prompt written out in full. Workflows live in byre's database, in the `workflows` table (`name`, and the definition's JSON as `definition`), shared by every project and never in a repo. Nothing runs a workflow yet: byre stores, validates, exports, imports and freezes them.
+A workflow is a named sequence of stages, each a stage name, a model and the stage agent's prompt written out in full. Workflows live in byre's database, in the `workflows` table (`name`, and the definition's JSON as `definition`), shared by every project and never in a repo. byre stores, validates, exports, imports and freezes them, and runs a task's frozen copy stage by stage (`internal/agents/workflows.go`). There is no UI for starting or managing them: the lead does both with its tools, and draws a workflow for the human as a Mermaid flowchart of its stage names, which its chat renders (see "Inbox").
 
 - **Names.** The name is the workflow's key and is shown to the human on buttons and pickers, so it is free text: one line of 1 to 100 characters, trimmed (`workflow.CleanName`). It isn't part of the definition, so the file doesn't carry it. In URLs it is a path-escaped segment.
 - **File.** One workflow per file: the `WorkflowDefinition` (`api/openapi.yaml`) as YAML, an optional `description` and its `stages`. `workflow.Format` writes it with multi-line prompts as literal blocks; `workflow.Parse` reads YAML or JSON and refuses unknown fields. The download is named after the workflow with `/\:*?"<>|` replaced by `_` (`workflow.Filename`).
-- **Validation.** `workflow.Validate` refuses a definition (400, everything wrong listed) with no stages, a stage name that isn't a valid one-line name or repeats another stage's, an empty prompt, or a `vendor` that isn't the one that runs its `model` (`workflow.VendorOf`, the same rule sessions use: `gpt-*` is Codex, everything else Claude Code).
+- **Validation.** `workflow.Validate` refuses a definition (400, everything wrong listed) with no stages, a stage name that isn't a valid one-line name or repeats another stage's, an empty model or an empty prompt. A stage's vendor follows from its model (`workflow.VendorOf`, the same rule sessions use: `gpt-*` is Codex, everything else Claude Code); a file with a `vendor` key is refused as an unknown field.
 - **API.** `GET /api/workflows` lists every workflow's name and description. `GET /api/workflows/{name}` downloads its file; `PUT` takes a file as the body and creates or replaces the workflow; `DELETE` removes it. Each save or delete records a `workflow.updated` or `workflow.deleted` event carrying the name. `sqlite3 byre.db 'SELECT name FROM workflows'` lists them. Saves and reads go through the Manager (`internal/agents/workflows.go`) so the HTTP API and the lead's tools share them.
 - **Lead tools.** `list_workflows`, `read_workflow` (`name`, or `task_id` for a task's frozen copy), `save_workflow` (`name` and `definition`, the file as YAML) and `delete_workflow`, with the same checks and errors as the API. Task agents don't get them.
-- **Freezing.** `POST /api/projects/{id}/tasks`, and the lead's `create_task` and `start_todo`, take an optional `workflow` (404 when there is none of that name). The task copies the workflow's name and definition as they are at that moment into `tasks.workflow_name` and `tasks.workflow_definition`, so later saves or deletes don't change it. The Task's `workflow` is the name; `GET /api/tasks/{id}/workflow` returns the copy as `{name, definition}` with the definition as YAML (404 for a task without one). A task agent's `create_task` and the Project tab's Start create tasks without one; a fork copies its source's.
+- **Freezing.** `POST /api/projects/{id}/tasks`, and the lead's `create_task` and `start_todo`, take an optional `workflow` (404 when there is none of that name). The task copies the workflow's name and definition as they are at that moment into `tasks.workflow_name` and `tasks.workflow_definition`, so later saves or deletes don't change it, and its brief into `tasks.workflow_brief`: the lead's `create_task` `prompt` (optional with a workflow, and may be empty), or for `start_todo` the TODO item as `todoPrompt` words it. The Task's `workflow` is the name; `GET /api/tasks/{id}/workflow` returns the copy as `{name, definition}` with the definition as YAML (404 for a task without one). A task agent's `create_task` and the Project tab's Start create tasks without one; a fork copies its source's, brief included. `POST /api/projects/{id}/tasks` only freezes: nothing starts its stages.
+- **Starting.** The lead's `create_task` and `start_todo` with `workflow` start stage 1 instead of an agent on the prompt. `model` together with `workflow` is refused, since each stage names its own.
+- **Stages.** Each stage runs in a new implementer session on the stage's model, created by the user who started the workflow; `agent_sessions.workflow_stage` (`workflow_stage` on the AgentSession, from 1) records which. Any other session the task gets later (a Compact handoff's, one started by a message after the last completed, a Sync or Merge with no session) inherits the latest session's stage. Starting stage n records a `task.workflow_stage` event (`workflow`, `stage`, `stages`, `name`; actor the lead for stage 1, `byre` after) and sends the session its first message, actor `byre`, so the transcript shows it collapsed under its first line: `Workflow "<name>", stage <n> of <N>: <stage>`, then the stage's prompt, a `## Brief` section when the brief isn't empty, and from stage 2 a `## From the previous stage (<stage>)` section holding the previous stage's output. Nothing else of earlier stages is passed on; the worktree and branch carry their work.
+- **Advancing.** A stage session gets the `finish_stage` tool (`output`, required) and a guidance line naming the workflow and its stage. The call stores the output in `agent_sessions.stage_output`, replacing any earlier one. When a turn of that session ends normally (not an error or interrupt, not a handoff, not a Merge turn that squashes) with an output stored, byre sets `agent_sessions.stage_finished` before marking the session idle, and then advances. If it isn't the last stage, it completes the session, starts the next stage's session with that output and, only once that session's first message is recorded, clears `stage_output` and `stage_finished`. After the last stage it records `task.workflow_finished` (`workflow`, `output`) and clears them in the same transaction, and the session stays idle for more messages. A turn that ends without the call leaves the task in its stage. The output belongs to the turn that stored it: when that turn ends any other way (an error, an interrupt, a handoff, a Merge turn that squashes, an agent process that exits mid-turn, or a session that can't be resumed after a restart), byre clears `stage_output` before marking the session idle, so a later turn can't hand off a stale output; the agent calls `finish_stage` again. A turn cut off by a byre restart and resumed keeps it. Queued messages and held answers of a finished stage aren't delivered to it: queued ones go to the next stage's session after its turn ends. If the next stage can't start, byre logs `advance workflow stage` and files the error in the task's inbox, and the flag stays set. On startup, before resuming running sessions, `recoverStages` advances every session with `stage_finished` set, skipping the steps already done: it completes the session if it isn't yet; it starts the next stage when there is no later session of that stage; and when there is one that never got a message (`SessionStarted`), it sends it the stage's first message, and the restart notice isn't sent to it. `sqlite3 byre.db 'SELECT id, task_id, workflow_stage FROM agent_sessions WHERE stage_finished = 1'` lists stages waiting to advance. `list_tasks` shows each task's workflow and `stage n/N <name>`, or `finished` once the task has a `task.workflow_finished` event.
 
 ## Events