Run each workflow stage as its own task, with fan-out and fan-in

docs/operations.md

@@ -145,7 +145,7 @@ Each user has one long-running lead agent, shown in the left pane. It is not tie
 
 - **Sessions.** A lead session is an `agent_sessions` row with `owner` set to the user's login, `role` `lead` and no `task_id`. Its events carry `owner` and no project or task. It runs in `leads/<login>` with `leadGuidance` (passed and re-attached the same way as a task agent's guidance) and the lead tools instead of the task tools. It has no Stop-hook bounce, no snapshots, no end-of-turn commit and no turn summary; a turn that ends in error records the error on its `session.state` event, which the pane shows as a note.
 - **Messages.** `POST /api/me/lead/messages` sends the caller's message to their lead, starting a lead session when there is none. While the lead is mid-turn the message is held in `lead_inputs`; the lead's PostToolUse hook (Codex: `turn/steer` after each completed tool item) hands held messages over after its next tool call, like a task's queued messages (see "Agent sessions"), and any still held become its next turn. Signals are never handed over mid-turn. The composer stays enabled. `GET /api/me/lead/sessions` lists the caller's lead sessions; interrupt and compact use the usual session endpoints.
-- **Signals.** Every published event goes through `Manager.Observe`. These become signals in `lead_inputs` for every user who has a lead session: `inbox.created` (a question with its choices and recommendation, an error or rate limit), `inbox.updated` when a human answered a question or error, `task.created` (naming the TODO item it was started from, if any), `task.merged`, `task.discarded`, `turn.ended` with `summary: true` ("<task> finished a turn:" with its `result` cut to 1500 characters), `task.workflow_stage` ("<task> started stage n/N of "<workflow>": <stage>.") and `task.workflow_finished` ("<task> finished workflow "<workflow>":" with its output cut to 1500 characters). The human switching the right pane is not a signal. A signal caused by the lead itself (actor `<login> (lead)`) is skipped, as is the update from an answer it gave. A task's `task.created` event carries the actor that created it, separately from its `created_by` login: tasks the lead creates with `create_task` or `start_todo` keep the human's login as `created_by` but have actor `<login> (lead)`. Whenever the lead is idle, byre sends the oldest held message from the human as its own turn, or else every held signal as one `byre` message headed "Signals since your last turn (n)". Its guidance tells it not to respond to the human's own actions, and to make its final message exactly `NO_REPLY` when a batch needs nothing from it. A transcript (`quietReply` in `web/src/task.ts`) never draws an `agent.message` whose last line is `NO_REPLY` (the model sometimes writes a sentence before it), and hides the live partial message while its text is still a prefix of it; the partial message starts hidden and is first shown by its 100ms render, so an empty one never takes up space. A lead turn that byre starts without a handoff (a signal batch or a handoff continuation) is held (`ctx.held`): its `byre` message isn't drawn and doesn't open the turn, so the spinner stays off, and its partial message stays hidden, until the lead draws something else, such as a tool call, thinking or a real message, which draws the `byre` message ahead of it. When such a turn ends in `NO_REPLY`, everything it drew is removed with it (its `byre` message, tool calls, thinking), so the transcript keeps nothing of it; a turn that ends in a real message keeps all of it.
+- **Signals.** Every published event goes through `Manager.Observe`. These become signals in `lead_inputs` for every user who has a lead session: `inbox.created` (a question with its choices and recommendation, an error or rate limit), `inbox.updated` when a human answered a question or error, `task.created` (naming the TODO item it was started from, if any), `task.merged`, `task.discarded`, `turn.ended` with `summary: true` ("<task> finished a turn:" with its `result` cut to 1500 characters), `task.workflow_stage` ("<task> started stage "<stage>" of workflow "<workflow>".") and `task.workflow_finished` ("<task> finished workflow "<workflow>":" with its output cut to 1500 characters). The human switching the right pane is not a signal. A signal caused by the lead itself (actor `<login> (lead)`) is skipped, as is the update from an answer it gave. A task's `task.created` event carries the actor that created it, separately from its `created_by` login: tasks the lead creates with `create_task` or `start_todo` keep the human's login as `created_by` but have actor `<login> (lead)`. Whenever the lead is idle, byre sends the oldest held message from the human as its own turn, or else every held signal as one `byre` message headed "Signals since your last turn (n)". Its guidance tells it not to respond to the human's own actions, and to make its final message exactly `NO_REPLY` when a batch needs nothing from it. A transcript (`quietReply` in `web/src/task.ts`) never draws an `agent.message` whose last line is `NO_REPLY` (the model sometimes writes a sentence before it), and hides the live partial message while its text is still a prefix of it; the partial message starts hidden and is first shown by its 100ms render, so an empty one never takes up space. A lead turn that byre starts without a handoff (a signal batch or a handoff continuation) is held (`ctx.held`): its `byre` message isn't drawn and doesn't open the turn, so the spinner stays off, and its partial message stays hidden, until the lead draws something else, such as a tool call, thinking or a real message, which draws the `byre` message ahead of it. When such a turn ends in `NO_REPLY`, everything it drew is removed with it (its `byre` message, tool calls, thinking), so the transcript keeps nothing of it; a turn that ends in a real message keeps all of it.
 - **Tools.** `list_tasks` (each project's active tasks, then a line listing the titles of its `TODO.md` items on the default branch that no active task was started from), `read_transcript` (a task's events in brief, the last `limit` or those after `after_seq`), `read_inbox`, `answer_question` (an `option`, `other` or `dismiss` response recorded as the user, through the same checks as `POST /api/inbox/responses`), `send_message` (starts the task's agent if it has none; refused while it is mid-turn), `create_task` and `start_todo` (each with an optional `workflow` to freeze and run), `interrupt`, `compact_task`, `merge_task`, `discard_task`, `show`, `compact_self`, and the workflow tools (see "Workflows"). The lead also has Claude Code's own tools and reads worktrees by path. Messages it sends task agents have actor `<login> (lead)`, so they don't make the user a participant; tasks, merges, discards and answers are done as the user. Its guidance says to answer a question only when the answer is clear or the human gave it, and to merge or discard only when the human asks.
 - **Focus.** `users.focus` holds what the user's right pane shows: `project_id`, `task_id`, `tabs` (the tabs put first, in order) and `tab` (the selected one). With a `task_id`, `project_id` is always that task's project, filled in by the server; with only a `project_id`, the pane shows that project with no task, which only the Project tab has anything for. `GET /api/me/focus` reads it; `PUT /api/me/focus` and the lead's `show` set it, refusing an unknown tab, a task that isn't active or an unknown project, and record a `focus.updated` event owned by the user. The page applies the focus on load and on every `focus.updated` it didn't cause itself; the human's own changes (⌘K, the right pane's task picker, clicking a tab, a task link, a notification) are sent with `PUT /api/me/focus`. The URL follows the focus: `#/task/<id>`, `#/project/<id>`, or `#/` for neither.
 - **Pane lock.** A `show` of what the pane already shows (the same task, or the same project with no task, on the same selected tab; `sameView`) tells the lead so and applies only its `tabs` order, whether or not the human is in the pane, so the human's selected tab stays selected. Otherwise, while the human is working in the right pane, the lead's `show` is refused with an error telling it to say in chat what it would have shown, unless its selected tab is `project` (the fallback the lead shows when nothing needs the human, so the human being there doesn't hold it) or the pane is finished (`Manager.paneFinished`): its task is merged or discarded, or its selected tab is `inbox` and the task has no open items. When an `inbox.updated`, `task.merged` or `task.discarded` event leaves a user's focused task finished that way, that user's signal for it gets a line "Your right pane is free: <task> has nothing left in its inbox." (or "is merged."/"is discarded."), even when the event itself makes no signal, such as a dismissed error; `inbox.updated` on a task that is no longer active adds none. The lead's guidance tells it to show the next thing that needs the human on that line, or, when nothing does, the project's Project tab with a short status in chat: one line per active task, then up to three TODO items it would start next, each with a reason, starting one only when the human picks it. It writes each item as a link to `#/todo/<project_id>/<URL-encoded title>`; Markdown views draw that (`web/src/todolink.ts`) as the title with a **Start** button that calls `POST /api/projects/{id}/todo/tasks` with the first model, as the Project tab's does, or, once an active task was started from that item, the task's colour, dot and state, linking to it. A chat view refills them whenever its transcript grows or tasks or sessions change. When that status says nothing new since its last one, it replies `NO_REPLY` instead. The page sends `PUT /api/me/pane-focus` with `focused` whenever that changes: true while the browser window has focus and the focused element is inside the right pane. The pane itself is focusable (`tabIndex` -1), so a click anywhere in it counts, not only on inputs and buttons; a re-render that removes the focused element hands focus back to the pane rather than counting as leaving. The page sends false on `pagehide` (with `keepalive`) and resends true when its event stream reconnects. byre keeps the flag in memory per user (`Manager.paneFocused`), so it starts false after a restart until the page resends it.
@@ -237,7 +237,7 @@ The transcript is the timeline. The UI learns a task's snapshots from the `snaps
 
 The cut point comes from `vendor_uuid`: every top-level `agent.message`, `agent.thinking`, `agent.tool_call` and `agent.tool_result` event (not a subagent's) carries the `uuid` of the Claude Code stream-json line it came from, which is the id of that entry in the CLI's own session transcript. `forkPoint` takes the last such event of the snapshot's session at or before the snapshot's `event_seq`, then extends the cut through the following tool calls and results until every tool call it has kept has its result, so the fork never resumes on an unanswered tool call. For a `turn_end` snapshot the cut is the turn's last message. A Codex session's events carry no `vendor_uuid` except its `turn.ended`, which carries the turn id, so for Codex `forkPoint` picks the last turn that ended at or before the snapshot. Snapshots whose session recorded no `vendor_uuid` by then, such as those from before byre recorded it, can't be forked.
 
-A fork of a task that froze a workflow gets a copy of the source's frozen workflow and brief, not the library's current one, and its session carries the snapshot session's stage, so the fork goes on through the workflow from there in its own worktree (see "Workflows").
+A fork of a workflow stage task gets a copy of the source's frozen workflow, brief, run and stage, not the library's current workflow, so the fork is another attempt at that stage of the same run in its own worktree; when it finishes, the stages after it start from it unless they already have tasks (see "Workflows").
 
 The fork's session guidance gets one more line (`forkGuidance` in `internal/agents/guidance.go`) naming the source task, the snapshot, the fork's branch and worktree and the source's worktree, telling the agent that everything before the fork happened in the other worktree and to work only in its own. Claude Code can reuse the system prompt recorded for the source conversation on a fork, the source's cwd included, so the `UserPromptSubmit` hook's copy is what reliably tells it. The fork's transcript starts with a note from its `task.created` event, "forked this task from <source> at snapshot #n", linking the source while it is active; the lead's signal for it reads "<task> was forked by <user> from <source> at snapshot n", and `list_tasks` notes the source. The fork's transcript shows only its own events, not the source's history.
 
@@ -254,17 +254,16 @@ Every new snapshot of a task gets a review: a separate agent session on the vend
 
 ## Workflows
 
-A workflow is a named sequence of stages, each a stage name, a model and the stage agent's prompt written out in full. Workflows live in byre's database, in the `workflows` table (`name`, and the definition's JSON as `definition`), shared by every project and never in a repo. byre stores, validates, exports, imports and freezes them, and runs a task's frozen copy stage by stage (`internal/agents/workflows.go`). There is no UI for starting or managing them: the lead does both with its tools, and draws a workflow for the human as a Mermaid flowchart of its stage names, which its chat renders (see "Inbox").
+A workflow is a named graph of stages, each a stage name, the stages it comes after, where its branch starts, a model and the stage agent's prompt written out in full. Every stage runs as its own task, once all the stages it comes after have finished, so one stage's output can go to several stages (fan-out) and one stage can wait for several (fan-in). Workflows live in byre's database, in the `workflows` table (`name`, and the definition's JSON as `definition`), shared by every project and never in a repo. byre stores, validates, exports, imports and freezes them, and runs each run's frozen copy stage by stage (`internal/agents/workflows.go`; the graph helpers `After`, `Next`, `Starts` and `FromMain` are in `internal/workflow/workflow.go`). There is no UI for starting or managing them: the lead does both with its tools, and draws a workflow for the human as a Mermaid flowchart of its stage names, which its chat renders (see "Inbox").
 
 - **Names.** The name is the workflow's key and is shown to the human on buttons and pickers, so it is free text: one line of 1 to 100 characters, trimmed (`workflow.CleanName`). It isn't part of the definition, so the file doesn't carry it. In URLs it is a path-escaped segment.
-- **File.** One workflow per file: the `WorkflowDefinition` (`api/openapi.yaml`) as YAML, an optional `description` and its `stages`. `workflow.Format` writes it with multi-line prompts as literal blocks; `workflow.Parse` reads YAML or JSON and refuses unknown fields. The download is named after the workflow with `/\:*?"<>|` replaced by `_` (`workflow.Filename`).
-- **Validation.** `workflow.Validate` refuses a definition (400, everything wrong listed) with no stages, a stage name that isn't a valid one-line name or repeats another stage's, an empty model or an empty prompt. A stage's vendor follows from its model (`workflow.VendorOf`, the same rule sessions use: `gpt-*` is Codex, everything else Claude Code); a file with a `vendor` key is refused as an unknown field.
+- **File.** One workflow per file: the `WorkflowDefinition` (`api/openapi.yaml`) as YAML, an optional `description` and its `stages`. A stage's `after` lists the stages whose outputs it gets; absent, it means the previous stage in the list (none for the first), so a file without `after` is a chain; `[]` means the stage starts from the brief. Its `base` is `parent` (the default: the commit its one `after` stage finished at, or for a stage with none, where the run was started from) or `main` (the project's default branch); a stage with more than one `after` stage always starts from the default branch. `workflow.Format` writes it with multi-line prompts as literal blocks; `workflow.Parse` reads YAML or JSON and refuses unknown fields. The download is named after the workflow with `/\:*?"<>|` replaced by `_` (`workflow.Filename`).
+- **Validation.** `workflow.Validate` refuses a definition (400, everything wrong listed) with no stages, a stage name that isn't a valid one-line name or repeats another stage's, an empty model or an empty prompt, a `base` other than `parent` or `main`, a `base` on a stage with more than one `after` stage, an `after` naming the stage itself, a name twice or a name that isn't a stage, a cycle, or anything but exactly one final stage (one no other stage comes after). A stage's vendor follows from its model (`workflow.VendorOf`, the same rule sessions use: `gpt-*` is Codex, everything else Claude Code); a file with a `vendor` key is refused as an unknown field.
 - **API.** `GET /api/workflows` lists every workflow's name and description. `GET /api/workflows/{name}` downloads its file; `PUT` takes a file as the body and creates or replaces the workflow; `DELETE` removes it. Each save or delete records a `workflow.updated` or `workflow.deleted` event carrying the name. `sqlite3 byre.db 'SELECT name FROM workflows'` lists them. Saves and reads go through the Manager (`internal/agents/workflows.go`) so the HTTP API and the lead's tools share them.
 - **Lead tools.** `list_workflows`, `read_workflow` (`name`, or `task_id` for a task's frozen copy), `save_workflow` (`name` and `definition`, the file as YAML) and `delete_workflow`, with the same checks and errors as the API. Task agents don't get them.
-- **Freezing.** `POST /api/projects/{id}/tasks`, and the lead's `create_task` and `start_todo`, take an optional `workflow` (404 when there is none of that name). The task copies the workflow's name and definition as they are at that moment into `tasks.workflow_name` and `tasks.workflow_definition`, so later saves or deletes don't change it, and its brief into `tasks.workflow_brief`: the lead's `create_task` `prompt` (optional with a workflow, and may be empty), or for `start_todo` the TODO item as `todoPrompt` words it. The Task's `workflow` is the name; `GET /api/tasks/{id}/workflow` returns the copy as `{name, definition}` with the definition as YAML (404 for a task without one). A task agent's `create_task` and the Project tab's Start create tasks without one; a fork copies its source's, brief included. `POST /api/projects/{id}/tasks` only freezes: nothing starts its stages.
-- **Starting.** The lead's `create_task` and `start_todo` with `workflow` start stage 1 instead of an agent on the prompt. `model` together with `workflow` is refused, since each stage names its own.
-- **Stages.** Each stage runs in a new implementer session on the stage's model, created by the user who started the workflow; `agent_sessions.workflow_stage` (`workflow_stage` on the AgentSession, from 1) records which. Any other session the task gets later (a Compact handoff's, one started by a message after the last completed, a Sync or Merge with no session) inherits the latest session's stage. Starting stage n records a `task.workflow_stage` event (`workflow`, `stage`, `stages`, `name`; actor the lead for stage 1, `byre` after) and sends the session its first message, actor `byre`, so the transcript shows it collapsed under its first line: `Workflow "<name>", stage <n> of <N>: <stage>`, then the stage's prompt, a `## Brief` section when the brief isn't empty, and from stage 2 a `## From the previous stage (<stage>)` section holding the previous stage's output. Nothing else of earlier stages is passed on; the worktree and branch carry their work.
-- **Advancing.** A stage session gets the `finish_stage` tool (`output`, required) and a guidance line naming the workflow and its stage. The call stores the output in `agent_sessions.stage_output`, replacing any earlier one. When a turn of that session ends normally (not an error or interrupt, not a handoff, not a Merge turn that squashes) with an output stored, byre sets `agent_sessions.stage_finished` before marking the session idle, and then advances. If it isn't the last stage, it completes the session, starts the next stage's session with that output and, only once that session's first message is recorded, clears `stage_output` and `stage_finished`. After the last stage it records `task.workflow_finished` (`workflow`, `output`) and clears them in the same transaction, and the session stays idle for more messages. A turn that ends without the call leaves the task in its stage. The output belongs to the turn that stored it: when that turn ends any other way (an error, an interrupt, a handoff, a Merge turn that squashes, an agent process that exits mid-turn, or a session that can't be resumed after a restart), byre clears `stage_output` before marking the session idle, so a later turn can't hand off a stale output; the agent calls `finish_stage` again. A turn cut off by a byre restart and resumed keeps it. Queued messages and held answers of a finished stage aren't delivered to it: queued ones go to the next stage's session after its turn ends. If the next stage can't start, byre logs `advance workflow stage` and files the error in the task's inbox, and the flag stays set. On startup, before resuming running sessions, `recoverStages` advances every session with `stage_finished` set, skipping the steps already done: it completes the session if it isn't yet; it starts the next stage when there is no later session of that stage; and when there is one that never got a message (`SessionStarted`), it sends it the stage's first message, and the restart notice isn't sent to it. `sqlite3 byre.db 'SELECT id, task_id, workflow_stage FROM agent_sessions WHERE stage_finished = 1'` lists stages waiting to advance. `list_tasks` shows each task's workflow and `stage n/N <name>`, or `finished` once the task has a `task.workflow_finished` event.
+- **Starting.** The lead's `create_task` and `start_todo` take an optional `workflow` (404 when there is none of that name) and then start a run of it instead of an agent on the prompt (`Manager.StartWorkflow`); `model` together with `workflow` is refused, since each stage names its own. A run gets a new id and creates a task for every stage with no `after` stages, branched from the default branch, or for a stage with `base: parent`, from the lead's `from_task_id` HEAD when given. Each stage task copies the workflow's name and definition as they are when the run starts into `tasks.workflow_name` and `tasks.workflow_definition`, so later saves or deletes don't change it; its brief into `tasks.workflow_brief` (the lead's `create_task` `prompt`, optional with a workflow and may be empty, or for `start_todo` the TODO item as `todoPrompt` words it); the run's title into `tasks.workflow_title`; the run id into `tasks.workflow_run`; and its stage's name into `tasks.workflow_stage`. Stage tasks are titled `<title>: <stage>`, with the run's title as their merge title, and carry the run's TODO item. The Task's `workflow`, `workflow_run` and `workflow_stage` are those; `GET /api/tasks/{id}/workflow` returns the copy as `{name, definition}` with the definition as YAML (404 for a task without one). A task agent's `create_task`, `POST /api/projects/{id}/tasks` and the Project tab's Start create tasks without one. A fork of a stage task copies its frozen workflow, brief, title, run and stage, so it is another attempt at the same stage of the same run. `sqlite3 byre.db "SELECT id, status, workflow_stage, stage_output != '' FROM tasks WHERE workflow_run = '<run>'"` lists a run's tasks and which have finished.
+- **Stages.** A stage task's agent is an implementer session on the stage's model, created by the user who started the run; any later session of the task (a Compact handoff's, one started by a message) runs the same stage, since the stage belongs to the task. Starting a stage records a `task.workflow_stage` event on its task (`workflow`, `run`, `stage` the name, `after` the ids of the tasks whose outputs it gets; actor the lead for the stages a run starts with, `byre` after) and sends the session its first message, actor `byre`, so the transcript shows it collapsed under its first line: `Workflow "<name>", stage "<stage>"`, then the stage's prompt, a `## Brief` section when the brief isn't empty, a `## From stage "<stage>" (task <id>, branch <branch>)` section with each `after` stage's output, and a line saying where the task starts: from the default branch, with the branches above to `git merge` or read as it needs, or from the commit its one `after` stage finished at. Nothing else of earlier stages is passed on; their branches carry their work.
+- **Advancing.** A stage session gets the `finish_stage` tool (`output`, required) and a guidance line naming the workflow, its stage and the stages after it. The call stores the output in `agent_sessions.stage_output`, replacing any earlier one; it is refused once the task's stage has finished. When a turn of that session ends normally (not an error or interrupt, not a handoff, not a Merge turn that squashes) with an output stored, byre sets `agent_sessions.stage_finished` before marking the session idle, and then advances (`nextStage`): it records the output and the worktree's HEAD in `tasks.stage_output` and `tasks.stage_commit`, then, holding `workflowMu` so two stages finishing at once can't both start the one after them, goes through each stage after this one. A stage starts once every stage it comes after has a finished task in the run (the latest one, when a fork made several); its task branches from that task's `stage_commit` for `base: parent` with one `after` stage, else from the default branch. Once every stage after it has been dealt with, byre clears `stage_output` and `stage_finished` on the session. The session stays idle in its task for more messages, and queued messages and held answers are delivered to it once advancing is done. When the final stage finishes, byre discards every other active task of the run (as `byre`, through the same `Discard` as the button), then records `task.workflow_finished` (`workflow`, `output`) on the final task and clears the flags in the same transaction, leaving the final task for the human to merge. A turn that ends without the call leaves the task in its stage. The output belongs to the turn that stored it: when that turn ends any other way (an error, an interrupt, a handoff, a Merge turn that squashes, an agent process that exits mid-turn, or a session that can't be resumed after a restart), byre clears `stage_output` before marking the session idle, so a later turn can't hand off a stale output; the agent calls `finish_stage` again. A turn cut off by a byre restart and resumed keeps it. If a stage can't start, byre logs `advance workflow stage` and files the error in the finishing task's inbox, and the flag stays set. On startup, before resuming running sessions, `recoverStages` advances every session with `stage_finished` set, skipping the steps already done: the recorded output isn't overwritten; a stage that already has a task in the run isn't created again, but when that task is active and has no session, its session is started, and when its session never got a message (`SessionStarted`), it is sent the stage's first message, and the restart notice isn't sent to it. `sqlite3 byre.db 'SELECT id, task_id FROM agent_sessions WHERE stage_finished = 1'` lists stages waiting to advance. `list_tasks` shows each stage task's workflow and `stage "<name>"`, with `(finished)` once its stage has finished and `workflow finished` once any task of the run has a `task.workflow_finished` event. Nothing reads or writes `agent_sessions.workflow_stage`; it stays in the schema so builds from older branches keep working against the same database.
 
 ## Events