Accept pasted images in inbox Other and Reply answers

This commit is contained in:
Ian Gulliver
2026-10-16 22:49:06 -07:00
parent a9bec7a7ea
commit ea90178fce
19 changed files with 579 additions and 469 deletions
+3 -1
View File
@@ -233,6 +233,8 @@ Agents ask questions with the `ask_human` MCP tool. Each question carries a `con
| `other` (free text in `text`: an answer, a question back, or new instructions) | questions, errors | `answered` |
| `dismiss` | errors | `dismissed` |
An `other` response, or a pick-any `option` one, may carry `images`, blob ids from `POST /api/blobs` (unknown ids are a 400; see "Images"); an `other` with images may have empty text. They are stored in `inbox_responses.images` and sent to the agent as images with the delivery message, whose text says how many each response attached. Since mid-turn delivery carries only text, answers with images are never handed over mid-turn (`deliver held mid-turn: response has images`); `Deliver` sends them as a new turn once the session is idle.
A submission is all or nothing. `Store.Respond` resolves each item with an `UPDATE` conditioned on `state = 'open'` in one transaction, so when any item was answered, dismissed or withdrawn by someone else in the meantime (a second tab, the lead's `answer_question`, the agent's `cancel_question`), the whole submission is rolled back and refused with 409, and no response is recorded.
Answers reach the agent as one message holding every undelivered `option` and `other` response of the session (`deliveryMessage` in `internal/agents/inbox.go`), never while one of its questions is still open and unanswered (`AwaitingHuman` in `internal/store/inbox.go`); until then they stay held (`inbox_responses.delivered_at` empty). While a turn is running, the task session's PostToolUse hook (`afterTool` in `internal/agents/snapshots.go`, through `deliverMidTurn`) hands them over after the agent's next tool call as the hook's `additionalContext`, after any queued messages (see "Agent sessions"), so the agent reads them alongside that tool's result without being interrupted (for Codex, after each completed tool item, `deliverMidTurn` hands them over with `turn/steer` instead; when the steer fails, `restoreMidTurn` puts the taken queued messages back under their original ids and clears the answers' `delivered_at`, so `Deliver` sends them when the turn ends, and logs `mid-turn delivery restored`); the `user.message` event recording it carries `mid_turn: true` and no `head`, and the transcript doesn't treat it as a turn start. A turn that makes no further tool call leaves them to `Deliver`, which runs after each submission and when a turn ends, and sends them as a new turn once the session is `idle`. The log traces each answer: `inbox response recorded` when it is stored, `deliver held` with the reason, `deliver sent mid-turn`, `deliver sent` with the session's state and whether its agent process had exited or byre was closing, and `undelivered responses` with a reason wherever a session has held answers when its turn ends in a handoff or merge, its agent process ends, startup resumes it, or startup finds it idle. `sqlite3 byre.db "SELECT * FROM inbox_responses WHERE delivered_at IS NULL"` lists the answers held now.
@@ -247,7 +249,7 @@ An open byre page shows a browser notification for each new inbox item, raised b
The left pane's **Settings** tab asks the browser for notification permission and edits the preferences.
The Inbox tab shows one item at a time, the task's oldest, filling the tab without a card frame; the next appears when it is answered. Each item is answered on its own; there is no batch submission. A pick-one question's choices are clickable blocks, and clicking one sends that answer at once. A choice's block (or a pick-any choice's checkbox label) holds the whole list item: its first line, which is the choice's text, and everything under it, such as an example's paragraph or code block. A pick-any question's choices are checkboxes, sent with the Submit button at the bottom of the card. Below the choices, an "Other…" block in the same style opens the only text field. Under it, every question has one submit row: a "Let the agent decide" button at its left, away from the choices so it isn't hit by accident, and the Submit button at its right. Let the agent decide sends at once an `other` answer reading "Use your judgment for this decision and continue within the existing task instructions." (`delegateText` in `web/src/inbox.ts`), which shows in the card's answer list like any other; the `ask_human` description tells agents that this answer settles only that decision and never grants an authorization the owner must give explicitly. The choice blocks, the checkboxes and the Other block take focus with Tab, and ArrowDown/ArrowUp move focus between them; the focused one gets a 2px accent ring. A question has exactly one Submit button, enabled once there is something to send: on a pick-one question it sends the Other text as `other`; on a pick-any question it sends the checked choices with the Other text alongside (one `option` response), or the Other text alone as `other` when nothing is checked. An error has a "Reply…" block (`other`) and a Dismiss button. The Other and Reply text boxes send with the Submit button under them or on Enter (Shift+Enter for a newline), like the message box, and fold back into their block when left empty. The list updates in place: a card stays attached while its item is unchanged, so task and inbox updates from other agents don't pull focus out of a text box being typed in.
The Inbox tab shows one item at a time, the task's oldest, filling the tab without a card frame; the next appears when it is answered. Each item is answered on its own; there is no batch submission. A pick-one question's choices are clickable blocks, and clicking one sends that answer at once. A choice's block (or a pick-any choice's checkbox label) holds the whole list item: its first line, which is the choice's text, and everything under it, such as an example's paragraph or code block. A pick-any question's choices are checkboxes, sent with the Submit button at the bottom of the card. Below the choices, an "Other…" block in the same style opens the only text field. Under it, every question has one submit row: a "Let the agent decide" button at its left, away from the choices so it isn't hit by accident, and the Submit button at its right. Let the agent decide sends at once an `other` answer reading "Use your judgment for this decision and continue within the existing task instructions." (`delegateText` in `web/src/inbox.ts`), which shows in the card's answer list like any other; the `ask_human` description tells agents that this answer settles only that decision and never grants an authorization the owner must give explicitly. The choice blocks, the checkboxes and the Other block take focus with Tab, and ArrowDown/ArrowUp move focus between them; the focused one gets a 2px accent ring. A question has exactly one Submit button, enabled once there is something to send: on a pick-one question it sends the Other text as `other`; on a pick-any question it sends the checked choices with the Other text alongside (one `option` response), or the Other text alone as `other` when nothing is checked. An error has a "Reply…" block (`other`) and a Dismiss button. The Other and Reply text boxes send with the Submit button under them or on Enter (Shift+Enter for a newline), like the message box, and fold back into their block when left empty with no images. Pasting or dropping an image into them adds a thumbnail, as in the composer (the tray is `imageTray` in `web/src/images.ts`, shared by both); images upload when the answer is submitted and are put back if the submission fails. Pending answer images are not kept across a reload. Clicking a pick-one choice sends no images. The list updates in place: a card stays attached while its item is unchanged, so task and inbox updates from other agents don't pull focus out of a text box being typed in.
There is no snoozing. The `snoozed_until` and `snooze_until` columns are still in the schema, always empty, so builds from older branches keep working against the same database; nothing reads them. For the same reason `inbox_items` keeps its `priority` and `blocking` columns, which nothing reads or writes; new rows take their defaults (0 and 1, which older builds read as a blocking question).