Screenshots from implementers

docs/operations.md

@@ -18,7 +18,7 @@
 
 ## Data
 
-State lives in the directory given by `byre serve --data`, by default the `byre` directory under Go's `os.UserConfigDir()` (on macOS, `~/Library/Application Support/byre`): `byre.db`, `worktrees/<project-id>/<task-id>`, `leads/<login>` (each lead agent's working directory, the login path-escaped), `blobs/<blob-id>` (uploaded images; the `blobs` table maps each id to its content type and its path relative to this directory). Task branches are `byre/<task-id>` in the project's repo.
+State lives in the directory given by `byre serve --data`, by default the `byre` directory under Go's `os.UserConfigDir()` (on macOS, `~/Library/Application Support/byre`): `byre.db`, `worktrees/<project-id>/<task-id>`, `worktrees/<project-id>/<task-id>-media` (the task's media folder, see "Screenshots from implementers"), `leads/<login>` (each lead agent's working directory, the login path-escaped), `blobs/<blob-id>` (uploaded images; the `blobs` table maps each id to its content type and its path relative to this directory). Task branches are `byre/<task-id>` in the project's repo.
 
 Schema changes are the `migrations` slice in `internal/store/store.go`: named entries, each with a short kebab-case name saying what it does. On open, byre applies, in slice order, every entry whose name isn't in the `migrations` table, each in its own transaction that also records the name. Each runs with foreign keys off, so a migration can rebuild a table other tables reference, and `PRAGMA foreign_key_check` must come back empty before it commits. byre is deployed from task branches that each append their own migrations, so after a merge a migration can land before others the database has already applied, and it still runs. Give a new migration a name no other branch will use, and make any data migration safe to run twice. `sqlite3 byre.db 'SELECT * FROM migrations'` lists what has been applied. A database from before the table existed is moved over on open: the first `PRAGMA user_version` entries are recorded as applied, and `user_version` is no longer read or written.
 
@@ -128,7 +128,7 @@ Clients reach `tailscale serve` over HTTP/2; it proxies to byre over HTTP/1.1 an
 Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`handsMCP` in `internal/agents/hands.go`), passed the same way as reviewers' Playwright server (see "Reviews"); reviewers and lead agents don't get it. It is a Swift executable built from `hands/` and installed in `~/bin`, which is on the service's `PATH`, and each session runs its own copy, so a hung or crashed one affects only that agent. It speaks MCP's JSON-RPC over stdin and stdout, one message per line, and returns captures as inline PNG image content. Its tools, whose descriptions in `hands/Sources/byre-hands/Tools.swift` are what agents read:
 
 - `screen_list`: running apps, on-screen windows (front to back) and displays.
-- `capture`: one window, or a display, through ScreenCaptureKit. A window is captured as its rectangle of its display with only its own app's windows included, so other apps covering it don't hide it and its menus, popups and sheets show; capturing the window alone fails while a popup is attached to it. The image is scaled to `max_edge` (default 1568, as much as a model sees; 16 to 8192) and the reply gives the pixel-to-point scale.
+- `capture`: one window, or a display, through ScreenCaptureKit. A window is captured as its rectangle of its display with only its own app's windows included, so other apps covering it don't hide it and its menus, popups and sheets show; capturing the window alone fails while a popup is attached to it. The image is scaled to `max_edge` (default 1568, as much as a model sees; 16 to 8192) and the reply gives the pixel-to-point scale. With `save`, it is also written as `capture-<random>.png` in the directory `byre-hands` was started with as `--media-dir`, and the reply gives the path; a `byre-hands` started without one (the macOS app's, which `client_hands` reaches) refuses `save`.
 - `ax_tree`: a window's or an app's menu bar's accessibility tree, one line per element with text or actions, each with a ref (`e12`) that `act` takes. Refs live in the process and each tree replaces them. They remember the window, or the app whose menu bar, they came from, and `act` refuses a ref that isn't from its window or its app's menu bar. Unlabelled containers are left out with their children shown in their place, and static text already contained in its labelled ancestor's text is dropped.
 - `act`: a list of steps in one window: accessibility actions on refs (press, set value, focus, confirm, any named action), then real mouse and keyboard events posted with `CGEvent` (click, double and right click, drag, scroll, typing, key combinations), Before each of those steps it reads the window's current frame, for `[x, y]`, and makes sure the window is in front: its app frontmost and the app's focused window (through accessibility, so the app's own popups don't count against it). When it isn't, it activates the app and raises the window, and stops with an error, sending nothing, if the window still isn't in front. Every session's `byre-hands` drives the same desktop, so a call that can change what is on screen or focused holds an exclusive `flock` on `byre-hands-input.lock` in the user's temporary directory (`$TMPDIR`): an `act` call with steps and every `browser` call except a `tabs` without capture, each for the whole call, its waiting, tree and capture included, so what it returns shows its own work. Another such call waits up to 30s for it, then fails without doing anything. The lock is released when the steps end or the process exits. Typing posts each ASCII character as its key code, with shift where needed, since Orion ignores key events that carry only a Unicode string. Afterwards it can wait (`wait_ms`, then `wait_for`) and return the tree and a capture, so one call clicks, waits and looks. `wait_for` counts the window's elements whose title, description or value contains the text, ignoring case, once before the steps and then every 200ms, and ends when the count rises above the first one (above zero with no steps), so text already on screen, such as another tab's title, doesn't end the wait. Some apps list `confirm` but ignore it, Orion's address bar among them; the tool description says to send Return instead.
 - `browser`: Orion through AppleScript: list tabs, navigate (then poll `document.readyState`), run JavaScript in a tab (`do JavaScript`), select and close tabs, with `wait_for` (the tab's `document.title` and `document.body.innerText`, ignoring case) and capture. Capturing a call that names a tab selects that tab first, since only the selected tab is on screen, and waits 300ms for it to draw. The Orion window is found by its AppleScript `id`, which is its window server window id. Durations (`wait_ms`, `timeout_ms`, `act`'s `wait` steps) must be 0 to 600000 ms. `do JavaScript` needs Orion's Develop → "Allow JavaScript from Apple Events".
@@ -137,6 +137,10 @@ Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`hand
 
 **Permissions.** Agents' processes, `byre-hands` included, run as byre's descendants, so macOS privacy controls (TCC) hold byre responsible for them: the Screen Recording, Accessibility and Automation grants in System Settings → Privacy & Security are byre's, made by clicking at the Mac's screen, and each app scripted for the first time asks for its own Automation grant there. `ax_tree` and `act` check `AXIsProcessTrusted` first and fail saying byre lacks Accessibility when it isn't granted, since an untrusted process otherwise just sees no windows and no menu bar. Accessibility can list byre switched off; only its switch being on counts. `codesign` gives byre the identifier `run.fc.byre` and the Apple Development certificate, which TCC keys grants on, so they carry over from build to build. `codesign -dv ~/bin/byre` shows the signature the running binary has.
 
+## Screenshots from implementers
+
+Each implementer session gets a media folder, `<worktree>-media` (`MediaDir` in `internal/agents/hands.go`), created when the session starts and removed when the task is merged or discarded (`RemoveTask`). It sits outside the worktree, so nothing in it is committed. `byre-hands` gets it as `--media-dir` (for `capture`'s `save`), and implementers also get the reviewers' Playwright server (`playwrightMCP`, see "Reviews") with it as `--output-dir`. The guidance (`screenshotGuidance`) tells implementers that when a turn changes something visible they can embed screenshots in the final message as Markdown images with the paths those tools return; `GET /api/tasks/{taskId}/image` serves the media folder as well as the worktree, and refuses (409) a closed task, so a closed task's screenshots show as "image not available".
+
 ## Previews
 
 An agent shows the human a server it runs with the `expose_port` MCP tool (`internal/agents/preview.go`): `name` labels it, `port` is where it listens. The agent starts the server itself, usually as a background command; byre doesn't supervise it, so it dies with the agent's `claude` process (a byre restart, compaction, the task closing) and the agent starts it again when asked. The call is refused when `port` is outside the task's block or nothing accepts a connection on `127.0.0.1:<port>`. Otherwise byre runs `tailscale serve --bg --https=<port> http://127.0.0.1:<port>` (`internal/tailscale`, which runs the macOS CLI at `/Applications/Tailscale.app/Contents/MacOS/Tailscale`, since byre's environment has no `tailscale` on its `PATH`, with `TAILSCALE_BE_CLI=1`, without which the app executable launched from the service exits 0 without running the subcommand), then reads `tailscale serve status --json` and fails the call if no `Web` host for that port is listed, so the server is at `https://<byre's ts.net name>:<port>/` with Tailscale's identity headers, and records a row in the `previews` table (task, name, port, time) with a `preview.exposed` event, which the transcript shows as a note linking the URL. Each port gets its own HTTPS port rather than a path under byre's, since path-prefix proxying breaks apps that use absolute paths. Exposing a name or port the task already has replaces that row, and a port it no longer uses is turned off with `tailscale serve --https=<port> off`. A server must listen on `127.0.0.1`: while `tailscale serve` holds a port, binding `0.0.0.0` on it fails with "address already in use". When the task is merged or discarded, byre turns off each of its ports (`Manager.Unexpose`); the rows stay. `GET /api/tasks/{taskId}/previews` lists a task's rows by name, and `tailscale serve status` shows what is served.
@@ -288,7 +292,7 @@ Every Markdown view (inbox items and the transcript) renders a line `<details><s
 
 A status mark that is the whole of a table cell, paragraph line, list item or heading, optionally in bold, renders coloured (`marks` in `mac/Sources/Byre/Markdown.swift`): checks (✓, ✅ and the like) in `Palette.ok`, `~`, ≈ and ⚠ in `Palette.warn`, `x`, ×, ✗, ❌ and the like in `Palette.danger`, and `-`, – and — in `Palette.muted`. A table cell whose first word is a mark, alone or followed by text (`✓ cross-vendor`), colours that mark and gets the matching tint background (`Palette.okTint` and the like), `Palette.surface` for the dashes; the rest of its text stays plain. A mark inside other text, or in code, stays plain.
 
-Markdown images (`![alt](src)`, or `![alt](<src>)`; the plain form also accepts spaces, since worktree paths under `Application Support` have one) render inline; clicking one opens it full size in an in-window overlay. A `src` that is an absolute path, a path relative to the worktree or a `file://` URL is loaded from `GET /api/tasks/{taskId}/image?path=` for the item's or event's task, which refuses (403) a path that resolves outside the task's worktree, so the `ask_human` description tells agents to copy a scratch image into the worktree first; a `data:image/` URL of PNG, JPEG, GIF or WebP is used as is; any other scheme, `http(s)` included, shows as a link with the alt text and is never fetched. An image that fails to load is replaced by "image not available: <src>". An inbox `image` attachment renders the same way from its `image` (data URL) or `path`.
+Markdown images (`![alt](src)`, or `![alt](<src>)`; the plain form also accepts spaces, since worktree paths under `Application Support` have one) render inline; clicking one opens it full size in an in-window overlay. A `src` that is an absolute path, a path relative to the worktree or a `file://` URL is loaded from `GET /api/tasks/{taskId}/image?path=` for the item's or event's task, which refuses (403) a path that resolves outside both the task's worktree and its media folder (see "Screenshots from implementers"), so the `ask_human` description tells agents to copy a scratch image into the worktree first; a `data:image/` URL of PNG, JPEG, GIF or WebP is used as is; any other scheme, `http(s)` included, shows as a link with the alt text and is never fetched. An image that fails to load is replaced by "image not available: <src>". An inbox `image` attachment renders the same way from its `image` (data URL) or `path`.
 
 A code block fenced as `mermaid`, and an inbox `diagram` attachment, render as a diagram drawn natively when the source is a flowchart (`mac/Sources/Byre/Diagram.swift`; `docs/macos.md`, "Diagrams", lists what it parses); any other source stays as source with "diagram not rendered: <reason>" under it. The image endpoint (`internal/server/images.go`) serves PNG, JPEG, GIF, WebP and SVG by extension, resolves symlinks on both the worktree and the file and answers 403 for anything outside the worktree, 404 for a missing file, 400 for another extension or a non-file, and 409 once the task is closed (its worktree is gone). Responses carry `X-Content-Type-Options: nosniff` and a sandboxing `Content-Security-Policy`, so an SVG opened directly runs no script, and `Cache-Control: no-cache` with `Last-Modified`, so a changed file is refetched.