diff --git a/docs/operations.md b/docs/operations.md index 7f253bb..1e1a048 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -144,7 +144,7 @@ Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`hand - `ax_tree`: a window's or an app's menu bar's accessibility tree, one line per element with text or actions, each with a ref (`e12`) that `act` takes. Refs live in the process and each tree replaces them. They remember the window, or the app whose menu bar, they came from, and `act` refuses a ref that isn't from its window or its app's menu bar. Unlabelled containers are left out with their children shown in their place, and static text already contained in its labelled ancestor's text is dropped. - `act`: a list of steps in one window: accessibility actions on refs (press, set value, focus, confirm, any named action), then real mouse and keyboard events posted with `CGEvent` (click, double and right click, drag, scroll, typing, key combinations), Before each of those steps it reads the window's current frame, for `[x, y]`, and makes sure the window is in front: its app frontmost and the app's focused window (through accessibility, so the app's own popups don't count against it). When it isn't, it activates the app and raises the window, and stops with an error, sending nothing, if the window still isn't in front. Every session's `byre-hands` drives the same desktop, so a call that can change what is on screen or focused holds an exclusive `flock` on `byre-hands-input.lock` in the user's temporary directory (`$TMPDIR`): an `act` call with steps and every `browser` call except a `tabs` without capture, each for the whole call, its waiting, tree and capture included, so what it returns shows its own work. Another such call waits up to 30s for it, then fails without doing anything. The lock is released when the steps end or the process exits. Typing posts each ASCII character as its key code, with shift where needed, since Orion ignores key events that carry only a Unicode string. Afterwards it can wait (`wait_ms`, then `wait_for`) and return the tree and a capture, so one call clicks, waits and looks. `wait_for` counts the window's elements whose title, description or value contains the text, ignoring case, once before the steps and then every 200ms, and ends when the count rises above the first one (above zero with no steps), so text already on screen, such as another tab's title, doesn't end the wait. Some apps list `confirm` but ignore it, Orion's address bar among them; the tool description says to send Return instead. - `browser`: Orion through AppleScript: list tabs, navigate (then poll `document.readyState`), run JavaScript in a tab (`do JavaScript`), select and close tabs, with `wait_for` (the tab's `document.title` and `document.body.innerText`, ignoring case) and capture. Capturing a call that names a tab selects that tab first, since only the selected tab is on screen, and waits 300ms for it to draw. The Orion window is found by its AppleScript `id`, which is its window server window id. Durations (`wait_ms`, `timeout_ms`, `act`'s `wait` steps) must be 0 to 600000 ms. `do JavaScript` needs Orion's Develop → "Allow JavaScript from Apple Events". -- `chrome`: Google Chrome through the DevTools protocol (`Chrome.swift`). The first call finds Chrome by `GET http://127.0.0.1:9222/json/version`; when nothing answers it runs `open -na "Google Chrome" --args --user-data-dir=~/Library/Application Support/byre/chrome --remote-debugging-port=9222 --no-first-run --no-default-browser-check` and polls for 15s. Chrome refuses remote debugging on its default profile, so byre's Chrome is its own profile, a separate Chrome beside any the human runs, and it keeps its logins: the human signs in there (passkeys included) and every later call, from any agent on that Mac, uses the session. The process holds one WebSocket to the browser endpoint for its life, reconnecting (and relaunching Chrome) on the next call after it drops, and attaches to each tab it touches with `Target.attachToTarget` (flattened sessions), enabling `Page`, `Network`, `Runtime` and `Log` there; concurrent calls for a tab share one attachment, and none uses the tab until all four are enabled. From then on it keeps that tab's requests (the last 5000) and console messages, exceptions and log entries (the last 2000) in memory; `navigate` clears both, `network` and `console` read them (each redirect hop is its own row, and the raw headers of `requestWillBeSentExtraInfo` and `responseReceivedExtraInfo`, which carry cookies, are paired with a request's hops in order, as DevTools does, with a hop whose response says `hasExtraInfo` (or `redirectHasExtraInfo`) false, such as a cached redirect, holding an empty place), and a request's body comes from `Network.getResponseBody` when asked for, shown up to 64 KiB. `js` is `Runtime.evaluate` with `awaitPromise` and `replMode`. `trace_start` and `trace_stop` run `Tracing` on the tab's session with the DevTools timeline categories, stream the trace to `byre-trace-.json` in the temporary directory (a client's own, on a client Mac) and summarise it from the trace events: FCP, LCP, DOMContentLoaded and load of the tab's main frame (whose trace frame id is the tab's target id, so other tabs and iframes don't count) after its last `navigationStart`, and the `RunTask`s of 50ms or more on its `CrRendererMain` thread, each with the longest script, parse, style, layout, paint or GC event inside it. `cdp` sends any method to the tab's session. Capture is `Page.bringToFront` then `Page.captureScreenshot`, scaled to `max_edge`. It drives no mouse or keyboard, but `select`, `close`, `navigate` with `new_tab`, capture, starting Chrome and any `cdp` method (some, such as `Page.bringToFront`, do) can change which page is in front, so those calls hold the input lock (see `act`) for the whole call, and starting Chrome holds it while it waits for the port; the other ops take no lock. Chrome comes from `brew install --cask google-chrome`; macOS's "downloaded from the Internet" prompt on its first launch must be answered Open (or `osascript -e 'tell application "System Events" to tell process "CoreServicesUIAgent" to click button "Open" of window 1'`), and until it is, Chrome runs without opening its port. +- `chrome`: Google Chrome through the DevTools protocol (`Chrome.swift`). The first call finds Chrome by `GET http://127.0.0.1:9222/json/version`; when nothing answers it runs `open -na "Google Chrome" --args --user-data-dir=~/Library/Application Support/byre/chrome --remote-debugging-port=9222 --no-first-run --no-default-browser-check` and polls for 15s. Chrome refuses remote debugging on its default profile, so byre's Chrome is its own profile, a separate Chrome beside any the human runs, and it keeps its logins: the human signs in there (passkeys included) and every later call, from any agent on that Mac, uses the session. The process holds one WebSocket to the browser endpoint for its life, reconnecting (and relaunching Chrome) on the next call after it drops, and attaches to each tab it touches with `Target.attachToTarget` (flattened sessions), enabling `Page`, `Network`, `Runtime` and `Log` there; concurrent calls for a tab share one attachment, and none uses the tab until all four are enabled. From then on it keeps that tab's requests (the last 5000) and console messages, exceptions and log entries (the last 2000) in memory; `navigate` clears both, `network` and `console` read them (each redirect hop is its own row, and the raw headers of `requestWillBeSentExtraInfo` and `responseReceivedExtraInfo`, which carry cookies, are paired with a request's hops in order, as DevTools does, with a hop whose response says `hasExtraInfo` (or `redirectHasExtraInfo`) false, such as a cached redirect, holding an empty place), and a request's body comes from `Network.getResponseBody` when asked for, shown up to 64 KiB. `js` is `Runtime.evaluate` with `awaitPromise` and `replMode`. `trace_start` and `trace_stop` run `Tracing` on the tab's session with the DevTools timeline categories, stream the trace to `byre-trace-.json` in the temporary directory (a client's own, on a client Mac) and summarise it from the trace events: FCP, LCP, DOMContentLoaded and load of the tab's main frame (whose trace frame id is the tab's target id, so other tabs and iframes don't count) after its last `navigationStart`, and the `RunTask`s of 50ms or more on its `CrRendererMain` thread, each with the longest script, parse, style, layout, paint or GC event inside it. `cdp` sends any method to the tab's session. Capture is `Page.bringToFront` then `Page.captureScreenshot`, scaled to `max_edge`; with `save` it is also written to the media folder as `capture`'s `save` does, through the same code. It drives no mouse or keyboard, but `select`, `close`, `navigate` with `new_tab`, capture, starting Chrome and any `cdp` method (some, such as `Page.bringToFront`, do) can change which page is in front, so those calls hold the input lock (see `act`) for the whole call, and starting Chrome holds it while it waits for the port; the other ops take no lock. Chrome comes from `brew install --cask google-chrome`; macOS's "downloaded from the Internet" prompt on its first launch must be answered Open (or `osascript -e 'tell application "System Events" to tell process "CoreServicesUIAgent" to click button "Open" of window 1'`), and until it is, Chrome runs without opening its port. `byre-hands` reads stdin on a background thread and starts each request as its own detached task as soon as it arrives, so one call stuck on an app doesn't hold up the others; responses are written whole, one line each, under a lock, in the order they finish. Ref names come from one counter for the life of the process, so no two trees ever share a name, even trees built at the same time; the ref table is swapped in under a lock once a tree is complete, so a ref from any tree but the last published one fails as unknown instead of naming another tree's element. Each `act` call takes the table as it stands when the call arrives and resolves every step's refs from that copy, so a tree finishing during its waits can't point a ref at a different element. AppleScript needs the main thread with a run loop, so every script runs there (`osa` in `Browser.swift`, `DispatchQueue.main.sync`, while the main thread runs `RunLoop.main`), one at a time, wrapped in `with timeout of 10 seconds`; `NSApplication.shared` connects the process to the window server at startup, without which ScreenCaptureKit asserts. An app that is in a menu-tracking loop or otherwise busy answers neither Apple events nor accessibility requests: a script fails with AppleScript's timeout error after 10s, and accessibility requests time out after 3s (the messaging timeout set on the system-wide element at startup), and every accessibility read (`attr` in `AX.swift`, and the action-name and settable checks in the tree walk) throws on the first timeout, so a tree walk, `wait_for`, a ref's position or the front-window check stops with an error saying the app isn't answering instead of waiting 3s on every attribute of every element. @@ -152,7 +152,7 @@ Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`hand ## Screenshots from implementers -Each implementer session gets a media folder, `-media` (`MediaDir` in `internal/agents/hands.go`), created when the session starts and removed when the task is merged or discarded (`RemoveTask`). It sits outside the worktree, so nothing in it is committed. `byre-hands` gets it as `--media-dir` (for `capture`'s `save`), and implementers also get the reviewers' Playwright server (`playwrightMCP`, see "Reviews") with it as `--output-dir`. The guidance (`screenshotGuidance`) tells implementers that when a turn changes something visible they can embed screenshots in the final message as Markdown images with the paths those tools return; `GET /api/tasks/{taskId}/image` serves the media folder as well as the worktree, and refuses (409) a closed task, so a closed task's screenshots show as "image not available". +Each implementer session gets a media folder, `-media` (`MediaDir` in `internal/agents/hands.go`), created when the session starts and removed when the task is merged or discarded (`RemoveTask`). It sits outside the worktree, so nothing in it is committed. `byre-hands` gets it as `--media-dir` (for `save` on `capture` and `chrome`), and implementers also get the reviewers' Playwright server (`playwrightMCP`, see "Reviews") with it as `--output-dir`. The guidance (`screenshotGuidance`) tells implementers that when a turn changes something visible they can embed screenshots in the final message as Markdown images with the paths those tools return; `GET /api/tasks/{taskId}/image` serves the media folder as well as the worktree, and refuses (409) a closed task, so a closed task's screenshots show as "image not available". ## Previews diff --git a/hands/Sources/byre-hands/Chrome.swift b/hands/Sources/byre-hands/Chrome.swift index 115d072..3c119a4 100644 --- a/hands/Sources/byre-hands/Chrome.swift +++ b/hands/Sources/byre-hands/Chrome.swift @@ -728,5 +728,8 @@ func chromeTool(_ args: [String: Any]) async throws -> [Content] { let shot = try await c.send("Page.captureScreenshot", ["format": "png"], session: t.session) let (png, shotNote) = try shrink(Data(base64Encoded: shot.string("data") ?? "") ?? Data(), maxEdge) log.append(shotNote) + if args.bool("save") { + log.append("saved to \(try save(png))") + } return [.text(log.joined(separator: "\n")), .png(png)] } diff --git a/hands/Sources/byre-hands/Screen.swift b/hands/Sources/byre-hands/Screen.swift index 7089df5..64334cc 100644 --- a/hands/Sources/byre-hands/Screen.swift +++ b/hands/Sources/byre-hands/Screen.swift @@ -139,10 +139,14 @@ func captureTool(_ args: [String: Any]) async throws -> [Content] { guard args.bool("save") else { return [.text(shot.note), .png(shot.png)] } + return [.text("\(shot.note); saved to \(try save(shot.png))"), .png(shot.png)] +} + +func save(_ png: Data) throws -> String { guard let dir = mediaDir else { throw ToolError("save needs byre's media folder, and this byre-hands has none") } let url = URL(filePath: dir).appending(path: "capture-\(UUID().uuidString.prefix(8).lowercased()).png") - try shot.png.write(to: url) - return [.text("\(shot.note); saved to \(url.path)"), .png(shot.png)] + try png.write(to: url) + return url.path } diff --git a/hands/Sources/byre-hands/Tools.swift b/hands/Sources/byre-hands/Tools.swift index 2c1d091..acd09b8 100644 --- a/hands/Sources/byre-hands/Tools.swift +++ b/hands/Sources/byre-hands/Tools.swift @@ -123,7 +123,8 @@ let tools: [Tool] = [ "wait_ms": {"type": "integer", "description": "Wait this long after the op, before wait_for and capture."}, "timeout_ms": {"type": "integer", "description": "How long navigate waits for the load (default 15000) and wait_for waits (default 10000). Giving up is reported, not an error."}, "capture": {"type": "boolean", "description": "Finish with a screenshot of the tab's page."}, - "max_edge": {"type": "number", "description": "Longest edge of the screenshot in pixels, 16 to 8192 (default 1568, which is as much as you can see)."} + "max_edge": {"type": "number", "description": "Longest edge of the screenshot in pixels, 16 to 8192 (default 1568, which is as much as you can see)."}, + "save": {"type": "boolean", "description": "With capture, also save the screenshot as a PNG in the task's media folder; the reply gives its path, which a Markdown image in your final message can show."} }, "required": ["op"]} """, run: chromeTool