ai/byre
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Add byre-hands: screen capture and app control tools for agents

Browse Source
This commit is contained in:
Ian Gulliver
2026-10-17 22:43:38 -07:00
parent 658f78934c
commit bb908e493f
14 changed files with 1101 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations.md
+19 -1
View File
@@ -12,6 +12,7 @@
- `internal/codex`: the `codex app-server` JSON-RPC client, with types generated from the CLI's schema by `go generate ./internal/codex` (see `docs/codex-protocol.md`). One goroutine reads the CLI's stdout and blocks on the bounded `Events` channel when its consumer falls behind, so responses queued behind those notifications wait too: the goroutine draining `Events` (`codexProcess.drive`) therefore never waits on a reply itself. It writes `turn/steer` in order and waits for the reply on another goroutine, and starts the prose-question nudge and the turn-end `Deliver` on their own goroutines. Every request gives up after `rpcTimeout`, or the caller's earlier deadline, and that deadline covers waiting for the write lock, writing the request to stdin and waiting for the reply. Stdin is an `os.Pipe`, so writes take a deadline. A request whose deadline passes before any of it is written fails and leaves the CLI running. One that times out after part of its line is written has broken the framing, so byre kills the CLI, logging `codex stopped reading its stdin mid-line`, and the session ends as any exited process does.
- `internal/tailscale`: `tailscale serve` for task previews (see "Previews").
- `internal/workflow`: validating workflow definitions and reading and writing their YAML files (see "Workflows").
- `hands/` is `byre-hands`, the Swift MCP server that lets task agents see and drive the Mac byre runs on (see "Screen and app control").
- `mac/` is the macOS app, byre's client; `docs/macos.md` describes how it is built, and "macOS app" below covers building, installing and its storage. It connects to byre's `tailscale serve` `https` origin, typed into its connection screen. The server serves only the API; it has no web pages.
## Data
@@ -99,6 +100,9 @@ The listen address is `byre serve --listen`, whose default is in `cmd/byre/serve
```
go install ./cmd/byre
codesign --force --sign "Apple Development" --identifier run.fc.byre ~/bin/byre
swift build --package-path hands --configuration release
install hands/.build/release/byre-hands ~/bin/
cp deploy/run.fc.byre.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/run.fc.byre.plist
tailscale serve --bg --https=443 http://<listen address>
@@ -106,7 +110,7 @@ tailscale serve --bg --https=443 http://<listen address>
byre is served at the root of the machine's `ts.net` name on 443 (`https://<machine>.<tailnet>.ts.net`), not under a path: the API uses absolute paths, and the macOS app takes an origin with no path. `tailscale serve status` shows the URL and what else the machine serves.
After rebuilding: `go install ./cmd/byre && launchctl kickstart -k gui/$(id -u)/run.fc.byre`. Logs go to `~/Library/Logs/byre.log`.
After rebuilding: step 1 of "Deploying" in `AGENTS.md`. Logs go to `~/Library/Logs/byre.log`.
Clients reach `tailscale serve` over HTTP/2; it proxies to byre over HTTP/1.1 and passes SSE through unbuffered.
@@ -118,6 +122,20 @@ Clients reach `tailscale serve` over HTTP/2; it proxies to byre over HTTP/1.1 an
**Storage.** The app keeps three kinds of record in SwiftData (`mac/Sources/Byre/Persistence.swift`): the server origin, when each task was last selected, and composer and answer drafts as JSON, one record per server and login (`docs/macos.md`, "Persistence" and "Drafts"). The store is `~/Library/Application Support/run.fc.byre.mac/byre.store`, with its `-shm` and `-wal` files and CloudKit's working files beside it in that directory; the app isn't sandboxed, so it names its own file rather than SwiftData's default `default.store`, which every unsandboxed app shares. The store syncs through the iCloud private database of container `iCloud.run.fc.byre.mac`, in CloudKit's Development environment; the CloudKit console (icloud.developer.apple.com, the container, Development, Private Database, zone `com.apple.coredata.cloudkit.zone`) shows the records. Deleting the local files while the app isn't running makes the next launch download the records again from iCloud; deleting the records in the console as well starts from the connection screen. Pasted images stay in memory and are dropped on Change Server… and quit. The app logs through `os.Logger`: `log show --last 10m --predicate 'subsystem == "run.fc.byre.mac"'` prints it.
## Screen and app control
Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`handsMCP` in `internal/agents/hands.go`), passed the same way as reviewers' Playwright server (see "Reviews"); reviewers and lead agents don't get it. It is a Swift executable built from `hands/` and installed in `~/bin`, which is on the service's `PATH`, and each session runs its own copy, so a hung or crashed one affects only that agent. It speaks MCP's JSON-RPC over stdin and stdout, one message per line, and returns captures as inline PNG image content. Its tools, whose descriptions in `hands/Sources/byre-hands/Tools.swift` are what agents read:
- `screen_list`: running apps, on-screen windows (front to back) and displays.
- `capture`: one window, or a display, through ScreenCaptureKit. A window is captured as its rectangle of its display with only its own app's windows included, so other apps covering it don't hide it and its menus, popups and sheets show; capturing the window alone fails while a popup is attached to it. The image is scaled to `max_edge` (default 1568, as much as a model sees; 16 to 8192) and the reply gives the pixel-to-point scale.
- `ax_tree`: a window's or an app's menu bar's accessibility tree, one line per element with text or actions, each with a ref (`e12`) that `act` takes. Refs live in the process and each tree replaces them. They remember the window, or the app whose menu bar, they came from, and `act` refuses a ref that isn't from its window or its app's menu bar. Unlabelled containers are left out with their children shown in their place, and static text already contained in its labelled ancestor's text is dropped.
- `act`: a list of steps in one window: accessibility actions on refs (press, set value, focus, confirm, any named action), then real mouse and keyboard events posted with `CGEvent` (click, double and right click, drag, scroll, typing, key combinations), Before each of those steps it reads the window's current frame, for `[x, y]`, and makes sure the window is in front: its app frontmost and the app's focused window (through accessibility, so the app's own popups don't count against it). When it isn't, it activates the app and raises the window, and stops with an error, sending nothing, if the window still isn't in front. Every session's `byre-hands` drives the same desktop, so a call that can change what is on screen or focused holds an exclusive `flock` on `byre-hands-input.lock` in the user's temporary directory (`$TMPDIR`): an `act` call with steps and every `browser` call except a `tabs` without capture, each for the whole call, its waiting, tree and capture included, so what it returns shows its own work. Another such call waits up to 30s for it, then fails without doing anything. The lock is released when the steps end or the process exits. Typing posts each ASCII character as its key code, with shift where needed, since Orion ignores key events that carry only a Unicode string. Afterwards it can wait (`wait_ms`, then `wait_for` text in the window's tree) and return the tree and a capture, so one call clicks, waits and looks.
- `browser`: Orion through AppleScript: list tabs, navigate (then poll `document.readyState`), run JavaScript in a tab (`do JavaScript`), select and close tabs, with the same `wait_for` (page text) and capture. Capturing a call that names a tab selects that tab first, since only the selected tab is on screen, and waits 300ms for it to draw. The Orion window is found by its AppleScript `id`, which is its window server window id. Durations (`wait_ms`, `timeout_ms`, `act`'s `wait` steps) must be 0 to 600000 ms. `do JavaScript` needs Orion's Develop → "Allow JavaScript from Apple Events".
AppleScript needs the main thread with a run loop, so `byre-hands` reads stdin on a background thread and runs each request on the main actor while the main thread runs `RunLoop.main`; `NSApplication.shared` connects it to the window server at startup, without which ScreenCaptureKit asserts.
**Permissions.** Agents' processes, `byre-hands` included, run as byre's descendants, so macOS privacy controls (TCC) hold byre responsible for them: the Screen Recording, Accessibility and Automation grants in System Settings → Privacy & Security are byre's, made by clicking at the Mac's screen, and each app scripted for the first time asks for its own Automation grant there. `codesign` gives byre the identifier `run.fc.byre` and the Apple Development certificate, which TCC keys grants on, so they carry over from build to build. `codesign -dv ~/bin/byre` shows the signature the running binary has.
## Previews
An agent shows the human a server it runs with the `expose_port` MCP tool (`internal/agents/preview.go`): `name` labels it, `port` is where it listens. The agent starts the server itself, usually as a background command; byre doesn't supervise it, so it dies with the agent's `claude` process (a byre restart, compaction, the task closing) and the agent starts it again when asked. The call is refused when `port` is outside the task's block or nothing accepts a connection on `127.0.0.1:<port>`. Otherwise byre runs `tailscale serve --bg --https=<port> http://127.0.0.1:<port>` (`internal/tailscale`, which runs the macOS CLI at `/Applications/Tailscale.app/Contents/MacOS/Tailscale`, since byre's environment has no `tailscale` on its `PATH`, with `TAILSCALE_BE_CLI=1`, without which the app executable launched from the service exits 0 without running the subcommand), then reads `tailscale serve status --json` and fails the call if no `Web` host for that port is listed, so the server is at `https://<byre's ts.net name>:<port>/` with Tailscale's identity headers, and records a row in the `previews` table (task, name, port, time) with a `preview.exposed` event, which the transcript shows as a note linking the URL. Each port gets its own HTTPS port rather than a path under byre's, since path-prefix proxying breaks apps that use absolute paths. Exposing a name or port the task already has replaces that row, and a port it no longer uses is turned off with `tailscale serve --https=<port> off`. A server must listen on `127.0.0.1`: while `tailscale serve` holds a port, binding `0.0.0.0` on it fails with "address already in use". When the task is merged or discarded, byre turns off each of its ports (`Manager.Unexpose`); the rows stay. `GET /api/tasks/{taskId}/previews` lists a task's rows by name, and `tailscale serve status` shows what is served.