hands test
This commit is contained in:
+2
-2
@@ -132,9 +132,9 @@ Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`hand
|
||||
- `act`: a list of steps in one window: accessibility actions on refs (press, set value, focus, confirm, any named action), then real mouse and keyboard events posted with `CGEvent` (click, double and right click, drag, scroll, typing, key combinations), Before each of those steps it reads the window's current frame, for `[x, y]`, and makes sure the window is in front: its app frontmost and the app's focused window (through accessibility, so the app's own popups don't count against it). When it isn't, it activates the app and raises the window, and stops with an error, sending nothing, if the window still isn't in front. Every session's `byre-hands` drives the same desktop, so a call that can change what is on screen or focused holds an exclusive `flock` on `byre-hands-input.lock` in the user's temporary directory (`$TMPDIR`): an `act` call with steps and every `browser` call except a `tabs` without capture, each for the whole call, its waiting, tree and capture included, so what it returns shows its own work. Another such call waits up to 30s for it, then fails without doing anything. The lock is released when the steps end or the process exits. Typing posts each ASCII character as its key code, with shift where needed, since Orion ignores key events that carry only a Unicode string. Afterwards it can wait (`wait_ms`, then `wait_for` text in the window's tree) and return the tree and a capture, so one call clicks, waits and looks.
|
||||
- `browser`: Orion through AppleScript: list tabs, navigate (then poll `document.readyState`), run JavaScript in a tab (`do JavaScript`), select and close tabs, with the same `wait_for` (page text) and capture. Capturing a call that names a tab selects that tab first, since only the selected tab is on screen, and waits 300ms for it to draw. The Orion window is found by its AppleScript `id`, which is its window server window id. Durations (`wait_ms`, `timeout_ms`, `act`'s `wait` steps) must be 0 to 600000 ms. `do JavaScript` needs Orion's Develop → "Allow JavaScript from Apple Events".
|
||||
|
||||
AppleScript needs the main thread with a run loop, so `byre-hands` reads stdin on a background thread and runs each request on the main actor while the main thread runs `RunLoop.main`; `NSApplication.shared` connects it to the window server at startup, without which ScreenCaptureKit asserts.
|
||||
`byre-hands` reads stdin on a background thread and starts each request as its own detached task as soon as it arrives, so one call stuck on an app doesn't hold up the others; responses are written whole, one line each, under a lock, in the order they finish. Ref names come from one counter for the life of the process, so no two trees ever share a name, even trees built at the same time; the ref table is swapped in under a lock once a tree is complete, so a ref from any tree but the last published one fails as unknown instead of naming another tree's element. Each `act` call takes the table as it stands when the call arrives and resolves every step's refs from that copy, so a tree finishing during its waits can't point a ref at a different element. AppleScript needs the main thread with a run loop, so every script runs there (`osa` in `Browser.swift`, `DispatchQueue.main.sync`, while the main thread runs `RunLoop.main`), one at a time, wrapped in `with timeout of 10 seconds`; `NSApplication.shared` connects the process to the window server at startup, without which ScreenCaptureKit asserts. An app that is in a menu-tracking loop or otherwise busy answers neither Apple events nor accessibility requests: a script fails with AppleScript's timeout error after 10s, and accessibility requests time out after 3s (the messaging timeout set on the system-wide element at startup), at which point a tree walk or `wait_for` stops with an error saying the app isn't answering instead of timing out on every element.
|
||||
|
||||
**Permissions.** Agents' processes, `byre-hands` included, run as byre's descendants, so macOS privacy controls (TCC) hold byre responsible for them: the Screen Recording, Accessibility and Automation grants in System Settings → Privacy & Security are byre's, made by clicking at the Mac's screen, and each app scripted for the first time asks for its own Automation grant there. `codesign` gives byre the identifier `run.fc.byre` and the Apple Development certificate, which TCC keys grants on, so they carry over from build to build. `codesign -dv ~/bin/byre` shows the signature the running binary has.
|
||||
**Permissions.** Agents' processes, `byre-hands` included, run as byre's descendants, so macOS privacy controls (TCC) hold byre responsible for them: the Screen Recording, Accessibility and Automation grants in System Settings → Privacy & Security are byre's, made by clicking at the Mac's screen, and each app scripted for the first time asks for its own Automation grant there. `ax_tree` and `act` check `AXIsProcessTrusted` first and fail saying byre lacks Accessibility when it isn't granted, since an untrusted process otherwise just sees no windows and no menu bar. Accessibility can list byre switched off; only its switch being on counts. `codesign` gives byre the identifier `run.fc.byre` and the Apple Development certificate, which TCC keys grants on, so they carry over from build to build. `codesign -dv ~/bin/byre` shows the signature the running binary has.
|
||||
|
||||
## Previews
|
||||
|
||||
|
||||
Reference in New Issue
Block a user