Fail hands accessibility reads fast when an app stops answering

docs/operations.md

@@ -134,7 +134,7 @@ Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`hand
 - `browser`: Orion through AppleScript: list tabs, navigate (then poll `document.readyState`), run JavaScript in a tab (`do JavaScript`), select and close tabs, with `wait_for` (the tab's `document.title` and `document.body.innerText`, ignoring case) and capture. Capturing a call that names a tab selects that tab first, since only the selected tab is on screen, and waits 300ms for it to draw. The Orion window is found by its AppleScript `id`, which is its window server window id. Durations (`wait_ms`, `timeout_ms`, `act`'s `wait` steps) must be 0 to 600000 ms. `do JavaScript` needs Orion's Develop → "Allow JavaScript from Apple Events".
 - `chrome`: Google Chrome through the DevTools protocol (`Chrome.swift`). The first call finds Chrome by `GET http://127.0.0.1:9222/json/version`; when nothing answers it runs `open -na "Google Chrome" --args --user-data-dir=~/Library/Application Support/byre/chrome --remote-debugging-port=9222 --no-first-run --no-default-browser-check` and polls for 15s. Chrome refuses remote debugging on its default profile, so byre's Chrome is its own profile, a separate Chrome beside any the human runs, and it keeps its logins: the human signs in there (passkeys included) and every later call, from any agent on that Mac, uses the session. The process holds one WebSocket to the browser endpoint for its life, reconnecting (and relaunching Chrome) on the next call after it drops, and attaches to each tab it touches with `Target.attachToTarget` (flattened sessions), enabling `Page`, `Network`, `Runtime` and `Log` there; concurrent calls for a tab share one attachment, and none uses the tab until all four are enabled. From then on it keeps that tab's requests (the last 5000) and console messages, exceptions and log entries (the last 2000) in memory; `navigate` clears both, `network` and `console` read them (each redirect hop is its own row, and the raw headers of `requestWillBeSentExtraInfo` and `responseReceivedExtraInfo`, which carry cookies, are paired with a request's hops in order, as DevTools does, with a hop whose response says `hasExtraInfo` (or `redirectHasExtraInfo`) false, such as a cached redirect, holding an empty place), and a request's body comes from `Network.getResponseBody` when asked for, shown up to 64 KiB. `js` is `Runtime.evaluate` with `awaitPromise` and `replMode`. `trace_start` and `trace_stop` run `Tracing` on the tab's session with the DevTools timeline categories, stream the trace to `byre-trace-<random>.json` in the temporary directory (a client's own, on a client Mac) and summarise it from the trace events: FCP, LCP, DOMContentLoaded and load of the tab's main frame (whose trace frame id is the tab's target id, so other tabs and iframes don't count) after its last `navigationStart`, and the `RunTask`s of 50ms or more on its `CrRendererMain` thread, each with the longest script, parse, style, layout, paint or GC event inside it. `cdp` sends any method to the tab's session. Capture is `Page.bringToFront` then `Page.captureScreenshot`, scaled to `max_edge`. It drives no mouse or keyboard, but `select`, `close`, `navigate` with `new_tab`, capture, starting Chrome and any `cdp` method (some, such as `Page.bringToFront`, do) can change which page is in front, so those calls hold the input lock (see `act`) for the whole call, and starting Chrome holds it while it waits for the port; the other ops take no lock. Chrome comes from `brew install --cask google-chrome`; macOS's "downloaded from the Internet" prompt on its first launch must be answered Open (or `osascript -e 'tell application "System Events" to tell process "CoreServicesUIAgent" to click button "Open" of window 1'`), and until it is, Chrome runs without opening its port.
 
-`byre-hands` reads stdin on a background thread and starts each request as its own detached task as soon as it arrives, so one call stuck on an app doesn't hold up the others; responses are written whole, one line each, under a lock, in the order they finish. Ref names come from one counter for the life of the process, so no two trees ever share a name, even trees built at the same time; the ref table is swapped in under a lock once a tree is complete, so a ref from any tree but the last published one fails as unknown instead of naming another tree's element. Each `act` call takes the table as it stands when the call arrives and resolves every step's refs from that copy, so a tree finishing during its waits can't point a ref at a different element. AppleScript needs the main thread with a run loop, so every script runs there (`osa` in `Browser.swift`, `DispatchQueue.main.sync`, while the main thread runs `RunLoop.main`), one at a time, wrapped in `with timeout of 10 seconds`; `NSApplication.shared` connects the process to the window server at startup, without which ScreenCaptureKit asserts. An app that is in a menu-tracking loop or otherwise busy answers neither Apple events nor accessibility requests: a script fails with AppleScript's timeout error after 10s, and accessibility requests time out after 3s (the messaging timeout set on the system-wide element at startup), at which point a tree walk or `wait_for` stops with an error saying the app isn't answering instead of timing out on every element.
+`byre-hands` reads stdin on a background thread and starts each request as its own detached task as soon as it arrives, so one call stuck on an app doesn't hold up the others; responses are written whole, one line each, under a lock, in the order they finish. Ref names come from one counter for the life of the process, so no two trees ever share a name, even trees built at the same time; the ref table is swapped in under a lock once a tree is complete, so a ref from any tree but the last published one fails as unknown instead of naming another tree's element. Each `act` call takes the table as it stands when the call arrives and resolves every step's refs from that copy, so a tree finishing during its waits can't point a ref at a different element. AppleScript needs the main thread with a run loop, so every script runs there (`osa` in `Browser.swift`, `DispatchQueue.main.sync`, while the main thread runs `RunLoop.main`), one at a time, wrapped in `with timeout of 10 seconds`; `NSApplication.shared` connects the process to the window server at startup, without which ScreenCaptureKit asserts. An app that is in a menu-tracking loop or otherwise busy answers neither Apple events nor accessibility requests: a script fails with AppleScript's timeout error after 10s, and accessibility requests time out after 3s (the messaging timeout set on the system-wide element at startup), and every accessibility read (`attr` in `AX.swift`, and the action-name and settable checks in the tree walk) throws on the first timeout, so a tree walk, `wait_for`, a ref's position or the front-window check stops with an error saying the app isn't answering instead of waiting 3s on every attribute of every element.
 
 **Permissions.** Agents' processes, `byre-hands` included, run as byre's descendants, so macOS privacy controls (TCC) hold byre responsible for them: the Screen Recording, Accessibility and Automation grants in System Settings → Privacy & Security are byre's, made by clicking at the Mac's screen, and each app scripted for the first time asks for its own Automation grant there. `ax_tree` and `act` check `AXIsProcessTrusted` first and fail saying byre lacks Accessibility when it isn't granted, since an untrusted process otherwise just sees no windows and no menu bar. Accessibility can list byre switched off; only its switch being on counts. `codesign` gives byre the identifier `run.fc.byre` and the Apple Development certificate, which TCC keys grants on, so they carry over from build to build. `codesign -dv ~/bin/byre` shows the signature the running binary has.
 

hands/Sources/byre-hands/AX.swift

@@ -40,16 +40,22 @@ let shownActions: [String: String] = [
 
 let maxVisits = 30000
 
-func attr(_ e: AXUIElement, _ name: String) -> CFTypeRef? {
+let notAnswering = ToolError("the app isn't answering accessibility requests; it may be busy or have a menu open (capture it to see, and escape closes a menu)")
+
+func attr(_ e: AXUIElement, _ name: String) throws -> CFTypeRef? {
     var v: CFTypeRef?
-    if AXUIElementCopyAttributeValue(e, name as CFString, &v) != .success {
+    let err = AXUIElementCopyAttributeValue(e, name as CFString, &v)
+    if err == .cannotComplete {
+        throw notAnswering
+    }
+    if err != .success {
         return nil
     }
     return v
 }
 
-func text(_ e: AXUIElement, _ name: String) -> String {
+func text(_ e: AXUIElement, _ name: String) throws -> String {
-    guard let v = attr(e, name) else {
+    guard let v = try attr(e, name) else {
         return ""
     }
     if let s = v as? String {
@@ -64,8 +70,8 @@ func text(_ e: AXUIElement, _ name: String) -> String {
     return ""
 }
 
-func children(_ e: AXUIElement) -> [AXUIElement] {
+func children(_ e: AXUIElement) throws -> [AXUIElement] {
-    return (attr(e, kAXChildrenAttribute) as? [AXUIElement]) ?? []
+    return (try attr(e, kAXChildrenAttribute) as? [AXUIElement]) ?? []
 }
 
 func appElement(_ pid: pid_t) -> AXUIElement {
@@ -79,17 +85,13 @@ func requireTrust() throws {
 }
 
 func axRole(_ e: AXUIElement) throws -> String {
-    var v: CFTypeRef?
+    return try attr(e, kAXRoleAttribute) as? String ?? ""
-    if AXUIElementCopyAttributeValue(e, kAXRoleAttribute as CFString, &v) == .cannotComplete {
-        throw ToolError("the app isn't answering accessibility requests; it may be busy or have a menu open (capture it to see, and escape closes a menu)")
-    }
-    return v as? String ?? ""
 }
 
 func axWindow(_ id: Int) throws -> (WindowInfo, AXUIElement) {
     try requireTrust()
     let info = try window(id)
-    for w in (attr(appElement(info.pid), kAXWindowsAttribute) as? [AXUIElement]) ?? [] {
+    for w in (try attr(appElement(info.pid), kAXWindowsAttribute) as? [AXUIElement]) ?? [] {
         var wid: CGWindowID = 0
         if _AXUIElementGetWindow(w, &wid) == .success && Int(wid) == id {
             return (info, w)
@@ -110,8 +112,8 @@ func element(_ ref: String, in info: WindowInfo, _ table: RefTable) throws -> AX
     return e
 }
 
-func frame(_ e: AXUIElement) -> CGRect? {
+func frame(_ e: AXUIElement) throws -> CGRect? {
-    guard let p = attr(e, kAXPositionAttribute), let s = attr(e, kAXSizeAttribute) else {
+    guard let p = try attr(e, kAXPositionAttribute), let s = try attr(e, kAXSizeAttribute) else {
         return nil
     }
     var pt = CGPoint.zero
@@ -153,23 +155,27 @@ func axTree(_ root: AXUIElement, _ source: RefSource, _ opts: TreeOptions) throw
         let role = try axRole(e)
         var parts: [String] = []
         for name in [kAXTitleAttribute, kAXDescriptionAttribute, kAXValueAttribute, kAXPlaceholderValueAttribute] {
-            let s = text(e, name)
+            let s = try text(e, name)
             if !s.isEmpty && !parts.contains(s) {
                 parts.append(s)
             }
         }
         var names: CFArray?
-        AXUIElementCopyActionNames(e, &names)
+        if AXUIElementCopyActionNames(e, &names) == .cannotComplete {
+            throw notAnswering
+        }
         var acts = ((names as? [String]) ?? []).compactMap { shownActions[$0] }
         var settable: DarwinBoolean = false
-        AXUIElementIsAttributeSettable(e, kAXValueAttribute as CFString, &settable)
+        if AXUIElementIsAttributeSettable(e, kAXValueAttribute as CFString, &settable) == .cannotComplete {
+            throw notAnswering
+        }
         if settable.boolValue {
             acts.append("set")
         }
-        if (attr(e, kAXFocusedAttribute) as? Bool) == true {
+        if (try attr(e, kAXFocusedAttribute) as? Bool) == true {
             acts.append("focused")
         }
-        if (attr(e, kAXEnabledAttribute) as? Bool) == false {
+        if (try attr(e, kAXEnabledAttribute) as? Bool) == false {
             acts.append("disabled")
         }
         let echo = role == kAXStaticTextRole && acts.isEmpty && parts.allSatisfy { p in parentParts.contains { $0.contains(p) } }
@@ -186,7 +192,7 @@ func axTree(_ root: AXUIElement, _ source: RefSource, _ opts: TreeOptions) throw
             if !acts.isEmpty {
                 line += " [" + acts.joined(separator: " ") + "]"
             }
-            if opts.frames, let f = frame(e) {
+            if opts.frames, let f = try frame(e) {
                 line += " @\(Int(f.minX - opts.origin.x)),\(Int(f.minY - opts.origin.y)) \(Int(f.width))x\(Int(f.height))"
             }
             lines.append(line)
@@ -195,7 +201,7 @@ func axTree(_ root: AXUIElement, _ source: RefSource, _ opts: TreeOptions) throw
         if depth >= opts.depth {
             return
         }
-        for c in children(e) {
+        for c in try children(e) {
             try walk(c, depth + 1, next, parts.isEmpty ? parentParts : parts)
         }
     }
@@ -228,7 +234,7 @@ func axTreeTool(_ args: [String: Any]) async throws -> [Content] {
         }
         let app = appElement(p)
         _ = try axRole(app)
-        guard let bar = attr(app, kAXMenuBarAttribute) else {
+        guard let bar = try attr(app, kAXMenuBarAttribute) else {
             throw ToolError("process \(pid) has no menu bar")
         }
         return [.text(try axTree(bar as! AXUIElement, RefSource(pid: p, window: nil), treeOptions(args, origin: .zero)))]
@@ -248,10 +254,10 @@ func windowMatches(_ root: AXUIElement, _ needle: String) throws -> Int {
             return
         }
         _ = try axRole(e)
-        if [kAXTitleAttribute, kAXDescriptionAttribute, kAXValueAttribute].contains(where: { text(e, $0).lowercased().contains(n) }) {
+        if try [kAXTitleAttribute, kAXDescriptionAttribute, kAXValueAttribute].contains(where: { try text(e, $0).lowercased().contains(n) }) {
             count += 1
         }
-        for c in children(e) {
+        for c in try children(e) {
             try walk(c)
         }
     }

hands/Sources/byre-hands/Act.swift

@@ -3,7 +3,7 @@ import ApplicationServices
 
 func point(_ v: Any?, _ info: WindowInfo, _ table: RefTable) throws -> CGPoint {
     if let ref = v as? String {
-        guard let f = frame(try element(ref, in: info, table)) else {
+        guard let f = try frame(try element(ref, in: info, table)) else {
             throw ToolError("\(ref) has no position on screen")
         }
         return CGPoint(x: f.midX, y: f.midY)

hands/Sources/byre-hands/Input.swift

@@ -65,9 +65,9 @@ func post(_ e: CGEvent?) {
     e?.post(tap: .cghidEventTap)
 }
 
-func isFront(_ info: WindowInfo) -> Bool {
+func isFront(_ info: WindowInfo) throws -> Bool {
     guard NSWorkspace.shared.frontmostApplication?.processIdentifier == info.pid,
-          let focused = attr(appElement(info.pid), kAXFocusedWindowAttribute)
+          let focused = try attr(appElement(info.pid), kAXFocusedWindowAttribute)
     else {
         return false
     }
@@ -77,14 +77,14 @@ func isFront(_ info: WindowInfo) -> Bool {
 
 func focusWindow(_ id: Int, _ w: AXUIElement) async throws -> WindowInfo {
     let info = try window(id)
-    if isFront(info) {
+    if try isFront(info) {
         return info
     }
     NSRunningApplication(processIdentifier: info.pid)?.activate()
     AXUIElementPerformAction(w, kAXRaiseAction as CFString)
     await pause(150)
     let now = try window(id)
-    if !isFront(now) {
+    if try !isFront(now) {
         throw ToolError("could not bring window \(id) to the front (\(NSWorkspace.shared.frontmostApplication?.localizedName ?? "nothing") is frontmost), so no input was sent")
     }
     return now