From 1020d7eb1561f0af78924403e905ce8fc9ecfb49 Mon Sep 17 00:00:00 2001 From: Ian Gulliver Date: Sun, 18 Oct 2026 07:47:10 -0700 Subject: [PATCH] browser integration --- docs/operations.md | 1 + hands/Sources/byre-hands/Chrome.swift | 732 ++++++++++++++++++++++++++ hands/Sources/byre-hands/Tools.swift | 32 ++ internal/agents/clients.go | 9 +- 4 files changed, 770 insertions(+), 4 deletions(-) create mode 100644 hands/Sources/byre-hands/Chrome.swift diff --git a/docs/operations.md b/docs/operations.md index d63a0f0..258eebe 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -132,6 +132,7 @@ Implementer sessions get `byre-hands` as a stdio MCP server named `hands` (`hand - `ax_tree`: a window's or an app's menu bar's accessibility tree, one line per element with text or actions, each with a ref (`e12`) that `act` takes. Refs live in the process and each tree replaces them. They remember the window, or the app whose menu bar, they came from, and `act` refuses a ref that isn't from its window or its app's menu bar. Unlabelled containers are left out with their children shown in their place, and static text already contained in its labelled ancestor's text is dropped. - `act`: a list of steps in one window: accessibility actions on refs (press, set value, focus, confirm, any named action), then real mouse and keyboard events posted with `CGEvent` (click, double and right click, drag, scroll, typing, key combinations), Before each of those steps it reads the window's current frame, for `[x, y]`, and makes sure the window is in front: its app frontmost and the app's focused window (through accessibility, so the app's own popups don't count against it). When it isn't, it activates the app and raises the window, and stops with an error, sending nothing, if the window still isn't in front. Every session's `byre-hands` drives the same desktop, so a call that can change what is on screen or focused holds an exclusive `flock` on `byre-hands-input.lock` in the user's temporary directory (`$TMPDIR`): an `act` call with steps and every `browser` call except a `tabs` without capture, each for the whole call, its waiting, tree and capture included, so what it returns shows its own work. Another such call waits up to 30s for it, then fails without doing anything. The lock is released when the steps end or the process exits. Typing posts each ASCII character as its key code, with shift where needed, since Orion ignores key events that carry only a Unicode string. Afterwards it can wait (`wait_ms`, then `wait_for`) and return the tree and a capture, so one call clicks, waits and looks. `wait_for` counts the window's elements whose title, description or value contains the text, ignoring case, once before the steps and then every 200ms, and ends when the count rises above the first one (above zero with no steps), so text already on screen, such as another tab's title, doesn't end the wait. Some apps list `confirm` but ignore it, Orion's address bar among them; the tool description says to send Return instead. - `browser`: Orion through AppleScript: list tabs, navigate (then poll `document.readyState`), run JavaScript in a tab (`do JavaScript`), select and close tabs, with `wait_for` (the tab's `document.title` and `document.body.innerText`, ignoring case) and capture. Capturing a call that names a tab selects that tab first, since only the selected tab is on screen, and waits 300ms for it to draw. The Orion window is found by its AppleScript `id`, which is its window server window id. Durations (`wait_ms`, `timeout_ms`, `act`'s `wait` steps) must be 0 to 600000 ms. `do JavaScript` needs Orion's Develop → "Allow JavaScript from Apple Events". +- `chrome`: Google Chrome through the DevTools protocol (`Chrome.swift`). The first call finds Chrome by `GET http://127.0.0.1:9222/json/version`; when nothing answers it runs `open -na "Google Chrome" --args --user-data-dir=~/Library/Application Support/byre/chrome --remote-debugging-port=9222 --no-first-run --no-default-browser-check` and polls for 15s. Chrome refuses remote debugging on its default profile, so byre's Chrome is its own profile, a separate Chrome beside any the human runs, and it keeps its logins: the human signs in there (passkeys included) and every later call, from any agent on that Mac, uses the session. The process holds one WebSocket to the browser endpoint for its life, reconnecting (and relaunching Chrome) on the next call after it drops, and attaches to each tab it touches with `Target.attachToTarget` (flattened sessions), enabling `Page`, `Network`, `Runtime` and `Log` there; concurrent calls for a tab share one attachment, and none uses the tab until all four are enabled. From then on it keeps that tab's requests (the last 5000) and console messages, exceptions and log entries (the last 2000) in memory; `navigate` clears both, `network` and `console` read them (each redirect hop is its own row, and the raw headers of `requestWillBeSentExtraInfo` and `responseReceivedExtraInfo`, which carry cookies, are paired with a request's hops in order, as DevTools does, with a hop whose response says `hasExtraInfo` (or `redirectHasExtraInfo`) false, such as a cached redirect, holding an empty place), and a request's body comes from `Network.getResponseBody` when asked for, shown up to 64 KiB. `js` is `Runtime.evaluate` with `awaitPromise` and `replMode`. `trace_start` and `trace_stop` run `Tracing` on the tab's session with the DevTools timeline categories, stream the trace to `byre-trace-.json` in the temporary directory (a client's own, on a client Mac) and summarise it from the trace events: FCP, LCP, DOMContentLoaded and load of the tab's main frame (whose trace frame id is the tab's target id, so other tabs and iframes don't count) after its last `navigationStart`, and the `RunTask`s of 50ms or more on its `CrRendererMain` thread, each with the longest script, parse, style, layout, paint or GC event inside it. `cdp` sends any method to the tab's session. Capture is `Page.bringToFront` then `Page.captureScreenshot`, scaled to `max_edge`. It drives no mouse or keyboard, but `select`, `close`, `navigate` with `new_tab`, capture, starting Chrome and any `cdp` method (some, such as `Page.bringToFront`, do) can change which page is in front, so those calls hold the input lock (see `act`) for the whole call, and starting Chrome holds it while it waits for the port; the other ops take no lock. Chrome comes from `brew install --cask google-chrome`; macOS's "downloaded from the Internet" prompt on its first launch must be answered Open (or `osascript -e 'tell application "System Events" to tell process "CoreServicesUIAgent" to click button "Open" of window 1'`), and until it is, Chrome runs without opening its port. `byre-hands` reads stdin on a background thread and starts each request as its own detached task as soon as it arrives, so one call stuck on an app doesn't hold up the others; responses are written whole, one line each, under a lock, in the order they finish. Ref names come from one counter for the life of the process, so no two trees ever share a name, even trees built at the same time; the ref table is swapped in under a lock once a tree is complete, so a ref from any tree but the last published one fails as unknown instead of naming another tree's element. Each `act` call takes the table as it stands when the call arrives and resolves every step's refs from that copy, so a tree finishing during its waits can't point a ref at a different element. AppleScript needs the main thread with a run loop, so every script runs there (`osa` in `Browser.swift`, `DispatchQueue.main.sync`, while the main thread runs `RunLoop.main`), one at a time, wrapped in `with timeout of 10 seconds`; `NSApplication.shared` connects the process to the window server at startup, without which ScreenCaptureKit asserts. An app that is in a menu-tracking loop or otherwise busy answers neither Apple events nor accessibility requests: a script fails with AppleScript's timeout error after 10s, and accessibility requests time out after 3s (the messaging timeout set on the system-wide element at startup), at which point a tree walk or `wait_for` stops with an error saying the app isn't answering instead of timing out on every element. diff --git a/hands/Sources/byre-hands/Chrome.swift b/hands/Sources/byre-hands/Chrome.swift new file mode 100644 index 0000000..115d072 --- /dev/null +++ b/hands/Sources/byre-hands/Chrome.swift @@ -0,0 +1,732 @@ +import AppKit +import ImageIO +import UniformTypeIdentifiers + +let chromePort = 9222 +let chromeProfile = FileManager.default.homeDirectoryForCurrentUser.appending(path: "Library/Application Support/byre/chrome").path + +final class Req { + let seq: Int + let id: String + let url: String + let method: String + let type: String + let start: Double + var end: Double? + var status: Int? + var proto = "" + var remote = "" + var mime = "" + var bytes = 0 + var error: String? + var cached = false + var hasPost = false + var redirected = false + var gotRespExtra = false + var reqHeaders: [String: Any] = [:] + var respHeaders: [String: Any] = [:] + + init(seq: Int, id: String, url: String, method: String, type: String, start: Double) { + self.seq = seq + self.id = id + self.url = url + self.method = method + self.type = type + self.start = start + } +} + +final class Chain { + var hops: [Req] = [] + var reqExtra: [[String: Any]?] = [] + var respExtra: [[String: Any]?] = [] + + func sync() { + for (i, hop) in hops.enumerated() { + if i < reqExtra.count, let h = reqExtra[i] { + hop.reqHeaders = h + } + if i < respExtra.count, let h = respExtra[i] { + hop.respHeaders = h + hop.gotRespExtra = true + } + } + } + + func noExtra(_ i: Int) { + if reqExtra.count == i { + reqExtra.append(nil) + } + if respExtra.count == i { + respExtra.append(nil) + } + } +} + +final class TabState { + let session: String + let target: String + var seq = 0 + var reqs: [Req] = [] + var chains: [String: Chain] = [:] + var console: [String] = [] + var traceStream: String? + + init(session: String, target: String) { + self.session = session + self.target = target + } + + func clear() { + seq = 0 + reqs = [] + chains = [:] + console = [] + } + + func chain(_ rid: String) -> Chain { + if let c = chains[rid] { + return c + } + let c = Chain() + chains[rid] = c + return c + } + + func last(_ rid: String) -> Req? { + return chains[rid]?.hops.last + } +} + +final class Chrome: @unchecked Sendable { + let ws: URLSessionWebSocketTask + let lock = NSLock() + var next = 0 + var pending: [Int: CheckedContinuation<[String: Any], Error>] = [:] + var tabs: [String: Task] = [:] + var sessions: [String: TabState] = [:] + var current: String? + var closed = false + + init(_ url: URL) { + ws = URLSession.shared.webSocketTask(with: url) + ws.maximumMessageSize = 1 << 28 + ws.resume() + receive() + } + + func receive() { + ws.receive { [self] result in + switch result { + case .failure(let e): + fail(e) + case .success(let msg): + switch msg { + case .string(let s): + dispatch(Data(s.utf8)) + case .data(let d): + dispatch(d) + @unknown default: + break + } + receive() + } + } + } + + func fail(_ e: Error) { + let waiting = lock.withLock { + closed = true + let p = pending + pending = [:] + return p + } + for c in waiting.values { + c.resume(throwing: ToolError("lost the connection to Chrome: \(e.localizedDescription)")) + } + } + + func resolve(_ id: Int, _ r: Result<[String: Any], Error>) { + guard let c = lock.withLock({ pending.removeValue(forKey: id) }) else { + return + } + c.resume(with: r) + } + + func send(_ method: String, _ params: [String: Any] = [:], session: String? = nil, timeout: Int = 30) async throws -> [String: Any] { + let id = lock.withLock { + next += 1 + return next + } + var msg: [String: Any] = ["id": id, "method": method, "params": params] + if let session { + msg["sessionId"] = session + } + let data = try JSONSerialization.data(withJSONObject: msg) + return try await withCheckedThrowingContinuation { c in + let dead = lock.withLock { + if closed { + return true + } + pending[id] = c + return false + } + if dead { + c.resume(throwing: ToolError("lost the connection to Chrome")) + return + } + ws.send(.string(String(decoding: data, as: UTF8.self))) { [self] err in + if let err { + resolve(id, .failure(err)) + } + } + Task { + await pause(timeout * 1000) + resolve(id, .failure(ToolError("\(method) got no answer from Chrome in \(timeout)s"))) + } + } + } + + func dispatch(_ data: Data) { + guard let m = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + return + } + if let id = m.int("id") { + if let err = m.dict("error") { + resolve(id, .failure(ToolError("Chrome: \(err.string("message") ?? "\(err)")"))) + } else { + resolve(id, .success(m.dict("result") ?? [:])) + } + return + } + let method = m.string("method") ?? "" + let p = m.dict("params") ?? [:] + lock.withLock { + if method == "Target.detachedFromTarget", let s = p.string("sessionId"), let t = sessions.removeValue(forKey: s) { + tabs.removeValue(forKey: t.target) + return + } + guard let s = m.string("sessionId"), let t = sessions[s] else { + return + } + event(method, p, t) + } + } + + func event(_ method: String, _ p: [String: Any], _ t: TabState) { + let rid = p.string("requestId") ?? "" + switch method { + case "Network.requestWillBeSent": + let r = p.dict("request") ?? [:] + let chain = t.chain(rid) + if let prev = chain.hops.last, let redirect = p.dict("redirectResponse") { + respond(prev, redirect) + prev.redirected = true + prev.end = p.double("timestamp") + if !p.bool("redirectHasExtraInfo") { + chain.noExtra(chain.hops.count - 1) + } + } + t.seq += 1 + let req = Req(seq: t.seq, id: rid, url: r.string("url") ?? "", method: r.string("method") ?? "", type: p.string("type") ?? "", start: p.double("timestamp") ?? 0) + req.reqHeaders = r.dict("headers") ?? [:] + req.hasPost = r.bool("hasPostData") + t.reqs.append(req) + chain.hops.append(req) + chain.sync() + if t.reqs.count > 5000 { + for old in t.reqs.prefix(1000) where t.last(old.id) === old { + t.chains.removeValue(forKey: old.id) + } + t.reqs.removeFirst(1000) + } + case "Network.requestWillBeSentExtraInfo": + let chain = t.chain(rid) + chain.reqExtra.append(p.dict("headers") ?? [:]) + chain.sync() + case "Network.responseReceived": + guard let chain = t.chains[rid], let req = chain.hops.last else { + return + } + respond(req, p.dict("response") ?? [:]) + if !p.bool("hasExtraInfo") { + chain.noExtra(chain.hops.count - 1) + } + case "Network.responseReceivedExtraInfo": + let chain = t.chain(rid) + chain.respExtra.append(p.dict("headers") ?? [:]) + chain.sync() + case "Network.loadingFinished": + t.last(rid)?.end = p.double("timestamp") + t.last(rid)?.bytes = p.int("encodedDataLength") ?? 0 + case "Network.loadingFailed": + t.last(rid)?.end = p.double("timestamp") + t.last(rid)?.error = p.string("blockedReason").map { "blocked: \($0)" } ?? p.string("errorText") + case "Runtime.consoleAPICalled": + let args = (p["args"] as? [[String: Any]] ?? []).map(remoteText) + let frame = (p.dict("stackTrace")?["callFrames"] as? [[String: Any]])?.first + t.console.append(consoleLine(p.string("type") ?? "log", args.joined(separator: " "), frame?.string("url"), frame?.int("lineNumber"))) + case "Runtime.exceptionThrown": + let d = p.dict("exceptionDetails") ?? [:] + t.console.append(consoleLine("exception", d.dict("exception")?.string("description") ?? d.string("text") ?? "", d.string("url"), d.int("lineNumber"))) + case "Log.entryAdded": + let e = p.dict("entry") ?? [:] + t.console.append(consoleLine("\(e.string("level") ?? "info") (\(e.string("source") ?? ""))", e.string("text") ?? "", e.string("url"), e.int("lineNumber"))) + case "Tracing.tracingComplete": + t.traceStream = p.string("stream") ?? "" + default: + break + } + if t.console.count > 2000 { + t.console.removeFirst(500) + } + } + + func respond(_ req: Req, _ r: [String: Any]) { + req.status = r.int("status") + req.proto = r.string("protocol") ?? "" + req.remote = r.string("remoteIPAddress") ?? "" + req.mime = r.string("mimeType") ?? "" + req.cached = r.bool("fromDiskCache") || r.bool("fromServiceWorker") || r.bool("fromPrefetchCache") + if !req.gotRespExtra { + req.respHeaders = r.dict("headers") ?? [:] + } + } + + func pages() async throws -> [[String: Any]] { + let r = try await send("Target.getTargets") + return (r["targetInfos"] as? [[String: Any]] ?? []).filter { $0.string("type") == "page" } + } + + func tab(_ id: String?) async throws -> (String, TabState) { + var target = id + if target == nil { + let open = try await pages().compactMap { $0.string("targetId") } + target = lock.withLock { current }.flatMap { open.contains($0) ? $0 : nil } ?? open.first + } + guard let target else { + throw ToolError("Chrome has no tabs; navigate with new_tab to open one") + } + let task = lock.withLock { + current = target + if let a = tabs[target] { + return a + } + let a = Task { try await attach(target) } + tabs[target] = a + return a + } + do { + return (target, try await task.value) + } catch { + lock.withLock { + if tabs[target] == task { + tabs.removeValue(forKey: target) + } + } + throw error + } + } + + func attach(_ target: String) async throws -> TabState { + let r = try await send("Target.attachToTarget", ["targetId": target, "flatten": true]) + guard let s = r.string("sessionId") else { + throw ToolError("Chrome attached to \(target) without a session") + } + let t = TabState(session: s, target: target) + lock.withLock { sessions[s] = t } + for m in ["Page.enable", "Network.enable", "Runtime.enable", "Log.enable"] { + _ = try await send(m, session: s) + } + return t + } +} + +func remoteText(_ o: [String: Any]) -> String { + if let v = o["value"] { + if let s = v as? String { + return s + } + if let d = try? JSONSerialization.data(withJSONObject: v, options: [.fragmentsAllowed, .withoutEscapingSlashes]) { + return String(decoding: d, as: UTF8.self) + } + } + if let p = o.dict("preview"), let props = p["properties"] as? [[String: Any]] { + let array = p.string("subtype") == "array" + let items = props.map { array ? ($0.string("value") ?? "") : "\($0.string("name") ?? ""): \($0.string("value") ?? "")" }.joined(separator: ", ") + let more = p.bool("overflow") ? ", …" : "" + return array ? "[\(items)\(more)]" : "{\(items)\(more)}" + } + return o.string("unserializableValue") ?? o.string("description") ?? o.string("type") ?? "" +} + +func consoleLine(_ level: String, _ text: String, _ url: String?, _ line: Int?) -> String { + guard let url, !url.isEmpty else { + return "\(level) \(text)" + } + return "\(level) \(text) (\(url):\((line ?? 0) + 1))" +} + +func chromeVersion() async -> [String: Any]? { + var req = URLRequest(url: URL(string: "http://127.0.0.1:\(chromePort)/json/version")!) + req.timeoutInterval = 2 + guard let (d, _) = try? await URLSession.shared.data(for: req) else { + return nil + } + return try? JSONSerialization.jsonObject(with: d) as? [String: Any] +} + +func openChrome() async throws -> (Chrome, String) { + var note = "" + var info = await chromeVersion() + if info == nil { + let lock = try await lockInput() + defer { + close(lock) + } + let p = Process() + p.executableURL = URL(filePath: "/usr/bin/open") + p.arguments = ["-na", "Google Chrome", "--args", "--user-data-dir=\(chromeProfile)", "--remote-debugging-port=\(chromePort)", "--no-first-run", "--no-default-browser-check"] + try p.run() + p.waitUntilExit() + if p.terminationStatus != 0 { + throw ToolError("could not open Google Chrome (open exited \(p.terminationStatus)); is it installed on this Mac?") + } + let start = Date() + while info == nil, Date().timeIntervalSince(start) < 15 { + await pause(200) + info = await chromeVersion() + } + note = "started Chrome with byre's profile\n" + } + guard let ws = info?.string("webSocketDebuggerUrl").flatMap(URL.init(string:)) else { + throw ToolError("Chrome's debugging port \(chromePort) didn't answer within 15s") + } + return (Chrome(ws), note) +} + +actor ChromeLink { + var conn: Task<(Chrome, String), Error>? + + func get() async throws -> (Chrome, String) { + if let conn, let (c, _) = try? await conn.value, !c.lock.withLock({ c.closed }) { + return (c, "") + } + let t = Task { try await openChrome() } + conn = t + return try await t.value + } +} + +let chromeLink = ChromeLink() + +func evaluate(_ c: Chrome, _ t: TabState, _ code: String) async throws -> String { + let r = try await c.send("Runtime.evaluate", ["expression": code, "returnByValue": true, "awaitPromise": true, "replMode": true, "userGesture": true], session: t.session) + if let d = r.dict("exceptionDetails") { + throw ToolError(d.dict("exception")?.string("description") ?? d.string("text") ?? "exception") + } + let v = r.dict("result") ?? [:] + if v.string("type") == "undefined" { + return "undefined" + } + return remoteText(v) +} + +func size(_ n: Int) -> String { + if n >= 1 << 20 { + return String(format: "%.1fM", Double(n) / Double(1 << 20)) + } + if n >= 1 << 10 { + return String(format: "%.1fk", Double(n) / Double(1 << 10)) + } + return "\(n)" +} + +func networkList(_ t: TabState, _ filter: String?) -> String { + let first = t.reqs.first?.start ?? 0 + var out = "# status type at_ms ms bytes method url\n" + for r in t.reqs where filter.map({ r.url.lowercased().contains($0.lowercased()) }) ?? true { + let status = r.error.map { "failed(\($0))" } ?? r.status.map { "\($0)\(r.cached ? "(cache)" : "")" } ?? "pending" + let ms = r.end.map { String(Int((($0 - r.start) * 1000).rounded())) } ?? "-" + out += "\(r.seq) \(status) \(r.type) \(Int(((r.start - first) * 1000).rounded())) \(ms) \(size(r.bytes)) \(r.method) \(r.url)\n" + } + return out +} + +func headerLines(_ h: [String: Any]) -> String { + return h.keys.sorted().map { " \($0): \(h[$0] ?? "")" }.joined(separator: "\n") +} + +func networkGet(_ c: Chrome, _ t: TabState, _ n: Int) async throws -> String { + let snap: (head: String, resp: String, id: String, hasPost: Bool, done: Bool, mime: String)? = c.lock.withLock { + guard let r = t.reqs.first(where: { $0.seq == n }) else { + return nil + } + var head = "#\(r.seq) \(r.method) \(r.url)\n" + head += "type \(r.type), status \(r.error.map { "failed: \($0)" } ?? r.status.map(String.init) ?? "pending")" + head += r.proto.isEmpty ? "" : " \(r.proto)" + head += r.remote.isEmpty ? "" : ", from \(r.remote)" + head += r.cached ? ", from cache" : "" + head += ", \(size(r.bytes)) transferred" + head += r.end.map { ", took \(Int((($0 - r.start) * 1000).rounded()))ms" } ?? "" + head += "\nrequest headers:\n\(headerLines(r.reqHeaders))\n" + return (head, "response headers:\n\(headerLines(r.respHeaders))\n", r.id, r.hasPost, r.status != nil && r.error == nil && !r.redirected, r.mime) + } + guard let snap else { + throw ToolError("no request \(n); network lists them") + } + var out = snap.head + if snap.hasPost, let post = try? await c.send("Network.getRequestPostData", ["requestId": snap.id], session: t.session) { + out += "request body:\n\(post.string("postData") ?? "")\n" + } + out += snap.resp + guard snap.done else { + return out + } + do { + let b = try await c.send("Network.getResponseBody", ["requestId": snap.id], session: t.session) + let body = b.string("body") ?? "" + let bytes: Data = b.bool("base64Encoded") ? Data(base64Encoded: body) ?? Data() : Data(body.utf8) + guard let text = String(data: bytes, encoding: .utf8) else { + return out + "response body: \(bytes.count) bytes of \(snap.mime), not text\n" + } + let limit = 64 << 10 + if bytes.count <= limit { + return out + "response body (\(snap.mime), \(bytes.count) bytes):\n\(text)\n" + } + return out + "response body (\(snap.mime), \(bytes.count) bytes, the first \(limit) shown):\n\(String(decoding: bytes.prefix(limit), as: UTF8.self))\n" + } catch let e as ToolError { + return out + "response body: \(e.message)\n" + } +} + +let traceCategories = [ + "devtools.timeline", "disabled-by-default-devtools.timeline", "disabled-by-default-devtools.timeline.frame", + "disabled-by-default-devtools.timeline.stack", "toplevel", "v8.execute", "disabled-by-default-v8.cpu_profiler", + "blink.console", "blink.user_timing", "loading", "latencyInfo", +] + +func traceStop(_ c: Chrome, _ t: TabState, _ target: String) async throws -> String { + c.lock.withLock { t.traceStream = nil } + _ = try await c.send("Tracing.end", session: t.session) + let start = Date() + var stream: String? + while stream == nil { + if Date().timeIntervalSince(start) > 60 { + throw ToolError("Chrome didn't finish the trace within 60s") + } + await pause(100) + stream = c.lock.withLock { t.traceStream } + } + var data = Data() + while true { + let r = try await c.send("IO.read", ["handle": stream!, "size": 4 << 20], session: t.session) + let chunk = r.string("data") ?? "" + data.append(r.bool("base64Encoded") ? Data(base64Encoded: chunk) ?? Data() : Data(chunk.utf8)) + if r.bool("eof") { + break + } + } + _ = try? await c.send("IO.close", ["handle": stream!], session: t.session) + let url = FileManager.default.temporaryDirectory.appending(path: "byre-trace-\(UUID().uuidString.prefix(8).lowercased()).json") + try data.write(to: url) + return "saved the trace to \(url.path) (\(size(data.count)); Chrome DevTools' Performance panel opens it)\n" + traceSummary(data, frame: target) +} + +let taskParts: Set = [ + "EvaluateScript", "v8.compile", "v8.compileModule", "v8.evaluateModule", "FunctionCall", + "ParseHTML", "ParseAuthorStyleSheet", "UpdateLayoutTree", "Layout", + "PrePaint", "Paint", "Layerize", "MajorGC", "MinorGC", "V8.GC_MARK_COMPACTOR", "V8.GC_SCAVENGER", +] + +func traceSummary(_ data: Data, frame: String) -> String { + let root = try? JSONSerialization.jsonObject(with: data) + let events = (root as? [[String: Any]]) ?? ((root as? [String: Any])?["traceEvents"] as? [[String: Any]]) ?? [] + guard !events.isEmpty else { + return "the trace has no events" + } + var main: [Int: Int] = [:] + for e in events where e.string("ph") == "M" && e.string("name") == "thread_name" && e.dict("args")?.string("name") == "CrRendererMain" { + if let pid = e.int("pid"), let tid = e.int("tid") { + main[pid] = tid + } + } + let frameOf: ([String: Any]) -> String? = { $0.dict("args")?.string("frame") ?? $0.dict("args")?.dict("data")?.string("frame") } + let nav = events.last { e in + let url = e.dict("args")?.dict("data")?.string("documentLoaderURL") ?? "" + return e.string("name") == "navigationStart" && frameOf(e) == frame && !url.isEmpty && !url.hasPrefix("about:") + } + var out = "" + let base: Double + var pid: Int? + if let nav { + base = nav.double("ts") ?? 0 + pid = nav.int("pid") + out += "times are ms from the navigation to \(nav.dict("args")?.dict("data")?.string("documentLoaderURL") ?? "")\n" + } else { + base = events.compactMap { $0.double("ts") }.filter { $0 > 0 }.min() ?? 0 + let frames = events.first { $0.string("name") == "TracingStartedInBrowser" }?.dict("args")?.dict("data")?["frames"] as? [[String: Any]] + pid = frames?.first { $0.string("frame") == frame }?.int("processId") + out += "no page navigation in the trace; times are ms from the trace's start\n" + } + guard let pid, let tid = main[pid] else { + return out + "couldn't find the page's main thread in the trace" + } + let ms: (Double) -> Int = { Int((($0 - base) / 1000).rounded()) } + let mine = events.filter { $0.int("pid") == pid && ($0.double("ts") ?? 0) >= base } + var marks: [String] = [] + for (label, name, last) in [("FCP", "firstContentfulPaint", false), ("LCP", "largestContentfulPaint::Candidate", true), ("DOMContentLoaded", "MarkDOMContent", false), ("load", "MarkLoad", false)] { + let hits = mine.filter { $0.string("name") == name && frameOf($0) == frame } + if let e = last ? hits.last : hits.first, let ts = e.double("ts") { + var s = "\(label) \(ms(ts))" + if label == "LCP", let d = e.dict("args")?.dict("data"), let type = d.string("type") { + s += " (\(type), \(d.int("size") ?? 0)px²)" + } + marks.append(s) + } + } + out += marks.isEmpty ? "no paint or load marks in the trace\n" : marks.joined(separator: " ") + "\n" + let thread = mine.filter { $0.int("tid") == tid && $0.string("ph") == "X" } + let long = thread.filter { $0.string("name") == "RunTask" && ($0.double("dur") ?? 0) >= 50_000 }.sorted { ($0.double("dur") ?? 0) > ($1.double("dur") ?? 0) } + let busy = thread.filter { $0.string("name") == "RunTask" }.reduce(0.0) { $0 + ($1.double("dur") ?? 0) } + out += "main thread busy \(Int((busy / 1000).rounded()))ms in tasks; \(long.count) long tasks (50ms or more)" + if long.isEmpty { + return out + } + out += ", longest first:\n" + for task in long.prefix(10) { + let ts = task.double("ts") ?? 0 + let end = ts + (task.double("dur") ?? 0) + out += " \(Int(((task.double("dur") ?? 0) / 1000).rounded()))ms at \(ms(ts))" + let part = thread.filter { e in + let s = e.double("ts") ?? 0 + return taskParts.contains(e.string("name") ?? "") && s >= ts && s < end + }.max { ($0.double("dur") ?? 0) < ($1.double("dur") ?? 0) } + if let part { + let d = part.dict("args")?.dict("data") + let what = [d?.string("url"), d?.string("functionName"), d?.string("type")].compactMap { $0 }.filter { !$0.isEmpty }.joined(separator: " ") + out += ": mostly \(part.string("name") ?? "") \(Int(((part.double("dur") ?? 0) / 1000).rounded()))ms\(what.isEmpty ? "" : " " + what)" + } + out += "\n" + } + return out +} + +func shrink(_ png: Data, _ maxEdge: Double) throws -> (Data, String) { + guard let src = CGImageSourceCreateWithData(png as CFData, nil), + let full = CGImageSourceCreateImageAtIndex(src, 0, nil) + else { + throw ToolError("Chrome's screenshot isn't a PNG") + } + guard Double(max(full.width, full.height)) > maxEdge else { + return (png, "page viewport, image \(full.width)x\(full.height)") + } + let opts: [CFString: Any] = [kCGImageSourceCreateThumbnailFromImageAlways: true, kCGImageSourceThumbnailMaxPixelSize: Int(maxEdge)] + guard let img = CGImageSourceCreateThumbnailAtIndex(src, 0, opts as CFDictionary) else { + throw ToolError("couldn't scale Chrome's screenshot") + } + let data = NSMutableData() + let dest = CGImageDestinationCreateWithData(data, UTType.png.identifier as CFString, 1, nil)! + CGImageDestinationAddImage(dest, img, nil) + CGImageDestinationFinalize(dest) + return (data as Data, "page viewport, image \(img.width)x\(img.height) scaled from \(full.width)x\(full.height)") +} + +func chromeTool(_ args: [String: Any]) async throws -> [Content] { + let op = try args.require("op", args.string) + let maxEdge = args.double("max_edge") ?? 1568 + if !maxEdge.isFinite || maxEdge < 16 || maxEdge > 8192 { + throw ToolError("max_edge must be between 16 and 8192 pixels") + } + let (c, note) = try await chromeLink.get() + var log: [String] = note.isEmpty ? [] : [note] + if op == "tabs" { + var out = "tab title | url\n" + for p in try await c.pages() { + out += "\(p.string("targetId") ?? "") \(p.string("title") ?? "") | \(p.string("url") ?? "")\n" + } + log.append(out) + if !args.bool("capture") && args["wait_for"] == nil && args["wait_ms"] == nil { + return [.text(log.joined(separator: "\n"))] + } + } + let visible = ["select", "close", "cdp"].contains(op) || (op == "navigate" && args.bool("new_tab")) || args.bool("capture") + let lock = visible ? try await lockInput() : nil + defer { + if let lock { + close(lock) + } + } + var tabId = args.string("tab") + if op == "navigate", args.bool("new_tab") { + let r = try await c.send("Target.createTarget", ["url": "about:blank"]) + tabId = r.string("targetId") + } + let (target, t) = try await c.tab(tabId) + switch op { + case "tabs": + break + case "navigate": + let url = try args.require("url", args.string) + c.lock.withLock { t.clear() } + let r = try await c.send("Page.navigate", ["url": url], session: t.session) + if let err = r.string("errorText") { + throw ToolError("navigating to \(url) failed: \(err)") + } + let timeout = Double(try args.ms("timeout_ms") ?? 15000) / 1000 + let start = Date() + await pause(300) + var loaded = false + while !loaded, Date().timeIntervalSince(start) < timeout { + loaded = (try? await evaluate(c, t, "document.readyState")) == "complete" + if !loaded { + await pause(200) + } + } + log.append(String(format: loaded ? "loaded after %.1fs" : "still loading after %.1fs", Date().timeIntervalSince(start))) + log.append("tab \(target): " + (try await evaluate(c, t, "document.title + ' | ' + location.href"))) + case "js": + log.append(try await evaluate(c, t, try args.require("code", args.string))) + case "select": + _ = try await c.send("Target.activateTarget", ["targetId": target]) + log.append("selected tab \(target)") + case "close": + _ = try await c.send("Target.closeTarget", ["targetId": target]) + return [.text(log.joined() + "closed tab \(target)")] + case "network": + if let n = args.int("request") { + log.append(try await networkGet(c, t, n)) + } else { + log.append(c.lock.withLock { networkList(t, args.string("filter")) }) + } + case "console": + let lines = c.lock.withLock { t.console } + log.append(lines.isEmpty ? "no console messages" : lines.joined(separator: "\n")) + case "cdp": + let method = try args.require("method", args.string) + let r = try await c.send(method, args.dict("params") ?? [:], session: t.session) + let d = try JSONSerialization.data(withJSONObject: r, options: [.sortedKeys, .withoutEscapingSlashes]) + log.append(String(decoding: d, as: UTF8.self)) + case "trace_start": + _ = try await c.send("Tracing.start", ["transferMode": "ReturnAsStream", "traceConfig": ["includedCategories": traceCategories]], session: t.session) + log.append("tracing tab \(target); trace_stop ends it") + case "trace_stop": + log.append(try await traceStop(c, t, target)) + default: + throw ToolError("unknown op \(op.debugDescription)") + } + let present: (String) async throws -> Bool = { needle in + (try? await evaluate(c, t, "String((document.title + '\\n' + (document.body ? document.body.innerText : '')).toLowerCase().includes(\(jsString(needle.lowercased()))))")) == "true" + } + if let note = try await waitFor(args, acted: false, present) { + log.append(note) + } + guard args.bool("capture") else { + return [.text(log.joined(separator: "\n"))] + } + _ = try await c.send("Page.bringToFront", session: t.session) + let shot = try await c.send("Page.captureScreenshot", ["format": "png"], session: t.session) + let (png, shotNote) = try shrink(Data(base64Encoded: shot.string("data") ?? "") ?? Data(), maxEdge) + log.append(shotNote) + return [.text(log.joined(separator: "\n")), .png(png)] +} diff --git a/hands/Sources/byre-hands/Tools.swift b/hands/Sources/byre-hands/Tools.swift index f079ea9..2c1d091 100644 --- a/hands/Sources/byre-hands/Tools.swift +++ b/hands/Sources/byre-hands/Tools.swift @@ -96,4 +96,36 @@ let tools: [Tool] = [ """, run: browserTool ), + Tool( + name: "chrome", + description: "Drive Chrome on this Mac through its DevTools protocol: pages, DOM, network, console and performance traces. " + + "It is Chrome with byre's own profile, started on first use and kept between calls, so a site the human has logged into there (passkeys included) stays logged in; when a site needs a login, ask the human to sign in in that Chrome window. " + + "ops: tabs (list tabs: id, title, url), navigate (url, optionally new_tab; waits for the page to load and clears the tab's network and console logs), " + + "js (run code in the tab and return its value: the last expression's, promises awaited, objects as JSON; use it to explore the DOM), select (tab), close (tab), " + + "network (the tab's requests since this tool first touched the tab or its last navigate: number, status, type, start and duration in ms, bytes, method, url; with request, one request's headers and bodies; filter keeps urls containing the text), " + + "console (console messages, exceptions and browser log entries), " + + "trace_start and trace_stop (a performance trace: start, navigate or act, stop; stop saves the trace file and reports FCP, LCP, DOMContentLoaded, load and the main thread's long tasks), " + + "cdp (send any DevTools protocol method with params to the tab, e.g. DOM.getDocument, CSS.getComputedStyleForNode, Emulation.setCPUThrottlingRate, Network.setCacheDisabled; returns its result as JSON). " + + "tab is a tab id from tabs; without it, the tab this tool last used, else Chrome's first tab. capture screenshots the tab's page. " + + "It drives Chrome without the mouse or keyboard; select, close, navigate with new_tab, cdp, capture and the first call that starts Chrome can change what is in front, so those hold the shared input lock like act does.", + schema: """ + {"type": "object", "properties": { + "op": {"type": "string", "enum": ["tabs", "navigate", "js", "select", "close", "network", "console", "trace_start", "trace_stop", "cdp"]}, + "tab": {"type": "string", "description": "Tab id from tabs (default: the tab this tool last used, else Chrome's first tab)."}, + "url": {"type": "string", "description": "For navigate."}, + "new_tab": {"type": "boolean", "description": "For navigate: open the url in a new tab."}, + "code": {"type": "string", "description": "For js."}, + "request": {"type": "integer", "description": "For network: the request number to show in full."}, + "filter": {"type": "string", "description": "For network: only requests whose url contains this text (case-insensitive)."}, + "method": {"type": "string", "description": "For cdp: the protocol method, e.g. DOM.getDocument."}, + "params": {"type": "object", "description": "For cdp: the method's parameters."}, + "wait_for": {"type": "string", "description": "Then wait until this text (case-insensitive) is in the tab's title or page text, polling every 200ms."}, + "wait_ms": {"type": "integer", "description": "Wait this long after the op, before wait_for and capture."}, + "timeout_ms": {"type": "integer", "description": "How long navigate waits for the load (default 15000) and wait_for waits (default 10000). Giving up is reported, not an error."}, + "capture": {"type": "boolean", "description": "Finish with a screenshot of the tab's page."}, + "max_edge": {"type": "number", "description": "Longest edge of the screenshot in pixels, 16 to 8192 (default 1568, which is as much as you can see)."} + }, "required": ["op"]} + """, + run: chromeTool + ), ] diff --git a/internal/agents/clients.go b/internal/agents/clients.go index f50d132..9fa1ce3 100644 --- a/internal/agents/clients.go +++ b/internal/agents/clients.go @@ -148,15 +148,16 @@ func (m *Manager) clientTools(l *live) []mcp.Tool { }, { Name: "client_hands", - Description: "Call one of your hands tools (screen_list, capture, ax_tree, act, browser) on a client Mac instead of the Mac byre runs on: " + - "tool is the tool's name and arguments exactly what that tool takes. Window ids, refs and displays are that Mac's, so start with screen_list there. " + + Description: "Call one of your hands tools (screen_list, capture, ax_tree, act, browser, chrome) on a client Mac instead of the Mac byre runs on: " + + "tool is the tool's name and arguments exactly what that tool takes. Window ids, refs, displays and Chrome tabs are that Mac's, so start with screen_list (or chrome tabs) there. " + "Every agent using that Mac shares one set of refs, so read the tree again if another agent may have. " + - "A call fails until the human has given byre's app Screen Recording and Accessibility (and Automation of Orion, for browser) on that Mac; say so when it does.", + "A call fails until the human has given byre's app Screen Recording and Accessibility (and Automation of Orion, for browser) on that Mac; say so when it does. " + + "chrome needs Google Chrome installed on that Mac; its trace files are saved there, so fetch one with client_download.", InputSchema: json.RawMessage(`{ "type": "object", "properties": { "client": {"type": "string", "description": "The client's name from list_clients."}, - "tool": {"type": "string", "enum": ["screen_list", "capture", "ax_tree", "act", "browser"]}, + "tool": {"type": "string", "enum": ["screen_list", "capture", "ax_tree", "act", "browser", "chrome"]}, "arguments": {"type": "object", "description": "The tool's arguments."} }, "required": ["client", "tool"]