ai/byre
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Paste images in iPhone message boxes; let hands drive the simulator

Browse Source
This commit is contained in:
Ian Gulliver
2026-10-23 06:50:55 -07:00
parent de9a830cc6
commit c6da60f728
5 changed files with 33 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/iphone.md
+1 -1
View File
@@ -14,7 +14,7 @@ iOS finds the app's App Intents (the Run workflow shortcut the Action button can
`mac/phone.sh --simulator` builds an ad-hoc-signed app for the iOS simulator instead, at `mac/.build/phone/iphonesimulator/Byre.app`; install it with `xcrun simctl install <device> <app>` and start it with `xcrun simctl launch <device> run.fc.byre.ios`. The simulator ignores entitlements given to `codesign` and refuses to launch an ad-hoc app signed with restricted ones, so the script links `mac/phone/byre.entitlements` into the binary's `__TEXT,__entitlements` section, as Xcode does; without the iCloud entitlement CloudKit raises at launch. Its node signs in to Tailscale like any other: `xcrun simctl launch --console-pty` prints libtailscale's sign-in URL, which changes on every launch.
The simulator's window (Xcode's Device Hub) lists the app's own accessibility elements, so hands' `ax_tree` and `act` `press` drive the app without a mouse. Hands' mouse steps (`click`, `drag`, `scroll`) refuse that window, because Device Hub reports no process id; a gesture such as dragging the keyboard away needs mouse events posted with `CGEvent` instead. `xcrun simctl io booted screenshot --display LCD <file>` captures the app at full resolution.
The simulator's window (Xcode's Device Hub) lists the app's own accessibility elements, so hands' `ax_tree` and `act` `press` drive the app without a mouse, though the frames it gives those elements are wrong; take tap points from a `capture` of the window. Hands' mouse steps work there too: a tap is a `click`, and a long press (the edit menu with Paste) is a `click` with `hold`. macOS reports Device Hub as the frontmost app with process id -1, not its window's owner, which is why hands decides a window is in front by the on-screen window order. `xcrun simctl pbcopy` leaves the simulator's clipboard empty; set this Mac's clipboard and run `xcrun simctl pbsync host booted` instead, which also carries images. `xcrun simctl io booted screenshot --display LCD <file>` captures the app at full resolution.
## Platform facts