ai/byre
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Paste screenshots into iPhone message boxes; add ⋯ menu to the picker

Browse Source
This commit is contained in:
Ian Gulliver
2026-10-23 07:30:53 -07:00
parent c28b100fd0
commit 48f2ef3ade
4 changed files with 127 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/iphone.md
+2 -1
View File
@@ -14,10 +14,11 @@ iOS finds the app's App Intents (the Run workflow shortcut the Action button can
`mac/phone.sh --simulator` builds an ad-hoc-signed app for the iOS simulator instead, at `mac/.build/phone/iphonesimulator/Byre.app`; install it with `xcrun simctl install <device> <app>` and start it with `xcrun simctl launch <device> run.fc.byre.ios`. The simulator ignores entitlements given to `codesign` and refuses to launch an ad-hoc app signed with restricted ones, so the script links `mac/phone/byre.entitlements` into the binary's `__TEXT,__entitlements` section, as Xcode does; without the iCloud entitlement CloudKit raises at launch. Its node signs in to Tailscale like any other: `xcrun simctl launch --console-pty` prints libtailscale's sign-in URL, which changes on every launch.
The simulator's window (Xcode's Device Hub) lists the app's own accessibility elements, so hands' `ax_tree` and `act` `press` drive the app without a mouse, though the frames it gives those elements are wrong; take tap points from a `capture` of the window. Hands' mouse steps work there too: a tap is a `click`, and a long press (the edit menu with Paste) is a `click` with `hold`. macOS reports Device Hub as the frontmost app with process id -1, not its window's owner, which is why hands decides a window is in front by the on-screen window order. `xcrun simctl pbcopy` leaves the simulator's clipboard empty; set this Mac's clipboard and run `xcrun simctl pbsync host booted` instead, which also carries images. `xcrun simctl io booted screenshot --display LCD <file>` captures the app at full resolution.
The simulator's window (Xcode's Device Hub) lists the app's own accessibility elements, so hands' `ax_tree` and `act` `press` drive the app without a mouse, though the frames it gives those elements are wrong; take tap points from a `capture` of the window. Hands' mouse steps work there too: a tap is a `click`, and a long press (the edit menu with Paste) is a `click` with `hold`. macOS reports Device Hub as the frontmost app with process id -1, not its window's owner, which is why hands decides a window is in front by the on-screen window order. `xcrun simctl pbcopy` leaves the simulator's clipboard empty; set this Mac's clipboard and run `xcrun simctl pbsync host booted` instead, which also carries images. `xcrun simctl io booted screenshot --display LCD <file>` captures the app at full resolution. To put clipboard types the Mac can't produce (such as `com.apple.uikit.image` alone), compile a small UIKit command-line program with `xcrun --sdk iphonesimulator swiftc -target arm64-apple-ios27.0-simulator` and run it with `xcrun simctl spawn booted <binary>`; it can set `UIPasteboard.general` and print its `types`. The simulator has no Photos app, and its AssistiveTouch menu draws no icons, so an iOS screenshot's Copy and Delete can't be tried there.
## Platform facts
- CloudKit on iOS requires the `remote-notification` background mode in `Info.plist`, even though the app sets up no push of its own.
- The phone's name is the one typed in Settings, kept in the phone's own `UserDefaults` (not synced through iCloud, since each phone has its own), and `iPhone` until one is typed. It goes in `Byre-Host` and the tailnet node's name, both fixed when the app starts. `UIDevice.current.name` returns just `iPhone` without Apple's `com.apple.developer.device-information.user-assigned-device-name` entitlement, which Apple refused because byre shows the name to other people and sends it to Tailscale and to the agents' model vendors.
- The message boxes are a `UITextView` subclass (`ImageTextView`), not SwiftUI's `TextEditor`. `TextEditor`'s text view offers Paste only for text, and a `.pasteDestination` on it didn't show Paste for an iPhone screenshot's Copy and Delete on a real phone, although it did for PNG and HEIC in the simulator. The subclass offers Paste whenever `UIPasteboard.general.hasImages` and reads `UIPasteboard.general.images`, the same read as ⋯ → Paste image, which works with screenshots. The edit menu's Paste reads the clipboard without iOS's "Allow Paste" prompt; the ⋯ menu's item gets the prompt.
- SwiftTerm's iOS `TerminalViewDelegate` requires `requestOpenLink`, which its macOS one provides a default for.