How ESP32 iPhone Agent works

The current system consists of an iPhone, an ESP32-S3, and a vision-model endpoint. The iPhone app runs the agent. The board receives the screen and supplies mouse and keyboard input.

Eyes: AirPlay and Bluetooth

The board opens its own Wi-Fi hotspot. The iPhone joins it and selects ESP32 Agent as an AirPlay screen-mirroring receiver.

The ESP32 does not decode the H.264 video. It forwards the stream over Bluetooth Low Energy to ESP32 Agent on the same phone. The app decodes it and prepares screenshots for the vision model.

iPhone screen → AirPlay / Wi-Fi → ESP32-S3
ESP32-S3 → H.264 over BLE → iPhone app → vision model

The phone uses cellular data for model requests while Wi-Fi is occupied by the board’s hotspot.

Brain: the iPhone app and model

A planner and actor translate the task and screenshots into operations such as tapping, swiping, drawing, typing, pressing keys, and going home. The model endpoint can be a cloud API or a reachable self-hosted service.

The agent runs in the background while interacting with other apps. A Live Activity can show its progress on supported devices. Model reasoning is not performed on the ESP32.

Hands: standard USB input

The board presents itself as a USB mouse and keyboard. iOS AssistiveTouch exposes a pointer, allowing a click to act as a tap and a drag to act as a swipe.

iPhone app → action over BLE → ESP32-S3
ESP32-S3 → USB mouse / keyboard → iPhone

USB supplies the input connection and board power. Bluetooth carries the app’s commands and screen stream. Do not confuse this BLE app link with a cable-free Bluetooth HID mouse: the current documented setup uses USB HID.

Closing the loop

Mouse movement is relative, so calibration and correction are important. The app tracks the pointer, periodically re-homes it, and can locate the pointer on screen before precise taps. After an action it waits for the screen to settle before deciding what to do next.

This is a visual control loop, not a direct semantic API for every app. Ambiguous screens, tiny targets, model mistakes, and latency can all affect the result.

Why use a board?

The ESP32 bridges two interfaces iOS already supports: a mirrored screen and a mouse/keyboard. It provides the physical input path; seeing a phone’s screen on another device alone does not supply that path to an on-phone agent.

This makes the approach useful for experimentation across app screens without building a dedicated integration for each one. It does not remove all setup requirements: the current app still needs initial Xcode installation, and unsupported screen content remains invisible.

Current version and legacy version

The v0.1.0 release used a Mac or Linux computer for UxPlay and a Python server. That version remains on the legacy branch. Current main moves the AirPlay receiver onto the board and the agent into the iPhone app.

Use the current setup guide for a new installation. Only follow legacy instructions when intentionally using the old architecture.