Build your first iPhone agent

ESP32 iPhone Agent connects a vision model to a real iPhone through an ESP32-S3. The current version runs the agent inside an iPhone app; you do not need a computer while a task is running.

Before you begin

Have these ready:

• An ESP32-S3 with PSRAM and native USB. The documented test board is the plain Seeed Studio XIAO ESP32S3 with 8 MB PSRAM.
• An iPhone on iOS 18 or later, with cellular data available. Lightning models also need a Lightning to USB Camera Adapter.
• A Mac with Xcode for the initial firmware and app setup, plus an Apple ID for signing.
• A vision-capable model endpoint that accepts images through an OpenAI-compatible chat API.

The board takes the phone’s Wi-Fi connection for screen mirroring. The phone reaches your model over cellular data. Check the hardware requirements before buying a board or adapter.

1. Get the source

Clone the project and enter its directory. All commands in this guide are run from that directory unless stated otherwise.

git clone https://github.com/onehumanbeing/esp32-iphone-agent.git
cd esp32-iphone-agent

These guides cover the current iPhone-app architecture. The legacy branch and v0.1.0 tag use a computer-side server and have a different setup.

2. Flash the board

Install arduino-cli, the ESP32 3.3.x board core, and NimBLE-Arduino 2.5.1. Connect the board to your Mac with a USB data cable, then use the project’s hardware/flash.sh script.

arduino-cli lib install "[email protected]"
hardware/flash.sh

Follow the firmware guide for board-core setup, serial-port selection, and recovery. Do not replace the script with a default Arduino build: it applies the BLE buffer configuration the video connection needs.

3. Install the iPhone app

Copy the local configuration, then set your signing team and a unique bundle identifier in the new file.

cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig
open ios/IPhoneAgent.xcodeproj

In Xcode, choose the IPhoneAgent scheme and your connected iPhone, then build and run. A free Apple ID can sign a development build, but it needs re-signing every seven days. Optional entitlements need special attention on a free account; read Install the iOS app.

4. Complete the four setup checks

Move the board from the Mac to your iPhone and open ESP32 Agent. The setup flow checks:

1. Plug Me In — find the board and pair over Bluetooth.
2. Let Me See the Screen — join the board’s hotspot, then select ESP32 Agent in Control Center → Screen Mirroring.
3. Let Me Use the Pointer — enable AssistiveTouch and let the app calibrate the pointer.
4. Give Me a Brain — enter your model endpoint, API key, and image-capable model.

Use the connection guide for each check and the model guide to verify the endpoint.

5. Run a small first task

Try “Open Settings and go to Wi-Fi.” Use the instruction in English if your model or phone interface works best that way. Tap Start, keep the phone unlocked, and watch the agent work. Stop from the Dynamic Island when available, or return to the app to stop it.

Start with a task you can verify easily. The keyboard currently types ASCII only; a Chinese or Japanese task instruction does not make Chinese or Japanese text entry available.

Check the result

Open Runs and replay the task. A model reporting completion is not independent confirmation that the task succeeded. Check the final screen yourself, and export the run if you need to inspect an error.

If setup stalls, follow troubleshooting in the order of the four setup checks. You do not need to reinstall everything to isolate one failed connection.