Connect a vision model

ESP32 Agent sends screenshots to the model endpoint you configure. You need a model that accepts image input through an OpenAI-compatible chat API. A provider preset or a successful model-list request alone does not prove image support.

Configure the endpoint

In the app’s model setup, choose a matching provider preset or Others for a custom base URL. Enter your own API key and the exact model identifier available to that account.

The base URL is the API root, not the complete chat-completions URL. For a service documented at https://your-model.example/v1/chat/completions, the base is typically https://your-model.example/v1. Use your service’s actual documented address; this example is not a working endpoint.

Do not paste a key for one provider into another provider’s endpoint. Credentials are stored in the phone’s Keychain; you should still verify the destination before sending screenshots.

Choose a model that can see

Confirm all three conditions:

1. The model accepts images in chat requests.
2. Your account has access to the exact model ID.
3. The endpoint supports the image request format the app uses.

The app can list models, lets you edit the model ID, and supports separate planner and actor models. Begin with one known image-capable model for both roles, then change one thing at a time when comparing behavior.

Test from the app

Use the connection test after editing the configuration. The current app tests with a generated color image. A passing result verifies that request, not overall reliability on every app or task.

• Authentication failure: check the host, key, account permissions, and billing status with your provider.
• Unknown model: check the exact model ID and access rights.
• Image rejected: confirm vision support and API compatibility.
• Timeout: check cellular connectivity and whether the endpoint is reachable from the phone.

Self-hosted models

The project supports compatible endpoints served by tools such as vLLM, SGLang, llama.cpp, and Ollama, provided the selected model and server accept the required image requests.

The board’s hotspot occupies Wi-Fi and provides no internet. The current setup reaches the model over cellular data. A LAN-only address, and localhost pointing at the phone itself, will not automatically reach a model running on your computer. Expose a properly secured endpoint reachable from the phone before testing this configuration.

Understand data and cost

Screenshots and task text leave the phone for your selected endpoint. That provider’s privacy and retention policies apply. Choosing a self-hosted model does not by itself make the entire network path offline.

API usage is separate from the hardware cost. No universal speed, success rate, or “best model” is promised here. Evaluate a small, repeatable task and compare its saved run data.

Change the prompts

Planner and actor prompts live in the repository’s prompts/ directory and are bundled into the app. Edit them and rebuild to change the agent’s instructions. Keep a record of your model and prompt configuration when comparing results.