The Project
Harness Voice began as Hermes Voice, an app for talking to a coding agent on my Mac. I built it to replace a rough Telegram voice experience. The iPhone app records a push-to-talk question and sends it to a FastAPI backend on the Mac. The backend transcribes the audio, passes the text to an agent, and returns the reply as text and audio.
The backend now has adapters for Claude Code, Codex, and OpenCode alongside Hermes, and Claude Code is the default harness for the public build.
One turn, end to end
A turn starts on the phone, which streams the recording to /api/audio/stream and falls back to /api/audio if the stream fails. If on-device transcription is enabled and ready, the phone transcribes the clip itself and sends only text.
The backend transcribes the upload and calls the selected harness. On the standard path it emits server-sent events in a fixed order: transcribed, live tool events, tools, assistant, audio, and done. Before speaking, it strips markdown from the reply, while the transcript keeps the raw text. With ElevenLabs, playback can start before synthesis finishes, because that provider streams its chunks. The agent’s own thinking and tool calls are the slowest part of a turn.
Speech providers behind small seams
Speech-to-text and text-to-speech each sit behind a small provider protocol. At startup the backend picks one from an override such as STT_PROVIDER, or from whichever API key is set, in a documented order. Speech-to-text tries OpenAI, Groq, ElevenLabs Scribe, and then local faster-whisper. Text-to-speech tries ElevenLabs, OpenAI, and then Piper. The override lets me compare providers without deleting keys, and /health reports which one is live. When a local voice is selected, the phone skips server speech and uses Apple’s synthesizer. The tests use fake providers and no network.
Keeping the server private
In the guided setup, the backend listens on loopback only. Tailscale Serve terminates HTTPS on the tailnet and forwards to that port. In Serve mode the backend refuses to start unless a token is set, the bind is 127.0.0.1, no TLS files are configured, and forwarded-header trust is limited to loopback. The provisioner manages only the port 8765 mapping and never calls tailscale serve reset.
The server token lives in the macOS Keychain. A small launcher reads it through Security.framework and passes it only to the backend’s process environment, so it stays out of the LaunchAgent plist and the command line.
The token never has to be copied by hand. The pair command prints a QR code with a single-use pairing ID and secret that expires after five minutes. The phone claims it over HTTPS and receives the token once, in the response. The setup prompt never asks anyone to paste a token into a chat.
A provisioner that undoes its changes
Mac setup runs as one transaction. It first records the current plist, token, loaded LaunchAgents, and Serve state, and it stops if the port 8765 mapping conflicts with what it expects. It then stages the new plist outside the folder launchd watches, adds the exact Serve mapping, and boots the new LaunchAgent. It commits only after a public HTTPS /health check and an authenticated harness check pass.
Changes register undo steps as they happen, and a failure runs them in reverse, restoring the previous plist and token and restarting any agent that was running. If a critical undo step fails, rollback halts before restoring more state. Cleanup after commit is separate, and a failure there is reported as a committed deployment with incomplete cleanup.
The app does not install anything on the Mac. It copies a setup prompt, and a coding agent already running on that Mac runs the provisioner, which avoids a remote-control channel with broad permissions. The undo logic grew out of live runs. Commits from late July include “halt rollback when new agent stays loaded” and “stage LaunchAgent plist outside watched directory.”
Wrap Up
Version 1.0 reached TestFlight in July 2026. Then voice modes built into Codex and Claude Code covered most of what I needed, so I stopped there instead of pushing to the App Store. The parts I would point to first still hold up: the provider seams, the loopback bind behind Tailscale Serve, and the undo logic that live runs exposed.