AI Phone Agents: A Developer’s Guide to Self-Hosting Voice Automation

Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

AI phone agents — automated systems that answer calls, hold natural conversations, and route or resolve requests without a human on the line — have moved from novelty to production infrastructure. If you run servers for a living, you’ve probably already been asked to “just wire up an AI agent to the support line.” This guide walks through what that actually takes: the architecture, the Docker-based deployment, the telephony integration, and the operational tradeoffs of running AI phone agents on infrastructure you control instead of a black-box SaaS.

What Are AI Phone Agents?

An AI phone agent is a pipeline that converts an inbound or outbound phone call into a real-time conversation with a language model. Under the hood, it’s a chain of four systems working in near real time:

Telephony layer — receives/places the call and streams audio (Twilio, Telnyx, or SIP trunking)

Speech-to-text (STT) — transcribes caller audio to text as it arrives

LLM orchestration — generates a response, calls tools/APIs, manages conversation state

Text-to-speech (TTS) — converts the LLM’s reply back into audio streamed to the caller

Commercial platforms like Vapi or Retell bundle all of this behind an API key. That’s fine for a prototype, but once you’re handling real call volume, you hit the same wall every SaaS-wrapped-open-source product eventually hits: per-minute pricing that scales linearly with usage, vendor lock-in on voice models, and no control over where audio and transcripts are stored — a real problem if you’re in a regulated industry.

Why Self-Host AI Phone Agents

Running your own stack makes sense once you clear a few hundred call-minutes a day, or once data residency actually matters. The tradeoffs are concrete:

Cost at scale — self-hosted STT/TTS on a GPU box is often 5-10x cheaper per minute than metered API pricing once utilization is decent

Latency control — colocating your LLM inference and telephony bridge in the same region cuts round-trip latency, which matters a lot for conversational feel

Data ownership — call transcripts and audio never leave infrastructure you control, which simplifies compliance

Model flexibility — swap in any open-weight LLM or fine-tune on your own call logs

The cost is operational complexity. You’re now responsible for uptime, GPU capacity, and a real-time audio pipeline instead of a REST call.

Core Architecture

A minimal self-hosted stack looks like this:

Caller --> Twilio Media Streams (WebSocket) --> Orchestrator (Python/Node)
                                                       |
                    +----------------------------------+----------------------------------+
                    |                                  |                                  |
              Whisper (STT)                     LLM (vLLM/Ollama)                  Piper/Coqui (TTS)

Twilio (or an equivalent provider) opens a bidirectional WebSocket media stream for the call. Your orchestrator service buffers incoming audio chunks, feeds them to a streaming STT model, passes the resulting text to an LLM, and streams the LLM’s response through TTS back over the same socket. All of this needs to happen with well under a second of round-trip latency to feel like a real conversation.

Docker Compose Setup

Here’s a working docker-compose.yml for the core services. It assumes a GPU-enabled host for the STT and LLM containers.

version: "3.9"

services:
  orchestrator:
    build: ./orchestrator
    ports:
      - "8080:8080"
    environment:
      - TWILIO_ACCOUNT_SID=${TWILIO_ACCOUNT_SID}
      - TWILIO_AUTH_TOKEN=${TWILIO_AUTH_TOKEN}
      - STT_URL=http://whisper:9000
      - LLM_URL=http://llm:11434
      - TTS_URL=http://tts:5002
    depends_on:
      - whisper
      - llm
      - tts

  whisper:
    image: onerahmet/openai-whisper-asr-webservice:latest
    environment:
      - ASR_MODEL=medium
      - ASR_ENGINE=faster_whisper
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  llm:
    image: ollama/ollama:latest
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  tts:
    image: ghcr.io/coqui-ai/tts:latest
    command: ["--model_name", "tts_models/en/vctk/vits"]

volumes:
  ollama_data:

Bring it up with:

docker compose up -d
docker compose logs -f orchestrator

If you haven’t tuned a Docker Compose stack for GPU workloads before, our Docker Compose GPU deployment guide covers the NVIDIA Container Toolkit setup you’ll need before these containers will actually see the GPU.

Setting Up Speech-to-Text and Text-to-Speech

For STT, OpenAI’s Whisper run through faster-whisper gives near real-time transcription on a single consumer GPU. The medium model is a good balance of accuracy and latency for phone-quality (8kHz) audio; large-v3 is more accurate but adds 200-400ms of latency per chunk, which is often not worth it for a live call.

For TTS, Coqui TTS and Piper are the two open-source options worth evaluating. Piper is CPU-friendly and extremely fast, which matters if you’re not running TTS on a GPU box — it can hit sub-200ms synthesis on modest hardware. Coqui’s VITS models sound more natural but need a GPU to stay fast enough for live conversation.

A minimal orchestrator loop (Python, using websockets and httpx) looks like this:

import asyncio
import httpx
import websockets

async def handle_call(websocket):
    async for message in websocket:
        audio_chunk = decode_media_stream(message)
        transcript = await transcribe(audio_chunk)
        if transcript:
            reply = await query_llm(transcript)
            audio_reply = await synthesize(reply)
            await websocket.send(encode_media_stream(audio_reply))

async def transcribe(chunk: bytes) -> str:
    async with httpx.AsyncClient() as client:
        resp = await client.post("http://whisper:9000/asr", content=chunk)
        return resp.json().get("text", "")

async def query_llm(text: str) -> str:
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://llm:11434/api/generate",
            json={"model": "llama3.1:8b", "prompt": text, "stream": False},
        )
        return resp.json()["response"]

async def synthesize(text: str) -> bytes:
    async with httpx.AsyncClient() as client:
        resp = await client.post("http://tts:5002/api/tts", json={"text": text})
        return resp.content

This is intentionally bare — production versions need interrupt handling (so the caller can talk over the agent), silence detection to know when the caller has finished speaking, and streaming responses from the LLM rather than waiting for the full completion.

Connecting to a Telephony Provider

Twilio’s Media Streams API is the easiest on-ramp: you point a TwiML <Connect><Stream> verb at your orchestrator’s WebSocket URL and Twilio handles the PSTN side entirely. For higher call volume or lower per-minute cost, a SIP trunk into Asterisk or FreeSWITCH gives you more control but adds real telephony ops work — NAT traversal, codec negotiation, and carrier peering are not trivial.

A basic TwiML response to bridge an inbound call into your stream:

<Response>
  <Connect>
    <Stream url="wss://agents.example.com/media" />
  </Connect>
</Response>

Scaling and Monitoring Your AI Phone Agent Infrastructure

Once you’re past a handful of concurrent calls, a single Docker host won’t cut it. Each active call pins an STT stream, an LLM context, and a TTS session simultaneously, so concurrency scales GPU memory usage fast. Plan for:

Horizontal scaling of the orchestrator behind a load balancer, with sticky WebSocket routing per call

GPU pooling — batch multiple Whisper/TTS requests where the model server supports it (faster-whisper and vLLM both do)

Call queuing — a Redis-backed queue to hold calls when GPU capacity is saturated, rather than dropping them

Monitoring matters more here than in most services because failures are invisible until a caller hangs up frustrated. We run BetterStack for uptime checks on the orchestrator’s health endpoint and log-based alerting on STT/TTS error rates — worth setting up before you put a phone number in front of real customers. If you haven’t built out alerting for a service like this before, see our guide to self-hosted monitoring stacks for the Prometheus/Grafana side of things.

For the hosting layer itself, GPU-backed instances are the main cost driver. DigitalOcean’s GPU droplets are a reasonable starting point if you want simple provisioning without committing to reserved capacity, and they scale down cleanly if call volume doesn’t materialize. If you’re running steady, predictable volume instead, Hetzner’s dedicated GPU servers come in significantly cheaper per hour for always-on workloads.

Security Considerations

Phone calls carry PII by default — names, account numbers, sometimes payment details read aloud. A few non-negotiables:

Encrypt call recordings and transcripts at rest, and set a retention policy instead of keeping everything indefinitely

Put the orchestrator’s public WebSocket endpoint behind TLS termination and IP allowlisting for your telephony provider’s signaling ranges

Never let the LLM’s tool-calling layer execute arbitrary actions (refunds, account changes) without a secondary confirmation step — prompt injection via a caller’s spoken input is a real attack surface, not a theoretical one

Front the public endpoints with Cloudflare for DDoS protection and WAF rules, since a public-facing voice endpoint is an easy target for automated abuse

If you’re exposing any of this stack to the public internet, our Linux server hardening checklist is a reasonable baseline before you take a first production call.

Cost Comparison: Self-Hosted vs SaaS

Rough numbers based on a single GPU instance handling moderate concurrent call volume:

| Approach | Cost per minute | Setup effort | Data control |
|—|—|—|—|
| SaaS (Vapi/Retell) | $0.05-$0.15 | Low | None |
| Self-hosted (cloud GPU) | $0.01-$0.03 | High | Full |
| Self-hosted (dedicated GPU) | $0.005-$0.015 | High | Full |

The breakeven point is usually somewhere around 5,000-10,000 call-minutes a month, depending on which models you’re running and how efficiently you batch inference. Below that volume, the engineering time to build and maintain the pipeline usually costs more than the SaaS markup.

Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Do AI phone agents need a GPU to run?
STT and TTS models can run on CPU, but latency suffers enough that conversations feel sluggish. For anything beyond a low-volume prototype, a GPU for Whisper and your LLM is effectively required to keep round-trip latency under a second.

Can I use GPT-4 or Claude instead of a self-hosted LLM?
Yes — nothing about this architecture requires an open-weight model. Swapping the query_llm function to call an API instead of Ollama works fine; you just give up the cost and data-residency benefits of full self-hosting for that piece of the pipeline.

How do I handle interruptions when the caller talks over the agent?
You need voice activity detection (VAD) running continuously on the incoming stream, and the orchestrator must be able to cancel an in-flight TTS response the moment new caller speech is detected. Libraries like webrtcvad or Silero VAD handle the detection; the cancellation logic is on you.

What’s the biggest latency bottleneck in this stack?
Usually the LLM generation step, especially if you wait for a full completion before starting TTS. Streaming tokens from the LLM into TTS as they arrive, rather than waiting for the full response, is the single biggest latency win available.

Is this legal for handling customer calls?
Generally yes, but call recording consent laws vary by state and country — some require two-party consent before recording. Check local requirements and play a disclosure message before recording begins.

How many concurrent calls can one GPU handle?
Depends heavily on model size and GPU memory, but a single mid-range GPU (e.g., an RTX 4090 or A10) running faster-whisper medium plus a small quantized LLM can typically handle 8-15 concurrent calls before latency degrades noticeably.

Wrapping Up

Self-hosting AI phone agents is a legitimate infrastructure project, not just a wrapper around an API key — you’re building a real-time audio pipeline with the same latency and reliability demands as VoIP itself. Start with the Docker Compose stack above on a single GPU box, get the conversation loop working end to end, and only invest in horizontal scaling once you have actual call volume to justify it.