Each of these can be a hosted API or an open-source model running in a container. The self-hosted approach swaps API calls for local inference where it makes sense, and keeps everything else containerized for reproducibility.

Why Self-Host Instead of Using a SaaS Platform

The economics change quickly at volume. A SaaS AI phone agent platform typically charges $0.05–$0.15 per minute on top of telephony carrier costs. If you’re running a support line or an outbound campaign doing thousands of minutes a month, that’s real money. Self-hosting shifts the cost to a fixed VPS bill plus carrier minutes, and gives you full control over latency, logging, and model choice.

The tradeoff is operational complexity — you’re now responsible for uptime, scaling, and monitoring, which is exactly the kind of work a DevOps-oriented reader is already comfortable with. If you haven’t containerized a networked service before, it’s worth reading our Docker networking basics guide first, since the telephony and STT containers need to talk to each other reliably.

Core Architecture

A minimal self-hosted stack looks like this:

Caller ↔ SIP Trunk ↔ Asterisk (container) ↔ Audio bridge (WebSocket)
                                                   ↓
                                        STT service (Whisper, container)
                                                   ↓
                                        LLM orchestrator (container)
                                                   ↓
                                        TTS service (Piper/Coqui, container)
                                                   ↓
                                        Back to Asterisk → Caller

Each box is its own container, connected over an internal Docker network. This separation matters operationally: you can scale STT and TTS independently, swap the LLM provider without touching telephony, and restart any single component without dropping the whole stack.

Setting Up the Stack with Docker

Start with a docker-compose.yml that defines the core services. This example uses Asterisk for SIP telephony, OpenAI’s Whisper for STT, a lightweight FastAPI orchestrator for the LLM logic, and Piper for TTS.

version: "3.9"
services:
  asterisk:
    image: andrius/asterisk:18
    ports:
      - "5060:5060/udp"
      - "10000-10100:10000-10100/udp"
    volumes:
      - ./asterisk-config:/etc/asterisk
    networks:
      - agent-net

  stt:
    build: ./stt
    environment:
      - MODEL_SIZE=small
    networks:
      - agent-net

  orchestrator:
    build: ./orchestrator
    environment:
      - LLM_API_KEY=${LLM_API_KEY}
    depends_on:
      - stt
      - tts
    networks:
      - agent-net

  tts:
    build: ./tts
    networks:
      - agent-net

networks:
  agent-net:
    driver: bridge

The orchestrator container is the piece you’ll customize most. A minimal version using FastAPI to bridge transcripts to an LLM might look like this:

# orchestrator/main.py
from fastapi import FastAPI, WebSocket
import httpx
import os

app = FastAPI()
LLM_API_KEY = os.environ["LLM_API_KEY"]

@app.websocket("/call")
async def handle_call(websocket: WebSocket):
    await websocket.accept()
    async with httpx.AsyncClient() as client:
        while True:
            transcript = await websocket.receive_text()
            response = await client.post(
                "https://api.openai.com/v1/chat/completions",
                headers={"Authorization": f"Bearer {LLM_API_KEY}"},
                json={
                    "model": "gpt-4o-mini",
                    "messages": [{"role": "user", "content": transcript}],
                },
            )
            reply = response.json()["choices"][0]["message"]["content"]
            await websocket.send_text(reply)

This is deliberately minimal — no retry logic, no conversation memory — but it’s enough to prove the pipeline end-to-end before you add production concerns like call state, interruption handling, and logging.

Connecting Telephony (Twilio or SIP Trunk)

You have two realistic options for the telephony leg:

For most teams starting out, Twilio Media Streams is the faster path — you skip SIP/NAT headaches entirely and can have a working prototype in an afternoon. Move to Asterisk once call volume justifies the added ops burden.

Deploying to a VPS

Audio streaming is latency-sensitive, so where you deploy matters. A few practical guidelines:

If you haven’t hardened a fresh VPS before running production telephony workloads on it, check our VPS hardening checklist before opening the required ports to the internet.

For the actual host, DigitalOcean droplets and Hetzner cloud servers are both solid picks — Hetzner tends to be cheaper per vCPU, DigitalOcean has slightly better network peering in North America. Either works fine for a stack this size; pick based on where your callers are.

Monitoring and Scaling

A phone agent that silently drops calls is worse than no phone agent at all, so monitoring isn’t optional here. At minimum, track:

BetterStack is a reasonable option for uptime and log monitoring across the whole stack — you can point it at your Docker container logs and get alerted the moment the orchestrator starts throwing errors, rather than finding out from an angry caller. Pair it with basic container health checks in your docker-compose.yml so Docker itself restarts a crashed STT or TTS container automatically.

Once a single instance is stable, scaling out is mostly a matter of running multiple orchestrator + STT + TTS replicas behind a load balancer, with Asterisk (or Twilio) distributing calls across them. If you’re new to running Compose stacks in production, our Docker Compose production guide covers restart policies, health checks, and resource limits in more depth.

If you’re serving the agent’s web-facing dashboard or webhook endpoints publicly, put Cloudflare in front of it for DDoS protection and TLS termination — telephony webhook endpoints are a common target for probing traffic once they’re discoverable.

Handling Conversation State

The minimal orchestrator above treats every transcript as a one-off request, which breaks down the moment a caller references something they said 20 seconds earlier. In production you’ll want:

Dead air is the single biggest giveaway that a caller is talking to a bot rather than a human, so latency budgeting deserves more attention than the LLM prompt itself.

Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Q: Do I need a GPU to run an AI phone agent’s speech models?
A: Not necessarily. Whisper’s smaller models run acceptably on CPU for low call volume, but if you’re handling more than a handful of concurrent calls, a GPU (or a hosted STT API) will keep latency down significantly.

Q: Is Twilio required, or can I avoid it entirely?
A: Twilio isn’t required — Asterisk with a SIP trunk provider gives you full self-hosted telephony — but Twilio’s Media Streams API is significantly faster to prototype with and worth using for an initial proof of concept.

Q: How much does self-hosting actually save compared to a SaaS AI phone agent?
A: It depends on volume. Below a few thousand minutes a month, a SaaS platform is often cheaper once you factor in your own engineering time. Above that, self-hosting on a $40–80/month VPS usually wins.

Q: What’s the biggest source of latency in the pipeline?
A: Usually STT transcription, followed by LLM response time. TTS is typically the fastest step if you’re using a lightweight model like Piper.

Q: Can this stack handle outbound calls, not just inbound?
A: Yes — both Asterisk and Twilio support originating calls. The orchestrator logic is identical; only the call-initiation step differs.

Q: How do I keep call recordings compliant with regulations like TCPA or GDPR?
A: Store recordings and transcripts encrypted at rest, log consent/disclosure at call start, and set a retention policy that automatically purges data after a defined period — this is a legal question as much as a technical one, so involve counsel for anything customer-facing.

Self-hosting an AI phone agent isn’t a weekend project, but if you’re already comfortable with Docker and basic Linux ops, none of the individual pieces here are exotic. Start with Twilio Media Streams and a hosted LLM API to validate the concept, then move pieces in-house — Whisper, then TTS, then full SIP — as volume and cost justify it.