AI Voice Agents for Customer Service: A Self-Hosted Deployment Guide

Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

AI voice agents for customer service have moved past the novelty phase. They now handle tier-1 support calls, appointment scheduling, and order status lookups at a fraction of the cost of a human agent bank. But most of the marketing around this space glosses over the actual infrastructure work: standing up speech-to-text, an LLM orchestration layer, text-to-speech, and telephony integration in a way that’s reliable, observable, and doesn’t leak customer data to five different SaaS vendors.

This guide is written for developers and sysadmins who need to actually deploy this stack, not just read about it. We’ll cover the architecture, containerize the core components, wire up monitoring, and talk through the tradeoffs between self-hosting and using managed APIs.

Why Self-Host AI Voice Agents

Most teams start with a managed platform (Vapi, Retell, Bland) because it’s fast to prototype. The problem shows up at scale: per-minute pricing adds up fast on high call volumes, and you’re locked into whatever latency and uptime your vendor delivers. If you’re running a support desk that handles thousands of calls a day, the economics shift hard toward self-hosting the pipeline on your own cloud infrastructure and only paying per-token for the LLM calls you actually make.

Self-hosting also matters for compliance. If you’re in healthcare, finance, or anywhere with strict data residency rules, sending raw customer audio to a third-party voice AI platform is a liability. Running the stack yourself means you control where transcripts and recordings live.

Core Components of a Voice Agent Pipeline

A production AI voice agent for customer service is really four services chained together:

Telephony ingress — SIP trunk or WebRTC gateway that receives the call (Twilio, Telnyx, or a self-hosted Asterisk/FreeSWITCH box)

Speech-to-text (STT) — streaming transcription, typically Whisper-based (faster-whisper, whisper.cpp) or a hosted API like Deepgram

LLM orchestration — the actual conversation logic, intent detection, and function calling into your CRM or ticketing system

Text-to-speech (TTS) — converts the LLM’s response back to audio, using something like Coqui TTS, ElevenLabs, or Piper for a fully local option

Each of these can run in its own container, which makes the whole thing easier to scale independently. STT and TTS are GPU-hungry; the orchestration layer usually isn’t.

Containerizing the Stack with Docker

Here’s a minimal docker-compose.yml that stands up a local voice agent pipeline using open-source components. This assumes you have an NVIDIA GPU available for the STT/TTS containers.

version: '3.8'

services:
  stt:
    image: onerahmet/openai-whisper-asr-webservice:latest-gpu
    environment:
      - ASR_MODEL=base.en
      - ASR_ENGINE=faster_whisper
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "9000:9000"

  orchestrator:
    build: ./orchestrator
    environment:
      - LLM_API_URL=http://llm:8000
      - STT_URL=http://stt:9000
      - TTS_URL=http://tts:5002
    ports:
      - "8080:8080"
    depends_on:
      - stt
      - tts

  tts:
    image: ghcr.io/coqui-ai/tts
    command: --model_name tts_models/en/vctk/vits
    ports:
      - "5002:5002"

  llm:
    image: ollama/ollama:latest
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "8000:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  ollama_data:

Bring it up with:

docker compose up -d
docker compose logs -f orchestrator

The orchestrator service is your own code — it handles the WebSocket stream from the telephony layer, sends audio chunks to stt, feeds the transcript into the LLM with function-calling tools for things like “look up order status” or “schedule callback,” and pipes the response text to tts before streaming audio back to the caller.

Wiring Up Telephony

For the telephony leg, Twilio’s Media Streams API is the fastest path to production if you don’t want to run your own SIP infrastructure. It streams raw audio over a WebSocket to your orchestrator in real time, which is exactly what the pipeline above expects.

A minimal webhook handler in Python looks like this:

from fastapi import FastAPI, WebSocket
import base64
import json

app = FastAPI()

@app.websocket("/media-stream")
async def media_stream(websocket: WebSocket):
    await websocket.accept()
    async for message in websocket.iter_text():
        data = json.loads(message)
        if data["event"] == "media":
            audio_chunk = base64.b64decode(data["media"]["payload"])
            # forward audio_chunk to your STT service here
            pass

If you’d rather avoid Twilio’s per-minute fees entirely, FreeSWITCH and Asterisk are both viable for running your own SIP trunk, though you’ll need a VoIP provider for actual PSTN connectivity either way.

Latency Is the Whole Game

The single biggest failure mode in AI voice agents for customer service isn’t accuracy — it’s latency. Callers tolerate a 1-2 second pause before a response starts, but anything beyond that feels broken and people hang up or start talking over the agent.

To hit acceptable latency you need:

Streaming STT (partial transcripts as the caller speaks, not wait-for-silence batch transcription)

A small, fast LLM for the orchestration layer — a 7B-13B parameter model is usually enough for structured customer service flows, and it’s dramatically faster than routing every turn through a large frontier model

Streaming TTS that starts playing audio before the full response is generated

Co-locating STT, LLM, and TTS on the same host or same low-latency network to avoid round-trip overhead between services

Running these components on a GPU-backed VPS close to your telephony provider’s edge nodes matters more than most people expect. A provider like DigitalOcean with GPU droplets in the right region can shave real milliseconds off your round trip compared to a generic cloud region pick.

Monitoring and Observability

Voice agent pipelines fail in ways that are hard to catch with standard uptime checks — a service can be “up” while silently producing garbled transcripts or dead air. You need call-level observability, not just container health checks.

At minimum, log and alert on:

STT confidence scores per call segment

End-to-end response latency (time from caller stops speaking to audio starts playing back)

LLM function-call failures (e.g., CRM lookup timeouts)

Call abandonment rate mid-conversation

A tool like BetterStack works well here for aggregating logs across all four containers into one place and alerting when latency or error rates spike, without you having to build a custom dashboard from scratch.

For deeper infrastructure monitoring on the underlying hosts, see our guide on setting up Prometheus and Grafana on a Docker host — the same setup applies directly to a voice agent stack, you’re just adding custom metrics for call latency and STT confidence alongside standard container metrics.

Scaling Beyond a Single Host

Once you’re past a handful of concurrent calls, a single Docker host runs out of GPU capacity fast. The orchestrator service is stateless per-call and scales horizontally without much thought — just run multiple replicas behind a load balancer. STT and TTS are the bottleneck since they need GPU memory per active stream.

A reasonable scaling pattern:

# Scale the orchestrator to handle more concurrent calls
docker compose up -d --scale orchestrator=4

# Check GPU utilization to decide if you need another STT/TTS host
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 5

At real scale, teams typically move from docker compose to Kubernetes so they can autoscale STT/TTS pods based on GPU queue depth, and route calls through a message queue rather than direct WebSocket connections to a single orchestrator instance. That’s a bigger architectural jump and worth its own dedicated writeup — for most teams under a few hundred concurrent calls, a well-tuned multi-host Docker Compose setup is enough.

Bare-metal or dedicated GPU hosts from a provider like Hetzner are worth considering once you’re running sustained GPU load 24/7, since the hourly cost of cloud GPU instances adds up fast compared to a dedicated box you’re not spinning down.

Security and Data Handling

Customer service calls often contain PII — names, addresses, account numbers, sometimes payment details if callers aren’t careful about what they say. A few non-negotiables:

Encrypt call recordings and transcripts at rest

Set a retention policy and actually enforce it with a cron job or lifecycle rule, don’t just document it

Redact or mask sensitive data before it hits your LLM context if you’re using any external API for the orchestration layer

Run the telephony ingress behind a WAF if it’s exposed to the public internet — Cloudflare covers this well for WebSocket and HTTP traffic without much configuration overhead

If you’re routing any part of the pipeline through a third-party LLM API instead of a self-hosted model, read their data retention terms carefully. Some providers train on API traffic by default unless you opt out, which is a non-starter for customer PII.

Recommended: Want to explore DigitalOcean yourself? DigitalOcean is a direct vendor link (not an affiliate/tracked link).

FAQ

Q: Do I need a GPU to run AI voice agents for customer service?
A: For the STT and TTS components, yes, if you want real-time streaming performance. You can run whisper.cpp on CPU for STT, but latency will be noticeably worse under concurrent load. The LLM orchestration layer can run on CPU if you use a small quantized model, though a GPU is still faster.

Q: How much does self-hosting save compared to a managed platform like Vapi or Retell?
A: It depends heavily on call volume. Managed platforms typically charge $0.05-$0.15 per minute. At low volume (a few hundred calls a month) that’s cheaper than running dedicated GPU infrastructure. Past a few thousand calls a month, self-hosting on your own DevOps infrastructure usually wins on cost, but you’re trading dollars for engineering time.

Q: Can I use ChatGPT or Claude as the LLM in this pipeline?
A: Yes, via API calls from your orchestrator service. It’s simpler than self-hosting an LLM but adds network latency per turn and per-token cost. For high-volume, low-complexity flows (order status, FAQs), a self-hosted small model is usually faster and cheaper. For complex, open-ended conversations, a frontier model API is worth the tradeoff.

Q: What’s an acceptable end-to-end latency target?
A: Aim for under 800ms from when the caller stops speaking to when the agent’s audio starts playing. Above 1.5 seconds, callers start perceiving the agent as broken or start talking over it.

Q: How do I handle callers who want to speak to a human?
A: Build explicit intent detection for escalation phrases (“talk to a person,” “human agent”) into your orchestration logic, and have a hard fallback path to transfer the call via your telephony provider’s API rather than relying on the LLM to figure it out contextually every time.

Q: Is it legal to record and transcribe customer service calls with an AI agent?
A: Recording laws vary by jurisdiction — some US states require two-party consent. You need an explicit disclosure at the start of the call (“this call may be recorded and is handled by an automated assistant”) regardless of the tech stack, and you should confirm compliance requirements with legal counsel for your specific region before going live.

Wrapping Up

AI voice agents for customer service are genuinely useful infrastructure now, not hype, but the deployment reality is closer to running a real-time media pipeline than deploying a simple chatbot. Get the latency budget right, containerize each stage so you can scale independently, and don’t skip observability — silent failures in a voice pipeline are worse than a service just going down, because the caller experience degrades without any alert firing. Start small with a Docker Compose setup like the one above, measure real call latency, and scale the pieces that actually bottleneck under your traffic before reaching for Kubernetes.