Each of these can run in its own container, which makes the whole thing easier to scale independently. STT and TTS are GPU-hungry; the orchestration layer usually isn’t.

Containerizing the Stack with Docker

Here’s a minimal docker-compose.yml that stands up a local voice agent pipeline using open-source components. This assumes you have an NVIDIA GPU available for the STT/TTS containers.

version: '3.8'

services:
  stt:
    image: onerahmet/openai-whisper-asr-webservice:latest-gpu
    environment:
      - ASR_MODEL=base.en
      - ASR_ENGINE=faster_whisper
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "9000:9000"

  orchestrator:
    build: ./orchestrator
    environment:
      - LLM_API_URL=http://llm:8000
      - STT_URL=http://stt:9000
      - TTS_URL=http://tts:5002
    ports:
      - "8080:8080"
    depends_on:
      - stt
      - tts

  tts:
    image: ghcr.io/coqui-ai/tts
    command: --model_name tts_models/en/vctk/vits
    ports:
      - "5002:5002"

  llm:
    image: ollama/ollama:latest
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "8000:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  ollama_data:

Bring it up with:

docker compose up -d
docker compose logs -f orchestrator

The orchestrator service is your own code — it handles the WebSocket stream from the telephony layer, sends audio chunks to stt, feeds the transcript into the LLM with function-calling tools for things like “look up order status” or “schedule callback,” and pipes the response text to tts before streaming audio back to the caller.

Wiring Up Telephony

For the telephony leg, Twilio’s Media Streams API is the fastest path to production if you don’t want to run your own SIP infrastructure. It streams raw audio over a WebSocket to your orchestrator in real time, which is exactly what the pipeline above expects.

A minimal webhook handler in Python looks like this:

from fastapi import FastAPI, WebSocket
import base64
import json

app = FastAPI()

@app.websocket("/media-stream")
async def media_stream(websocket: WebSocket):
    await websocket.accept()
    async for message in websocket.iter_text():
        data = json.loads(message)
        if data["event"] == "media":
            audio_chunk = base64.b64decode(data["media"]["payload"])
            # forward audio_chunk to your STT service here
            pass

If you’d rather avoid Twilio’s per-minute fees entirely, FreeSWITCH and Asterisk are both viable for running your own SIP trunk, though you’ll need a VoIP provider for actual PSTN connectivity either way.

Latency Is the Whole Game

The single biggest failure mode in AI voice agents for customer service isn’t accuracy — it’s latency. Callers tolerate a 1-2 second pause before a response starts, but anything beyond that feels broken and people hang up or start talking over the agent.

To hit acceptable latency you need:

Running these components on a GPU-backed VPS close to your telephony provider’s edge nodes matters more than most people expect. A provider like DigitalOcean with GPU droplets in the right region can shave real milliseconds off your round trip compared to a generic cloud region pick.

Monitoring and Observability

Voice agent pipelines fail in ways that are hard to catch with standard uptime checks — a service can be “up” while silently producing garbled transcripts or dead air. You need call-level observability, not just container health checks.

At minimum, log and alert on:

A tool like BetterStack works well here for aggregating logs across all four containers into one place and alerting when latency or error rates spike, without you having to build a custom dashboard from scratch.

For deeper infrastructure monitoring on the underlying hosts, see our guide on setting up Prometheus and Grafana on a Docker host — the same setup applies directly to a voice agent stack, you’re just adding custom metrics for call latency and STT confidence alongside standard container metrics.

Scaling Beyond a Single Host

Once you’re past a handful of concurrent calls, a single Docker host runs out of GPU capacity fast. The orchestrator service is stateless per-call and scales horizontally without much thought — just run multiple replicas behind a load balancer. STT and TTS are the bottleneck since they need GPU memory per active stream.

A reasonable scaling pattern:

# Scale the orchestrator to handle more concurrent calls
docker compose up -d --scale orchestrator=4

# Check GPU utilization to decide if you need another STT/TTS host
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 5

At real scale, teams typically move from docker compose to Kubernetes so they can autoscale STT/TTS pods based on GPU queue depth, and route calls through a message queue rather than direct WebSocket connections to a single orchestrator instance. That’s a bigger architectural jump and worth its own dedicated writeup — for most teams under a few hundred concurrent calls, a well-tuned multi-host Docker Compose setup is enough.

Bare-metal or dedicated GPU hosts from a provider like Hetzner are worth considering once you’re running sustained GPU load 24/7, since the hourly cost of cloud GPU instances adds up fast compared to a dedicated box you’re not spinning down.

Security and Data Handling

Customer service calls often contain PII — names, addresses, account numbers, sometimes payment details if callers aren’t careful about what they say. A few non-negotiables:

If you’re routing any part of the pipeline through a third-party LLM API instead of a self-hosted model, read their data retention terms carefully. Some providers train on API traffic by default unless you opt out, which is a non-starter for customer PII.

Recommended: Want to explore DigitalOcean yourself? DigitalOcean is a direct vendor link (not an affiliate/tracked link).

FAQ

Q: Do I need a GPU to run AI voice agents for customer service?
A: For the STT and TTS components, yes, if you want real-time streaming performance. You can run whisper.cpp on CPU for STT, but latency will be noticeably worse under concurrent load. The LLM orchestration layer can run on CPU if you use a small quantized model, though a GPU is still faster.

Q: How much does self-hosting save compared to a managed platform like Vapi or Retell?
A: It depends heavily on call volume. Managed platforms typically charge $0.05-$0.15 per minute. At low volume (a few hundred calls a month) that’s cheaper than running dedicated GPU infrastructure. Past a few thousand calls a month, self-hosting on your own DevOps infrastructure usually wins on cost, but you’re trading dollars for engineering time.

Q: Can I use ChatGPT or Claude as the LLM in this pipeline?
A: Yes, via API calls from your orchestrator service. It’s simpler than self-hosting an LLM but adds network latency per turn and per-token cost. For high-volume, low-complexity flows (order status, FAQs), a self-hosted small model is usually faster and cheaper. For complex, open-ended conversations, a frontier model API is worth the tradeoff.

Q: What’s an acceptable end-to-end latency target?
A: Aim for under 800ms from when the caller stops speaking to when the agent’s audio starts playing. Above 1.5 seconds, callers start perceiving the agent as broken or start talking over it.

Q: How do I handle callers who want to speak to a human?
A: Build explicit intent detection for escalation phrases (“talk to a person,” “human agent”) into your orchestration logic, and have a hard fallback path to transfer the call via your telephony provider’s API rather than relying on the LLM to figure it out contextually every time.

Q: Is it legal to record and transcribe customer service calls with an AI agent?
A: Recording laws vary by jurisdiction — some US states require two-party consent. You need an explicit disclosure at the start of the call (“this call may be recorded and is handled by an automated assistant”) regardless of the tech stack, and you should confirm compliance requirements with legal counsel for your specific region before going live.

Wrapping Up

AI voice agents for customer service are genuinely useful infrastructure now, not hype, but the deployment reality is closer to running a real-time media pipeline than deploying a simple chatbot. Get the latency budget right, containerize each stage so you can scale independently, and don’t skip observability — silent failures in a voice pipeline are worse than a service just going down, because the caller experience degrades without any alert firing. Start small with a Docker Compose setup like the one above, measure real call latency, and scale the pieces that actually bottleneck under your traffic before reaching for Kubernetes.