How to Self-Host an AI Call Agent on Linux with Docker

Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

If you’ve spent any time researching customer support automation, you’ve probably run into the term AI call agent — a voice-driven system that answers, routes, and even resolves phone calls without a human on the line. Most vendors want you to rent their hosted platform at a per-minute rate. If you run Linux boxes for a living, you can build the same functionality yourself, keep your call data on infrastructure you control, and avoid the recurring SaaS bill.

This guide walks through a self-hosted AI call agent stack: SIP trunking with Asterisk, speech-to-text and text-to-speech pipelines, an LLM for reasoning, and Docker Compose to tie it all together. It assumes you’re comfortable with a Linux shell and basic networking.

Why Self-Host an AI Call Agent

Commercial AI call agent platforms are fast to set up but come with tradeoffs:

Per-minute or per-seat pricing that scales badly past a few hundred calls a month

Your call transcripts and customer data living on someone else’s servers

Limited ability to swap in a different LLM or fine-tune responses

Vendor lock-in on integrations and webhooks

A self-hosted stack costs you a VPS and some setup time, but you own the whole pipeline. If you already manage a Docker Compose stack for other services, adding a call agent is a natural extension.

Core Components of the Stack

An AI call agent isn’t one piece of software — it’s a pipeline. At minimum you need:

SIP/PBX layer — handles the actual phone call (we’ll use Asterisk)

Speech-to-text (STT) — converts caller audio to text in real time

LLM reasoning layer — decides what to say back, using tools/functions if needed

Text-to-speech (TTS) — converts the response back to audio

Telephony provider — a SIP trunk provider that connects to the public phone network (Twilio, Telnyx, or a local carrier)

Each of these runs as its own container, which keeps the system easy to update piece by piece without breaking the whole pipeline.

Setting Up the Docker Environment

Start with a clean VPS. A 4 vCPU / 8GB RAM instance is a reasonable starting point if you’re running a small local STT/TTS model alongside Asterisk. If you’re calling out to a hosted LLM API instead of running one locally, you can get away with less.

# update and install docker
sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Verify the install:

docker --version
docker compose version

Now lay out the project directory:

mkdir -p ~/ai-call-agent/{asterisk,stt,tts,agent}
cd ~/ai-call-agent

Docker Compose Configuration

Here’s a minimal docker-compose.yml that wires up the four core services. Each container is isolated on its own bridge network, with only the ports that need to be exposed actually exposed.

version: "3.9"

services:
  asterisk:
    image: andrius/asterisk:latest
    ports:
      - "5060:5060/udp"
      - "10000-10100:10000-10100/udp"
    volumes:
      - ./asterisk/conf:/etc/asterisk
    networks:
      - callagent

  stt:
    build: ./stt
    environment:
      - MODEL=whisper-small
    networks:
      - callagent

  tts:
    build: ./tts
    environment:
      - VOICE=en-us-standard
    networks:
      - callagent

  agent:
    build: ./agent
    depends_on:
      - stt
      - tts
    environment:
      - LLM_ENDPOINT=http://llm:8000/v1/chat/completions
    networks:
      - callagent

networks:
  callagent:
    driver: bridge

Bring the stack up:

docker compose up -d
docker compose logs -f agent

The agent service is where the actual decision logic lives — it receives transcribed text from stt, sends it to an LLM endpoint, and passes the response text to tts for synthesis. Whether that LLM endpoint is a local model or a hosted API is entirely your call agent’s choice.

Connecting a SIP Trunk

Asterisk needs a SIP trunk to actually receive calls from the public phone network. Providers like Twilio Elastic SIP Trunking or Telnyx work well here — you register your Asterisk server’s public IP or domain with the provider and configure a pjsip.conf endpoint:

[trunk]
type=endpoint
transport=transport-udp
context=from-trunk
disallow=all
allow=ulaw
allow=alaw
outer_call = yes

[trunk-auth]
type=auth
auth_type=userpass
username=your_sip_username
password=your_sip_password

[trunk-registration]
type=registration
transport=transport-udp
outer_call = yes
server_uri=sip:your-provider.com
client_uri=sip:[email protected]

Open UDP port 5060 and the RTP range (10000-10100 in the compose file above) on your firewall. If you’re behind a cloud provider’s security groups, make sure those match your ufw or iptables rules too — a mismatch here is the most common reason calls connect but produce no audio.

sudo ufw allow 5060/udp
sudo ufw allow 10000:10100/udp

Reasoning and Guardrails in the Agent Layer

The LLM layer is where most of the actual “intelligence” of an AI call agent lives, but it’s also where things go wrong if you don’t constrain it. A phone call is real-time and irreversible — unlike a chat interface, the caller can’t scroll back and re-read a bad answer. A few practices that matter in production:

Keep responses short. Long LLM outputs sound unnatural and increase perceived latency once run through TTS.

Set a hard timeout (2-3 seconds) on the LLM call and fall back to a canned response if it’s exceeded.

Log every transcript and response pair for review — this is also how you catch hallucinated answers before a customer complains.

Use function calling to hand off to a real human or a scripted flow for anything involving payments, account changes, or legal commitments.

Rate-limit inbound calls per phone number to reduce abuse from robocall probing.

If you’re already running a monitoring stack for your Docker services, add the call agent containers to it. Call latency and STT/TTS failure rates are exactly the kind of metrics you want alerting on before a customer notices dropped audio.

Hosting and Reliability Considerations

A phone system has different uptime expectations than a blog or internal dashboard — people expect the phone to just work. A few infrastructure choices make a real difference:

Run the stack on a VPS with a dedicated public IP so SIP registration stays stable

Use a provider with good network peering to your telephony vendor to minimize jitter

Put a monitoring agent in front of your container health checks so a crashed STT container doesn’t silently kill inbound audio

Consider running Asterisk behind Cloudflare for DDoS protection on any web-facing management UI, though the SIP/RTP traffic itself typically bypasses Cloudflare’s proxy

For the actual compute, a provider like DigitalOcean or Hetzner gives you predictable pricing and enough headroom to run local STT/TTS models without per-minute billing surprises. If you want managed uptime monitoring on top of your call agent stack, BetterStack’s status pages and incident alerting are worth checking out too.

Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Q: Do I need my own LLM, or can I call a hosted API?
A: Either works. A hosted API (OpenAI, Anthropic, etc.) is simpler to set up and requires less compute, but adds network latency and per-call cost. A locally hosted open-source model keeps everything on your own server but needs more RAM/GPU and more maintenance.

Q: How much does a self-hosted AI call agent cost compared to a SaaS platform?
A: Roughly the cost of a mid-size VPS ($40-80/month) plus your telephony provider’s per-minute rate, versus SaaS platforms that often charge $0.10-0.30 per minute on top of a monthly platform fee. Break-even usually happens somewhere around a few thousand minutes a month.

Q: What’s the biggest source of latency in this pipeline?
A: Usually the STT step, especially if you’re running a larger Whisper model on CPU only. Use a smaller model or GPU acceleration if you notice callers experiencing awkward pauses.

Q: Can this handle outbound calls too?
A: Yes — Asterisk supports originating calls through the same SIP trunk. You’d trigger an outbound call via the Asterisk AMI/ARI interface and route the audio through the same STT/TTS/agent pipeline.

Q: Is this legal for handling customer calls?
A: Generally yes, but call recording laws vary by jurisdiction (some US states require two-party consent). Make sure your agent’s opening message discloses that the call may be recorded and handled by an automated system.

Q: How do I scale this past one server?
A: Put Asterisk instances behind a SIP load balancer (like Kamailio) and run the STT/TTS/agent containers as a horizontally scaled pool behind an internal queue. Most teams don’t need this until they’re well past a few hundred concurrent calls.

Wrapping Up

Building your own AI call agent takes more upfront work than signing up for a SaaS platform, but if you’re already comfortable with Docker and Linux administration, it’s a very achievable weekend project — and one that pays for itself quickly at any real call volume. Start with the four-container stack above, get a single SIP trunk talking to it, and iterate on the agent’s prompt and guardrails from there.

If you’re setting this up alongside other self-hosted infrastructure, check out our guide on running a reverse proxy for internal services to keep your management interfaces off the public internet.