Building an AI Customer Support Agent: A Self-Hosted Docker Guide

Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

Every SaaS founder and support-ops lead eventually hits the same wall: ticket volume grows faster than headcount. The default fix is bolting on a hosted AI helpdesk tool, but that means shipping customer data to a third party, paying per-seat or per-resolution fees, and accepting whatever rate limits the vendor gives you. If you run infrastructure for a living, there’s a better option: run your own ai customer support agent on hardware you control, using open models and a Docker Compose stack you can version, back up, and scale like any other service.

We’ll build a self-hosted support agent that:

Runs a local LLM via Ollama so you’re not paying per-token to OpenAI or Anthropic

Uses retrieval-augmented generation (RAG) against your own docs/knowledge base

Exposes a simple REST API your frontend or helpdesk widget can call

Ships logs and metrics so you can actually observe what the agent is doing

If you’ve already read our Docker Compose beginner’s guide, this will feel familiar — we’re layering an AI service on top of the same patterns.

Why Self-Host an AI Customer Support Agent

Three reasons come up constantly in DevOps forums and in our own client work:

Data residency and compliance. Support tickets often contain PII, account details, or proprietary product info. Keeping inference on infrastructure you own avoids sending that data to a third-party inference API.

Cost at scale. Per-resolution or per-seat SaaS pricing gets expensive once you’re handling thousands of tickets a month. A self-hosted GPU or CPU instance has a flat, predictable cost.

Control over behavior. You can fine-tune retrieval sources, swap models, and adjust prompts without waiting on a vendor’s roadmap.

The tradeoff is operational: you own uptime, patching, and scaling. If your team already runs Docker in production, this is a manageable lift — not a research project.

Architecture Overview

The stack has four moving parts:

LLM runtime — Ollama serving a quantized open model (Llama 3.1, Mistral, or Qwen2.5 all work well for support use cases)

Vector database — Qdrant or Chroma, storing embeddings of your help docs, FAQs, and past resolved tickets

Agent API — a lightweight Python (FastAPI) service that handles retrieval, prompt assembly, and calls the LLM

Reverse proxy — Nginx or Traefik terminating TLS and routing requests

All four run as containers on a single Docker Compose file, which makes the whole thing reproducible and easy to move between environments.

Prerequisites

You’ll need a VPS or bare-metal box with at least 8GB RAM (16GB+ recommended if you’re running a 7B-parameter model without a GPU), Docker Engine 24+, and Docker Compose v2.

# Verify Docker and Compose versions
docker --version
docker compose version

# Create the project directory
mkdir ai-support-agent && cd ai-support-agent
mkdir -p data/qdrant data/ollama

If you’re provisioning fresh infrastructure for this, a standard 4vCPU/8GB droplet is enough to prototype before you commit to GPU pricing. DigitalOcean droplets spin up in under a minute and let you resize once you know your real load.

Setting Up the Docker Compose Stack

Here’s the full docker-compose.yml for the four services described above:

version: "3.9"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ./data/ollama:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped

  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    volumes:
      - ./data/qdrant:/qdrant/storage
    ports:
      - "6333:6333"
    restart: unless-stopped

  agent-api:
    build: ./agent-api
    container_name: agent-api
    depends_on:
      - ollama
      - qdrant
    environment:
      - OLLAMA_HOST=http://ollama:11434
      - QDRANT_HOST=http://qdrant:6333
    ports:
      - "8000:8000"
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    container_name: agent-proxy
    depends_on:
      - agent-api
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    ports:
      - "80:80"
      - "443:443"
    restart: unless-stopped

Bring it up and pull a model:

docker compose up -d
docker exec -it ollama ollama pull llama3.1:8b

Building the Retrieval-Augmented Agent API

The agent API is where the actual “support agent” logic lives. It embeds a customer question, retrieves the most relevant chunks from your knowledge base, and feeds both into the LLM as context. A minimal FastAPI implementation looks like this:

# agent-api/main.py
from fastapi import FastAPI
from pydantic import BaseModel
import httpx
import os

app = FastAPI()
OLLAMA_HOST = os.environ["OLLAMA_HOST"]
QDRANT_HOST = os.environ["QDRANT_HOST"]

class Question(BaseModel):
    query: str

@app.post("/ask")
async def ask(question: Question):
    async with httpx.AsyncClient() as client:
        # 1. Embed the query (using a local embedding model via Ollama)
        embed_resp = await client.post(
            f"{OLLAMA_HOST}/api/embeddings",
            json={"model": "nomic-embed-text", "prompt": question.query},
        )
        vector = embed_resp.json()["embedding"]

        # 2. Search Qdrant for relevant support docs
        search_resp = await client.post(
            f"{QDRANT_HOST}/collections/support_docs/points/search",
            json={"vector": vector, "limit": 3, "with_payload": True},
        )
        context_chunks = [hit["payload"]["text"] for hit in search_resp.json()["result"]]
        context = "n---n".join(context_chunks)

        # 3. Ask the LLM with retrieved context injected
        prompt = f"Use the context below to answer the support question.nnContext:n{context}nnQuestion: {question.query}nAnswer:"
        chat_resp = await client.post(
            f"{OLLAMA_HOST}/api/generate",
            json={"model": "llama3.1:8b", "prompt": prompt, "stream": False},
        )
        return {"answer": chat_resp.json()["response"]}

This is intentionally minimal — no auth, no rate limiting, no caching. Treat it as a starting point, not something to expose to the public internet as-is.

Loading Your Knowledge Base into Qdrant

Before the agent can answer anything useful, you need to populate the vector database with your actual help docs, FAQ pages, and resolved ticket summaries:

# ingest.py — run once, or on a schedule when docs change
import httpx

docs = [
    "To reset your password, go to Settings > Security > Reset Password.",
    "Refunds are processed within 5-7 business days to the original payment method.",
    # ... load from your actual CMS/helpdesk export
]

for i, text in enumerate(docs):
    embed = httpx.post(
        "http://localhost:11434/api/embeddings",
        json={"model": "nomic-embed-text", "prompt": text},
    ).json()["embedding"]

    httpx.put(
        f"http://localhost:6333/collections/support_docs/points",
        json={"points": [{"id": i, "vector": embed, "payload": {"text": text}}]},
    )

Run this as a cron job or trigger it from your CI pipeline whenever your help center content changes, so the agent’s answers stay current.

Monitoring and Logging Your Agent

An AI support agent that hallucinates a wrong refund policy is worse than no agent at all, so observability isn’t optional here. At minimum, log every query, the retrieved context, and the generated answer so you can audit bad responses later.

Ship container logs to a central location with docker compose logs -f piped into your log aggregator, or mount a volume and use Filebeat/Vector

Track response latency and error rates — LLM calls are slow compared to typical API calls, and you’ll want alerts if the 95th percentile creeps past a few seconds

Set up uptime checks on the /ask endpoint itself, not just the reverse proxy

We rely on BetterStack for uptime monitoring and log management across our own self-hosted services — it’s a straightforward way to get alerting without standing up a full Prometheus/Grafana stack if you don’t already have one. If you do run Prometheus, the official Prometheus documentation has a solid guide for scraping custom FastAPI metrics with prometheus-fastapi-instrumentator.

For a deeper dive into building out a full observability stack, see our self-hosted monitoring stack guide.

Scaling and Production Hardening

Once the prototype works, a few changes matter before you point real customer traffic at it:

Put the agent-api behind authentication — an API key header is the minimum bar

Add request rate limiting in Nginx or Traefik to prevent abuse driving up compute costs

Move to a GPU instance if query volume makes CPU inference too slow; quantized 8B models are usable on CPU but a GPU cuts latency dramatically

Put Cloudflare in front of the public-facing endpoint for DDoS protection and TLS termination at the edge

Separate the vector database onto its own volume-backed storage so re-indexing doesn’t compete with inference for disk I/O

For the reverse proxy and TLS setup itself, our Traefik reverse proxy guide walks through automatic certificate renewal, which saves you from manually managing Let’s Encrypt certs for this stack.

Security Considerations

A support agent with RAG access to internal docs is effectively a service with read access to potentially sensitive data. Treat it accordingly:

Never index raw customer PII into the vector database — sanitize ticket exports before ingestion

Restrict the Qdrant and Ollama ports (6333, 11434) to the internal Docker network only; don’t expose them publicly

Rotate API keys used between your frontend and the agent-api service

Log prompts and responses for a retention window that matches your data policy, then purge

Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Do I need a GPU to run an AI customer support agent locally?
No. Quantized 7B–8B models run acceptably on CPU with 8-16GB of RAM, though latency will be higher — expect several seconds per response instead of sub-second. A GPU becomes worthwhile once you’re handling concurrent requests at scale.

Which open-source LLM works best for support use cases?
Llama 3.1 8B and Mistral 7B both perform well for straightforward Q&A grounded in retrieved context. If your support docs are highly technical, a larger model (13B+) or a fine-tuned variant will reduce hallucination.

How is this different from just using ChatGPT with a plugin?
A hosted API call sends your prompt (and any injected context, including customer data) to a third party. Self-hosting keeps that data inside your own infrastructure and removes per-token billing, at the cost of managing the infrastructure yourself.

Can this replace human support agents entirely?
For tier-1 questions with clear answers in your docs (password resets, billing policy, shipping times), yes, largely. Escalation logic for ambiguous or emotionally charged tickets should still route to a human.

How often should I re-index the knowledge base?
Any time your help docs or FAQ content changes meaningfully. Many teams run the ingestion script nightly via cron or trigger it from a CI job when the docs repo changes.

Is Docker Compose enough, or do I need Kubernetes?
Compose is fine for a single-node deployment handling moderate traffic. Move to Kubernetes only once you need multi-node scaling, rolling updates without downtime, or GPU scheduling across a cluster.

Wrapping Up

A self-hosted AI customer support agent isn’t a weekend toy project, but it’s also not the multi-month engineering effort it sounds like. With Docker Compose, Ollama, and a vector database, you get a working RAG-based agent in an afternoon — the real work is in curating your knowledge base and hardening the deployment before it touches real customer traffic. Start small, log everything, and expand the model and infrastructure once you know your actual query volume.