How to Deploy an AI Agent for Business on Self-Hosted Infrastructure
Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.
Every SaaS vendor now sells an “AI agent for business,” usually as a black-box API with a per-seat price tag and zero control over where your data goes. For developers and sysadmins who already run their own stacks, that’s a bad trade. You can run an equally capable agent on a VPS you control, using open-source tooling, for a fraction of the recurring cost — and you get full visibility into logs, prompts, and data flow.
This guide walks through the architecture, a working Docker Compose deployment, security hardening, and monitoring for a self-hosted AI agent for business use — things like internal support triage, document Q&A, and workflow automation.
What Is an AI Agent for Business, Really?
Strip away the marketing and an AI agent is a loop: an LLM receives a goal, decides on an action (call a tool, query a database, hit an API), observes the result, and repeats until the goal is satisfied or it hits a limit. For business use cases, that loop typically wraps around:
The frameworks doing the heavy lifting — LangChain, LlamaIndex, and CrewAI — are open source and framework-agnostic about which model runs behind them. That’s the leverage point: you’re not locked into a single vendor’s agent product.
Why Self-Host Instead of Using a SaaS Agent Platform
Three reasons come up in almost every infrastructure review:
The tradeoff is operational: you own the uptime, the patching, and the security posture. If you’re already comfortable running Docker containers in production, that overhead is manageable.
Architecture of a Self-Hosted AI Agent Stack
A minimal, production-viable stack looks like this:
Keeping each piece in its own container makes the stack portable across any VPS provider and easy to reproduce for staging environments.
Core Components in a docker-compose.yml
Here’s a working baseline you can adapt. It runs a local model server, a vector database, Redis for the task queue, and the agent API itself.
version: "3.9"
services:
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
restart: unless-stopped
qdrant:
image: qdrant/qdrant:latest
ports:
- "6333:6333"
volumes:
- qdrant_data:/qdrant/storage
restart: unless-stopped
redis:
image: redis:7-alpine
restart: unless-stopped
agent-api:
build: ./agent-api
environment:
- OLLAMA_HOST=http://ollama:11434
- QDRANT_HOST=http://qdrant:6333
- REDIS_URL=redis://redis:6379/0
ports:
- "8000:8000"
depends_on:
- ollama
- qdrant
- redis
restart: unless-stopped
volumes:
ollama_data:
qdrant_data:Deploying the Agent API
The agent-api service is a small FastAPI app that exposes an endpoint for triggering agent runs. A minimal version:
from fastapi import FastAPI
from pydantic import BaseModel
import httpx
app = FastAPI()
class AgentRequest(BaseModel):
query: str
@app.post("/run")
async def run_agent(req: AgentRequest):
async with httpx.AsyncClient() as client:
resp = await client.post(
"http://ollama:11434/api/generate",
json={"model": "llama3", "prompt": req.query, "stream": False},
timeout=60,
)
return resp.json()Pull the base model and bring the stack up:
docker compose up -d
docker exec -it $(docker compose ps -q ollama) ollama pull llama3
curl -X POST http://localhost:8000/run
-H "Content-Type: application/json"
-d '{"query": "Summarize the last 5 support tickets tagged billing"}'Step-by-Step Rollout Checklist
When moving this from a laptop test to something a team relies on, work through this order:
If you haven’t hardened a fresh Linux box before, our Linux server hardening checklist covers the baseline steps in more detail.
Securing an AI Agent for Business Use
Agents that can call internal tools are effectively privileged automation, and they should be treated with the same scrutiny as a service account. A few non-negotiables:
The OWASP Top 10 for LLM Applications is worth reading before you connect an agent to anything that writes data, not just reads it.
Monitoring and Scaling
Once the agent is handling real traffic, you need visibility into latency, error rates, and cost per run — especially if you’re metering calls to a paid model API. A basic Prometheus + Grafana setup exporting request duration and token counts from the FastAPI service gets you most of the way there; pair it with uptime alerting so you know immediately if the model backend goes down. Our guide on monitoring Docker containers with Prometheus and Grafana walks through wiring up the exporters for a stack like this one.
For scaling past a single VPS, split the model backend onto its own GPU-backed instance and keep the lightweight agent API and vector store on cheaper compute — there’s no reason to pay for GPU hours to run a FastAPI wrapper.
Choosing Infrastructure for Your Agent Stack
The model server is the resource-hungry piece; everything else is lightweight. A sensible split:
Splitting cost centers this way keeps your bill predictable instead of paying GPU rates for idle orchestration containers.
Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.
FAQ
Do I need a GPU to self-host an AI agent for business?
Not necessarily. Smaller quantized models (7B parameters) run acceptably on CPU with enough RAM, though responses will be slower. For production workloads with real-time expectations, a GPU instance or a hosted model API is usually worth the cost.
Is self-hosting actually cheaper than a SaaS AI agent platform?
At low volume, SaaS is often cheaper because you avoid infrastructure overhead. Past a few hundred agent runs per day, self-hosted infrastructure typically wins on cost, especially if you’re using an open-weight model instead of paying per-token for a hosted API.
Can I use a hosted model API instead of running Ollama locally?
Yes — swap the ollama service call in the agent API for a request to OpenAI, Anthropic, or another provider’s API. The rest of the architecture (vector store, task queue, tool-calling logic) stays the same.
How do I stop the agent from taking destructive actions on its own?
Add an explicit approval gate in the tool-calling layer for any action tagged as destructive, and require a human to confirm before the agent executes it. Never grant the agent’s API credentials write access it doesn’t need.
What’s the minimum team size where this is worth building?
If you already run a few production Docker services and have someone comfortable with basic Linux administration, the operational overhead is small. Below that, a managed SaaS agent product is probably the faster path.
How do I keep the vector store data in sync with our source documents?
Run a scheduled job (cron or Celery beat) that re-embeds changed documents and upserts them into Qdrant. For most internal knowledge bases, a nightly sync is frequent enough.
Wrapping Up
A self-hosted AI agent for business isn’t more complicated than any other containerized service you’re already running — it’s a model backend, a vector store, a task queue, and a thin API tying them together. The real work is in the operational discipline: scoping permissions tightly, logging every tool call, and putting humans in the loop for anything irreversible. Start with the Docker Compose baseline above, get it running against a low-stakes internal use case, and expand from there once you trust the logs.
Leave a Reply