ThinkStreamTV — Blog

Blog

  • Ai Agents Use Cases

    AI Agents Use Cases: A Practical Guide for DevOps Teams

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    Understanding real AI agents use cases helps engineering teams decide where autonomous automation actually earns its keep versus where a simple script or workflow tool is enough. This guide walks through concrete, self-hostable patterns for deploying AI agents in production infrastructure, along with the operational tradeoffs each pattern brings.

    AI agents are no longer confined to chatbots. They now handle customer support triage, monitor infrastructure, generate content pipelines, and orchestrate multi-step workflows that used to require a human in the loop at every stage. For DevOps teams, the interesting question isn’t whether agents work — it’s which use cases justify the added operational complexity of running one.

    What Makes an AI Agent Different From a Script

    Before diving into ai agents use cases, it’s worth being precise about terminology. A traditional script executes a fixed sequence of steps. An AI agent, by contrast, uses a language model to decide which steps to take, in what order, based on the current state of the world and a defined goal.

    This distinction matters operationally. A script fails predictably — you know exactly which line broke. An agent can fail in more varied ways: it might call the wrong tool, misinterpret a response, or loop on a task it can’t complete. That’s why teams evaluating ai agents use cases need to think about observability and guardrails from day one, not as an afterthought.

    Core Components of an Agent System

    Most production agent deployments share the same basic architecture:

  • A language model (hosted or self-hosted) that performs reasoning
  • A set of tools or APIs the agent can call
  • A memory or state store for context across steps
  • An orchestration layer that manages the loop between reasoning and action
  • Logging and monitoring for every decision the agent makes

    • If you’re setting up this stack for the first time, How to Create an AI Agent: A Developer’s Guide walks through the initial build from scratch, and How to Build Agentic AI: A Developer’s Guide covers the broader architectural patterns behind multi-step reasoning loops.

    Where Agents Fit in an Existing Stack

    Agents rarely replace an entire system. More often, they sit alongside existing services, triggered by a webhook, a queue message, or a scheduled job, and they call out to your existing APIs rather than reinventing them. This makes container orchestration a natural fit: an agent process can run as its own service, scaled independently, with its own resource limits and restart policy.

    Common AI Agents Use Cases in Production

    The most durable ai agents use cases share a common trait: a task that requires judgment across variable inputs, but where the cost of an occasional wrong decision is low and recoverable.

    Customer Support and Ticket Triage

    Support automation is one of the most mature ai agents use cases today. An agent can read an incoming ticket, classify its urgency, pull relevant account data, and either draft a response or route it to the right human team. This works well because the agent’s mistakes are cheap to catch — a human reviewer or a confidence threshold can catch misroutes before they cause harm. For a full deployment walkthrough, see Customer Service AI Agents: Self-Hosted Deployment Guide and Customer Support AI Agent: Self-Hosted Docker Guide.

    Infrastructure Monitoring and Incident Response

    Agents can watch logs, metrics, and alerts, correlate them against known incident patterns, and either take a predefined remediation action or escalate with a structured summary. This is distinct from a static alerting rule because the agent can reason across multiple signals — say, a spike in error rate combined with a recent deploy — before deciding whether to page someone.

    Data Analysis and Reporting

    Feeding an agent structured or semi-structured data and asking it to summarize trends, flag anomalies, or generate a report is a low-risk, high-value use case. The agent doesn’t need write access to production systems, which limits the blast radius if it makes a mistake. See AI Agents for Data Analysis: A DevOps Guide for patterns specific to pipeline-driven analytics.

    Content and SEO Pipelines

    Agents can drive multi-stage content pipelines — drafting, scoring against SEO criteria, and queuing for review — without a human writing every prompt by hand. This is one of the ai agents use cases where orchestration tools like n8n pair naturally with an agent’s reasoning step, since the workflow engine handles the deterministic parts (fetching data, writing to a sheet, publishing) while the agent handles the judgment calls. Automated SEO: A DevOps Pipeline for Site Monitoring and SEO AI Agent: Build & Deploy One with Docker both cover this pattern in more depth.

    Recruitment and HR Screening

    Screening resumes, scheduling interviews, and answering candidate questions are repetitive, judgment-light tasks well suited to agents, provided the system logs its reasoning for auditability. AI Recruitment Agents: Self-Hosted Deployment Guide covers a self-hosted approach to this use case.

    Building and Orchestrating Agents With Workflow Tools

    Not every agent needs a custom-built orchestration loop. Workflow automation platforms like n8n let you wire an LLM call into a broader pipeline — fetching data, calling APIs, writing results to a database — without hand-rolling the plumbing yourself. This is a practical middle ground between a fully custom agent framework and a single prompt-and-response API call.

    How to Build AI Agents With n8n: Step-by-Step Guide is a good starting point if you already run n8n for other automation. If you’re deciding between orchestration tools generally, n8n vs Make: Workflow Automation Comparison Guide 2026 compares the two most common options for teams building agent-adjacent workflows.

    A Minimal Self-Hosted Agent Loop Example

    Below is a simplified example of a containerized agent worker that polls a task queue and calls an LLM API to decide the next action. This is illustrative of the pattern, not a full production implementation:

    # docker-compose.yml
version: "3.8"
services:
  agent-worker:
    build: ./agent
    restart: unless-stopped
    environment:
      - QUEUE_URL=redis://redis:6379/0
      - MODEL_API_KEY=${MODEL_API_KEY}
    depends_on:
      - redis
    deploy:
      resources:
        limits:
          memory: 512M

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    volumes:
      - redis-data:/data

volumes:
  redis-data:

    Running the worker with a resource limit and a restart policy is a small but important detail — agent loops that hang or retry aggressively can consume unexpected CPU or API budget if left unconstrained. Combining this with a Redis-backed queue lets you inspect and replay tasks the agent got wrong, which is invaluable for debugging.

    docker compose up -d
docker compose logs -f agent-worker

    If your agent stack grows into multiple services, reviewing Docker Compose Rebuild: Complete Guide & Best Tips and Docker Compose Logs: The Complete Debugging Guide will save time when you need to iterate on the agent container without tearing down dependent services.

    Choosing the Right Model and Hosting Approach

    Deciding between a hosted API model and a self-hosted model is one of the first infrastructure decisions in any ai agents use cases evaluation. Hosted APIs like OpenAI’s are simpler to integrate but introduce a per-call cost and an external dependency. If you’re budgeting a project around a hosted model, OpenAI API Pricing: A Developer’s Cost Guide 2026 and OpenAI API Cost: Pricing Breakdown & Ways to Cut Spend are useful references, and OpenAI API Reference: Complete Developer Guide 2026 documents the actual request/response shapes you’ll be building against.

    Self-Hosting Considerations

    Self-hosting a model gives you control over data residency and predictable compute cost, at the price of managing GPU infrastructure yourself. For teams already running a VPS-based stack, this is worth weighing against simply keeping the model call external and self-hosting only the orchestration layer. Unmanaged VPS Hosting: A Practical Guide for Devs is a reasonable starting point if you’re evaluating where to run the surrounding services. A provider like DigitalOcean or Hetzner can work for the orchestration and queue layer even if the model inference itself stays hosted externally.

    Security and Access Control

    Any agent with write access to production systems needs the same access-control discipline as a human operator — scoped API keys, audit logging, and a clear boundary on what tools it’s allowed to call. AI Agent Security: A Practical Guide for DevOps covers this in more detail, and it’s worth reading before granting an agent any destructive capability (deleting records, sending emails, modifying infrastructure).

    Evaluating Whether an Agent Is the Right Tool

    Not every automation problem needs an agent. Before committing to one of these ai agents use cases, it helps to ask a few questions:

  • Does the task require judgment across variable, unpredictable inputs, or is the logic actually deterministic?
  • Is the cost of an occasional wrong decision low and recoverable?
  • Can you log and audit every decision the agent makes?
  • Is there a simpler workflow-automation solution (a fixed pipeline, a rules engine) that would solve 80% of the problem without the added complexity?

    • If the answer to the last question is yes, a deterministic workflow — built in something like n8n — is often the more maintainable choice. Agents earn their complexity when the task genuinely can’t be reduced to a fixed set of rules.


    Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Do AI agents need a dedicated framework to run in production?
    No. A framework can help structure the reasoning loop, but a simple worker process that calls a model API, executes the chosen tool, and logs the result is enough for most ai agents use cases. Frameworks add value mainly when you need multi-agent coordination or complex memory management.

    What’s the biggest operational risk with deploying agents?
    Unbounded retries or tool calls that consume compute or API budget without a clear ceiling. Always set resource limits, timeouts, and a maximum step count per task.

    Can AI agents run entirely self-hosted, without calling an external API?
    Yes, if you self-host the underlying language model. This adds GPU infrastructure to manage but removes the external API dependency and per-call cost, which matters for high-volume ai agents use cases.

    How do I decide between a workflow tool and a custom agent?
    Start with a workflow tool if the task is mostly deterministic with one or two decision points. Move to a custom agent loop only if the task requires ongoing multi-step reasoning that a fixed pipeline can’t express cleanly.

    Conclusion

    The strongest ai agents use cases share the same pattern: variable inputs, low-risk decisions, and a clear audit trail. Support triage, infrastructure monitoring, data analysis, content pipelines, and recruitment screening all fit this profile well. Before building a custom agent framework, evaluate whether a workflow tool like n8n already covers most of the requirement — agents are worth the added complexity only when genuine judgment is required at each step. For further reading on the underlying model APIs and orchestration platforms, the official n8n documentation and Kubernetes documentation are both solid references for the infrastructure side of these deployments.

    Related articles:

  • AI Agent Company: Build vs Buy and Self-Hosting Guide

    What an AI Agent Company Actually Builds (And How to Deploy It Yourself)

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    Every week another ai agent company shows up promising to automate your support desk, your DevOps pipeline, or your entire back office with a fleet of autonomous LLM agents. Some of these companies are genuinely shipping useful infrastructure. Others are a thin wrapper around a single API call and a marketing deck. If you’re a developer or sysadmin evaluating whether to hire one, buy a platform, or just build the thing yourself, this article breaks down what’s actually under the hood — and shows you how to stand up a comparable stack on your own infrastructure.

    What Is an AI Agent Company?

    Strip away the branding and an AI agent company is, at its core, selling three things: an orchestration layer that chains LLM calls into multi-step workflows, a set of tool integrations (search, code execution, CRM, ticketing systems), and a hosting/ops layer that keeps all of it running reliably. The “agent” is really a loop: the model reasons about a goal, picks a tool, executes it, observes the result, and repeats until the task is done or it hits a limit.

    The pitch from most vendors is that you don’t have to build this loop yourself. In practice, the loop itself is the easy part — frameworks like LangChain and its agent abstractions have made that trivial. The hard part, and the part you’re actually paying for, is everything around the loop: rate limiting, retry logic, secrets management, observability, and guardrails that stop an agent from doing something expensive or destructive.

    Core Components of an AI Agent Stack

    When you look at what these companies deploy behind the scenes, the architecture is fairly consistent across vendors:

  • Model layer — one or more LLM providers (OpenAI, Anthropic, or a self-hosted open-weight model) accessed through a unified API gateway.
  • Orchestration layer — the agent loop itself, usually built on a framework, handling planning, tool selection, and memory.
  • Tool/action layer — connectors to external systems: web search, code sandboxes, databases, ticketing tools, webhooks.
  • State and memory store — typically a vector database plus a relational store for conversation history and task state.
  • Ops layer — logging, monitoring, cost tracking, and human-in-the-loop approval steps for risky actions.

    • If you can stand up all five of these on your own servers, you’ve effectively replicated the product. The remaining question is whether it’s worth your time to do that versus paying a vendor’s monthly fee.

    Build vs. Buy: When Hiring an AI Agent Company Makes Sense

    Hiring a vendor makes sense when you need something running next week and don’t have spare engineering capacity. It also makes sense if your use case genuinely needs the polish a vendor has built — things like fine-tuned guardrails for regulated industries, or pre-built connectors to dozens of SaaS tools you’d otherwise have to write integrations for yourself.

    Building it yourself makes sense when:

  • You already run infrastructure in-house and have a DevOps team comfortable with containers.
  • Your agent workloads touch sensitive data you don’t want passing through a third party.
  • You need custom tool integrations that no vendor offers out of the box.
  • Cost at scale matters — a self-hosted stack on a VPS is often a fraction of a per-seat SaaS agent platform once you’re running more than a handful of agents continuously.

    • For teams that already manage their own Docker Compose stacks, the self-hosted route is usually not that much extra work — you’re mostly reusing infrastructure patterns you already know.

    Self-Hosting Your Own AI Agent Infrastructure

    If you decide to build rather than buy, the good news is that the components are all things sysadmins already know how to run: containers, a reverse proxy, a database, and a message queue. Here’s a minimal but production-oriented setup.

    Docker Compose Setup for an Agent Runtime

    Below is a working docker-compose.yml for a self-hosted agent stack: a Python agent runtime, Redis for short-term state, Postgres with pgvector for memory, and a reverse proxy.

    version: "3.9"

services:
  agent-runtime:
    build: ./agent
    container_name: agent-runtime
    restart: unless-stopped
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - REDIS_URL=redis://redis:6379/0
      - DATABASE_URL=postgresql://agent:${DB_PASSWORD}@postgres:5432/agentdb
    depends_on:
      - redis
      - postgres
    networks:
      - agent-net

  redis:
    image: redis:7-alpine
    container_name: agent-redis
    restart: unless-stopped
    volumes:
      - redis-data:/data
    networks:
      - agent-net

  postgres:
    image: ankane/pgvector:latest
    container_name: agent-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_USER=agent
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=agentdb
    volumes:
      - pg-data:/var/lib/postgresql/data
    networks:
      - agent-net

  caddy:
    image: caddy:2-alpine
    container_name: agent-proxy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy-data:/data
    networks:
      - agent-net

networks:
  agent-net:
    driver: bridge

volumes:
  redis-data:
  pg-data:
  caddy-data:

    The agent runtime itself is a small Python service. A minimal loop using LangChain’s agent executor looks like this:

    from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from tools import search_tool, db_lookup_tool

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are an ops assistant. Use tools to answer accurately."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

tools = [search_tool, db_lookup_tool]
agent = create_openai_tools_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, max_iterations=6)

if __name__ == "__main__":
    result = executor.invoke({"input": "Check disk usage on prod-db-01 and summarize"})
    print(result["output"])

    Bring the stack up with:

    docker compose up -d --build
docker compose logs -f agent-runtime

    Networking, Secrets, and a Reverse Proxy

    Don’t expose the agent runtime directly to the internet. Put it behind a reverse proxy that terminates TLS and, ideally, sits in front of an authentication layer if the agent has a web-facing endpoint. Caddy is a good default here because it handles Let’s Encrypt certificates automatically:

    agent.yourdomain.com {
    reverse_proxy agent-runtime:8000
    basicauth /admin/* {
        admin JDJhJDEwJDdEUFRDZ...
    }
}

    Keep API keys and database credentials in an .env file excluded from version control, or better, in a secrets manager. If you’re already running a Traefik or Nginx reverse proxy setup elsewhere on your infrastructure, it’s usually simpler to fold the agent stack into that existing proxy rather than standing up a second one.

    Choosing Infrastructure for Production AI Agents

    Once the stack is running, the next question is where to host it. Agent workloads are bursty — mostly idle, then a spike of API calls and tool executions when a task runs — so you want infrastructure that’s cheap at idle but doesn’t choke under a burst.

    Compute, GPU, and Cost Considerations

    If you’re only calling hosted model APIs (OpenAI, Anthropic) and not running your own model weights, you don’t need a GPU box at all — a standard VPS handles the orchestration layer fine, since the heavy lifting happens on the provider’s infrastructure. This is the setup most self-hosted agent stacks actually need:

  • 2-4 vCPU / 4-8GB RAM is plenty for the orchestration layer, Redis, and Postgres for low-to-moderate agent volume.
  • Object storage or a mounted volume for logs and any documents the agent ingests.
  • A GPU instance only becomes necessary if you’re self-hosting an open-weight model (e.g., Llama or Mistral variants) instead of calling a hosted API.

    • For most teams, a mid-tier droplet from DigitalOcean is enough to run the entire orchestration and memory layer, with the model calls themselves billed separately by the LLM provider. If you want European data residency or better price-per-core for the database and Redis layers, Hetzner cloud instances are worth pricing out — they’re consistently cheaper for CPU-bound workloads like this.

    Monitoring, Logging, and Uptime

    An agent that silently fails is worse than one that fails loudly — you need to know when a tool call errors out, when the model starts looping, or when costs spike unexpectedly. At minimum, instrument:

  • Request/response logging for every LLM call, including token counts.
  • Tool execution logs with success/failure status.
  • Cost tracking per agent run, aggregated daily.
  • Uptime checks on the agent’s public endpoint if it’s user-facing.

    • If you already run a Prometheus and Grafana monitoring stack, export agent metrics into it rather than building a separate dashboard. For uptime and alerting specifically, a service like BetterStack will page you the moment the agent endpoint stops responding, which matters more than it sounds like once an agent is wired into anything customer-facing.

    Security Considerations for AI Agent Deployments

    Agents that can execute code, hit databases, or send emails are a real attack surface, not a toy. Before putting one in production:

  • Sandbox any code-execution tool — never let the agent run arbitrary shell commands on the host it lives on.
  • Scope API keys and database credentials to the minimum permissions the agent actually needs.
  • Log every tool invocation so you can audit what the agent did, not just what it said.
  • Put a human-approval step in front of any action with real-world consequences (sending money, deleting records, emailing customers).
  • Rate-limit and cap max_iterations so a bad prompt can’t spin the agent into an expensive infinite loop.

    • If your agents sit behind a public domain, it’s also worth putting Cloudflare in front of the reverse proxy for basic DDoS protection and WAF rules — cheap insurance for something that’s directly reachable from the internet.

    Running this on a Linux host also means the usual hardening rules apply: keep the kernel and container runtime patched, disable password SSH auth, and restrict the agent’s outbound network access to only the APIs it actually needs to call.

    Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Is an AI agent company worth it for a small team?
    If you need something running this week and don’t have spare engineering time, yes. If you already run Docker infrastructure and have a few days to spend, self-hosting the equivalent stack is usually cheaper within a couple of months.

    What’s the real cost difference between buying and self-hosting?
    Most vendor pricing is per-seat or per-agent monthly, often $50-500/month per active agent. A self-hosted stack on a $20-40/month VPS can run several agents, with your only variable cost being the underlying LLM API calls, which you’d pay either way.

    Do I need a GPU to self-host an AI agent?
    Only if you’re running your own open-weight model instead of calling a hosted API like OpenAI or Anthropic. For orchestration-only stacks, a standard CPU VPS is enough.

    Can I run this on a Raspberry Pi or small home server?
    For low-volume, non-critical agents, yes — the orchestration layer is lightweight. For anything customer-facing or business-critical, use a proper VPS with backups and monitoring instead.

    How do I stop an agent from doing something destructive?
    Scope credentials tightly, sandbox code execution, cap iteration counts, and require human approval for irreversible actions like deletions, payments, or outbound emails.

    What framework should I start with if I’m building my own agent?
    LangChain and its agent executor are the most documented starting point, with a large ecosystem of tool integrations. For simpler needs, a hand-rolled loop calling the OpenAI or Anthropic API directly is often easier to reason about and debug.

    Whether you end up hiring an ai agent company or standing up the stack yourself, the underlying infrastructure decisions are the same ones you already make for any production service: containerize it, put it behind a proxy, monitor it, and lock down what it’s allowed to touch. The “AI” part is mostly a new client for infrastructure you already know how to run.

    Related articles:

  • Agentic AI Workflow: A DevOps Guide to Autonomous AI

    Agentic AI Workflow: A Practical Guide for DevOps Teams

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    Every few years a new abstraction shows up that promises to change how software gets built. Right now it’s the agentic AI workflow — a pattern where large language models don’t just answer prompts, they plan, call tools, inspect results, and loop until a goal is met. For DevOps teams, this isn’t hype; it’s a practical way to automate the messy, decision-heavy tasks that scripts alone can’t handle, like triaging alerts, writing remediation PRs, or summarizing incident timelines.

    This guide walks through what an agentic AI workflow actually is, how to design one, how to containerize and deploy it with Docker, and how to keep it observable once it’s running in production. Every code sample here is meant to run, not just illustrate a concept.

    What Is an Agentic AI Workflow?

    An agentic AI workflow is a pipeline where an LLM acts as a controller: it receives a goal, decides which tool or function to call next, evaluates the output, and repeats until the goal is satisfied or a stop condition is hit. This is fundamentally different from a single prompt-response API call. You’re not asking the model one question — you’re giving it a loop, a toolbox, and a way to check its own work.

    Core Components of an Agentic System

    Most agentic workflows share the same building blocks, regardless of framework:

  • Planner/controller loop — the LLM decides the next action based on current state
  • Tool registry — a set of functions the model can call (shell commands, API calls, database queries)
  • Memory/state store — short-term context plus longer-term memory (often a vector store or Redis)
  • Executor — the runtime that actually invokes tools and returns results to the model
  • Guardrails — validation, timeouts, and approval gates that stop the agent from doing something destructive

    • If you’ve worked with CI/CD pipelines, think of the planner as a dynamic version of a pipeline YAML file — except the “stages” are decided at runtime by the model instead of being hardcoded in advance.

    Agentic AI Workflow vs. Traditional Automation Pipelines

    A traditional automation pipeline (think Jenkins, GitHub Actions, or a bash script triggered by cron) executes a fixed sequence of steps. It’s deterministic and predictable, which is exactly what you want for something like a deployment pipeline. An agentic workflow trades some of that determinism for flexibility: the agent can branch, retry with a different approach, or ask a clarifying question before proceeding.

    The practical takeaway is that you shouldn’t replace your deployment pipeline with an agent. You should use agentic workflows for the parts of your operations that currently require a human to read logs, make a judgment call, and take an action — log triage, root-cause analysis, dependency upgrade PRs, or writing the first draft of a postmortem.

    Designing Your First Agentic AI Workflow

    Before writing any code, define three things: the goal the agent is trying to achieve, the tools it’s allowed to use, and the conditions under which it must stop and hand control back to a human. Skipping this step is the number one reason agentic pipelines spiral into expensive, useless tool-calling loops.

    Choosing an Orchestration Framework

    You don’t need to build the loop from scratch. LangGraph and CrewAI are the two most common choices for orchestrating multi-step, multi-tool agent behavior in Python. Here’s a minimal LangGraph-style agent that watches for failed Docker health checks and proposes a fix:

    from langgraph.graph import StateGraph, END
from typing import TypedDict
import subprocess

class AgentState(TypedDict):
    container: str
    logs: str
    diagnosis: str
    action_taken: bool

def fetch_logs(state: AgentState) -> AgentState:
    result = subprocess.run(
        ["docker", "logs", "--tail", "100", state["container"]],
        capture_output=True, text=True, timeout=10
    )
    state["logs"] = result.stdout + result.stderr
    return state

def diagnose(state: AgentState) -> AgentState:
    # In production this calls your LLM of choice (OpenAI, Anthropic, local model)
    # with the logs and a system prompt describing the diagnosis task.
    state["diagnosis"] = "OOMKilled: container exceeded memory limit"
    return state

def remediate(state: AgentState) -> AgentState:
    if "OOMKilled" in state["diagnosis"]:
        subprocess.run([
            "docker", "update", "--memory", "512m", state["container"]
        ])
        state["action_taken"] = True
    return state

graph = StateGraph(AgentState)
graph.add_node("fetch_logs", fetch_logs)
graph.add_node("diagnose", diagnose)
graph.add_node("remediate", remediate)
graph.set_entry_point("fetch_logs")
graph.add_edge("fetch_logs", "diagnose")
graph.add_edge("diagnose", "remediate")
graph.add_edge("remediate", END)

app = graph.compile()
result = app.invoke({"container": "api-service", "logs": "", "diagnosis": "", "action_taken": False})
print(result)

    This is deliberately simple — a real deployment would swap the hardcoded diagnose function for an actual LLM call, and would add a guardrail node that requires human approval before remediate runs against production containers. We cover container resource limits in more depth in our Docker Compose guide if you want the full breakdown of memory and CPU constraints.

    Wiring Tools and Function Calling

    Most providers now support native function/tool calling, which is cleaner than parsing free-text output. Here’s a bare-bones example using the OpenAI-compatible tool-calling format:

    import openai

tools = [
    {
        "type": "function",
        "function": {
            "name": "restart_container",
            "description": "Restarts a Docker container by name",
            "parameters": {
                "type": "object",
                "properties": {
                    "container_name": {"type": "string"}
                },
                "required": ["container_name"]
            }
        }
    }
]

response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "The nginx container keeps returning 502s, fix it."}],
    tools=tools,
    tool_choice="auto"
)

print(response.choices[0].message.tool_calls)

    The model decides whether restart_container is the right call and supplies the arguments. Your job is to validate those arguments before actually executing anything — never pipe model output directly into a shell command without a whitelist check.

    Deploying Agentic Workflows with Docker

    Containerizing your agent keeps its dependencies, API keys, and tool access isolated from the host — which matters a lot when the agent has permission to run shell commands. A minimal setup looks like this:

    FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Run as a non-root user — the agent should never run as root
RUN useradd -m agent
USER agent

CMD ["python", "agent_runner.py"]

    And a docker-compose.yml that pairs the agent with a Redis instance for state/memory:

    services:
  agent:
    build: .
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 512M

  redis:
    image: redis:7-alpine
    volumes:
      - redis-data:/data

volumes:
  redis-data:

    Notice the memory limit and restart: unless-stopped policy. Agentic loops that go wrong tend to go wrong repeatedly and quickly — a runaway tool-calling loop can burn through API credits or hammer a downstream service in minutes if it’s not bounded. Set hard limits on iteration count in your agent code, not just on the container.

    Hosting and Scaling Considerations

    Agentic workflows are typically light on CPU but can be memory-hungry if you’re running local embeddings or a vector store alongside the agent. For most small-to-medium agentic pipelines, a modestly sized VPS handles the job fine — you don’t need a GPU unless you’re self-hosting the LLM itself rather than calling a hosted API.

    If you’re setting this up from scratch, DigitalOcean droplets are a solid starting point for running the agent container plus Redis, with predictable pricing as you scale tool-call volume. For workloads where you want more raw compute per dollar — useful if you’re batching a lot of embedding generation — Hetzner cloud instances tend to be more cost-efficient. Both providers work fine with the Docker Compose setup above with no changes needed.

    We go deeper on picking a host for self-hosted AI workloads in our self-hosted LLM VPS guide, including a breakdown of memory requirements per model size.

    Cost Control for Agentic Loops

    Unbounded agent loops are the single biggest cost risk in this pattern. A few concrete guardrails:

  • Set a hard max-iteration count (10-15 is usually plenty for infra tasks)
  • Log every tool call with its token cost so you can audit spend after the fact
  • Add a circuit breaker that halts the agent if the same tool is called 3+ times with similar arguments
  • Use a cheaper model for the planning/diagnosis step and reserve the expensive model for final decisions

    • Monitoring and Observability for Agentic Pipelines

    You cannot debug an agentic workflow by staring at logs after the fact — you need structured tracing of every decision the agent made, what tool it called, and what the result was. Treat each agent run like a distributed trace: one root span for the goal, child spans for each tool call.

    At minimum, instrument these three things:

  • Latency per tool call — slow tools cause the agent to time out or retry unnecessarily
  • Tool call success/failure rate — a spike in failures usually means an upstream API changed
  • Loop iteration count — a sudden jump indicates the agent is stuck reasoning in circles

    • For alerting on these signals in production, we run BetterStack to monitor uptime and get paged when the agent container itself goes unhealthy — it integrates cleanly with the health check defined in the Compose file above. If you’re already tracking container metrics, our container monitoring walkthrough covers wiring up dashboards for exactly this kind of workload.

    Logging Agent Decisions for Auditability

    Because agents make autonomous decisions, you need an audit trail — especially before you let one touch production infrastructure. A simple structured log entry per decision is enough:

    import json, time

def log_decision(state, tool_name, args, result):
    entry = {
        "timestamp": time.time(),
        "tool": tool_name,
        "args": args,
        "result": str(result)[:500],
        "iteration": state.get("iteration", 0)
    }
    with open("agent_audit.log", "a") as f:
        f.write(json.dumps(entry) + "n")

    Ship this log file to whatever aggregator you already use — the format is deliberately boring JSON lines so it drops into an existing pipeline without extra parsing work.

    Common Pitfalls When Building Agentic Workflows

  • No stop condition — agents that loop indefinitely until they “figure it out” will burn tokens and time
  • Unvalidated tool arguments — never let model output reach a shell command unchecked
  • Running as root in production — the container example above uses a non-root user for a reason
  • No human approval gate for destructive actions — restarts are usually fine to automate; deletions and scale-downs usually aren’t
  • Treating the agent as deterministic — the same input can produce different tool-call sequences across runs; test for that variance, don’t assume it away

    • Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    What’s the difference between an agentic AI workflow and a chatbot?
    A chatbot responds to a single message and stops. An agentic workflow runs a loop: it can call tools, observe results, and decide on follow-up actions without a human prompting each step.

    Do I need a GPU to run an agentic AI workflow?
    No, not if you’re calling a hosted LLM API like OpenAI or Anthropic. You only need GPU compute if you’re self-hosting the model weights yourself, which most DevOps teams don’t need to do for infra-automation use cases.

    Is LangGraph the only option for building these pipelines?
    No. LangGraph and CrewAI are the most common Python frameworks, but you can build a basic agentic loop yourself with plain function calling and a while-loop if your use case is narrow enough — you don’t always need a framework.

    How do I stop an agent from taking destructive actions?
    Add an explicit approval gate: have the agent propose the action and wait for human confirmation (Slack message, PR review, etc.) before any tool that deletes, scales down, or modifies production data actually executes.

    How much does running an agentic workflow typically cost?
    It depends entirely on iteration count and model choice. A well-bounded workflow with a 10-iteration cap using a mid-tier model typically costs a few cents per run; an unbounded one can cost dollars per run if it loops.

    Can I run an agentic AI workflow entirely on a small VPS?
    Yes, for API-based agents (not self-hosted models), a 2-4 GB RAM VPS is usually sufficient to run the agent container plus a Redis instance for state.

    Agentic AI workflows are still young as a pattern, but the core discipline is the same as any other production automation: bound your loops, validate your inputs, log everything, and gate anything destructive behind human approval. Start with a narrow, low-risk use case — log triage or PR drafting — before handing an agent broader infrastructure access.

    Related articles:

  • Hong Kong VPS Hosting: Best Options for Low-Latency Asia

    Hong Kong VPS Hosting: A Practical Guide for Low-Latency APAC Deployments

    If you’re serving users in mainland China, Taiwan, Southeast Asia, or greater APAC, a server in Virginia or Frankfurt isn’t going to cut it. Round-trip latency from Hong Kong to Shanghai is typically under 30ms, versus 200ms+ from US-East. That difference matters for real-time apps, streaming edge nodes, gaming backends, and API services. This guide covers what Hong Kong VPS hosting actually gets you, how to pick a provider, and how to deploy a hardened instance from scratch.

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    Why Hong Kong for VPS Hosting

    Hong Kong sits at the crossroads of major submarine cable systems (APG, AAE-1, FASTER) and has minimal regulatory friction compared to mainland China hosting, which requires ICP licensing for any site with a .cn presence or content served to mainland users through domestic infrastructure. A Hong Kong VPS gives you:

  • Low latency to mainland China without needing an ICP license
  • Direct peering with major Asian ISPs (China Telecom, China Unicom, PCCW, HGC)
  • No content censorship at the network level (unlike mainland China-based hosting)
  • Free trade port status, meaning fewer import/export and data sovereignty complications
  • Strong connectivity to Japan, Korea, Singapore, and Australia

    • The tradeoff is cost — Hong Kong bandwidth is more expensive than US or EU bandwidth because of the region’s cable capacity constraints and demand. Expect to pay a premium of 20-40% over a comparable US VPS.

    Latency Benchmarks You Should Actually Run

    Don’t trust marketing pages. Test latency yourself before committing to a provider. From a US-based machine, compare against a Hong Kong candidate:

    # Basic ICMP latency test
ping -c 20 your-candidate-hk-ip

# MTR gives you the full path and packet loss per hop
mtr -rwc 50 your-candidate-hk-ip

# TCP connect time (more realistic for web workloads)
curl -o /dev/null -s -w "connect: %{time_connect}s total: %{time_total}sn" https://your-candidate-hk-ip

    Run these tests from multiple vantage points if you can — a cheap VPS in Tokyo, Singapore, and Sydney will tell you far more about real-world performance than a single ping from your home connection. Services like Cloudflare Radar also publish regional latency and outage data you can cross-reference.

    Picking a Provider: What Actually Matters

    Most “Hong Kong VPS hosting” listicles rank by affiliate payout, not technical merit. Here’s what to actually check:

  • Network provider / carrier — ask whether they use CN2 GIA (China Telecom’s premium route), standard CN2, or a generic transit route. CN2 GIA dramatically reduces latency and jitter to mainland China.
  • KVM vs. OpenVZ virtualization — KVM gives you a real kernel and full isolation; avoid OpenVZ/container-based VPS for anything production-facing.
  • DDoS protection included — Hong Kong is a common target for cross-border attack traffic; confirm mitigation capacity (measured in Gbps) is actually included, not an upsell.
  • IPv6 support — increasingly required for APAC mobile carriers.
  • Snapshot and backup options — test restore time, not just backup frequency.
  • Data center location — “Hong Kong” listings sometimes route through Guangzhou or Shenzhen with relabeled IPs; verify with a traceroute and reverse DNS lookup.

    • If your workload doesn’t strictly require Hong Kong (i.e., you don’t need mainland China proximity specifically), it’s worth comparing against a Singapore or Tokyo deployment — both often have cheaper bandwidth and equally good connectivity to the rest of APAC minus mainland China.

    Setting Up a Hardened Hong Kong VPS

    Once you’ve picked a provider and provisioned a KVM instance running Ubuntu 24.04 or Debian 12, treat the initial setup the same way you would any internet-facing box — assume it will be scanned within minutes of getting a public IP.

    Initial Server Hardening

    # Update packages first
apt update && apt full-upgrade -y

# Create a non-root user with sudo access
adduser deploy
usermod -aG sudo deploy

# Copy your SSH key over before disabling root login
rsync --archive --chown=deploy:deploy ~/.ssh /home/deploy

# Harden SSH config
sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

# Basic firewall rules
ufw allow OpenSSH
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable

# Install fail2ban to slow down brute-force attempts
apt install -y fail2ban
systemctl enable --now fail2ban

    This is the minimum bar. If you’re running anything production-facing, also set up unattended security upgrades and a monitoring agent — we cover the full checklist in our Linux server hardening guide.

    Deploying with Docker

    Most workloads on a Hong Kong VPS end up being reverse-proxied APIs, small databases, or edge caching nodes. Docker keeps this manageable and portable if you later migrate providers.

    # Install Docker Engine
curl -fsSL https://get.docker.com | sh
usermod -aG docker deploy

# Verify installation
docker --version
docker compose version

    A minimal docker-compose.yml for a reverse-proxied app with automatic TLS via Caddy:

    services:
  app:
    image: your-app:latest
    restart: unless-stopped
    expose:
      - "3000"

  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data

volumes:
  caddy_data:

    Caddy handles Let’s Encrypt certificate issuance automatically, which is one less thing to manage on a box you’re already paying a latency premium for.

    Monitoring Cross-Region Latency Over Time

    A single benchmark at provisioning time doesn’t tell you how the route performs during peak hours or under submarine cable maintenance (which happens more often in this region than people expect). Set up a lightweight synthetic monitor:

    # Simple cron-based latency logger
*/5 * * * * ping -c 4 8.8.8.8 | tail -1 >> /var/log/latency.log

    For anything beyond a hobby project, use a real uptime and latency monitoring service so you get alerted before customers notice. We use BetterStack for multi-region uptime checks and status pages — their free tier is enough to monitor a single Hong Kong VPS with alerting via Slack or email.

    Common Pitfalls

  • Assuming “Hong Kong” means CN2 GIA routing — many budget providers resell capacity over congested standard transit. Always test, don’t assume.
  • Skipping DDoS protection to save cost — Hong Kong-hosted IPs see more background attack noise than US/EU ranges; an unprotected box can get knocked offline by opportunistic scanning alone.
  • Ignoring data residency requirements — if you’re handling EU or California user data, confirm your compliance obligations before routing traffic through APAC infrastructure.
  • Underestimating egress costs — bandwidth overage fees in Hong Kong can be steep; check the included allowance and overage rate before committing to a plan.
  • Not testing failover — if mainland China latency is your reason for choosing Hong Kong, have a Plan B region ready in case of cable cuts or routing issues, which do happen.

    • If your project also needs a CDN in front of the VPS, Cloudflare has solid points of presence across Hong Kong and Asia that can absorb traffic spikes and mask your origin IP — worth pairing with any origin server in this region.

    Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Is Hong Kong VPS hosting legal and unrestricted like other regions?
    Yes. Hong Kong operates under a separate legal and network framework from mainland China, with no ICP licensing requirement and no mainland-style content filtering at the infrastructure level. Standard acceptable-use policies from your provider still apply.

    How much latency improvement should I expect over a US VPS for mainland China traffic?
    Typically 150-180ms less round-trip latency compared to US-East, depending on the specific route and whether your provider uses CN2 GIA. Test with mtr before and after migration to confirm the actual gain for your use case.

    Is Hong Kong VPS hosting more expensive than US or EU options?
    Generally yes, by roughly 20-40% for comparable specs, due to higher bandwidth costs in the region. The premium is usually justified only if your traffic is genuinely APAC-heavy.

    Can I run a mainland China-facing website without an ICP license using Hong Kong hosting?
    You can serve content to mainland users from Hong Kong without an ICP license, but expect variable performance since traffic still crosses the border through China’s national gateway, which applies its own filtering and throttling regardless of where your server is hosted.

    Should I choose Hong Kong or Singapore for a broader APAC audience?
    If mainland China proximity is your priority, Hong Kong wins on latency. If your audience is spread across Southeast Asia, Australia, and India with less mainland China traffic, Singapore often has better overall connectivity and lower bandwidth costs.

    Do I need a CDN if I already have a Hong Kong VPS?
    A CDN is still worth it if you have global visitors outside APAC, or if you want DDoS absorption and caching in front of your origin. For a purely APAC-regional audience already close to Hong Kong, a CDN adds less benefit but still helps with TLS termination and traffic spikes.

    Wrapping Up

    Hong Kong VPS hosting makes sense when your traffic genuinely concentrates in APAC, particularly mainland China, and you need the latency advantage badly enough to justify the cost premium. Test actual routing performance before committing, insist on KVM virtualization and real DDoS protection, and harden the box the same way you would any other public-facing server. If your audience is more evenly distributed globally, a multi-region setup with a CDN in front may serve you better than a single Hong Kong instance.

    For teams comparing broader infrastructure options beyond this single region, our cloud VPS provider comparison breaks down pricing and specs across DigitalOcean, Hetzner, and other major players.

    Related articles:

  • OpenAI Whisper API: Developer Guide for Speech-to-Text

    OpenAI Whisper API: A Practical Guide for Developers

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    If you’re building anything that touches audio or video — auto-generated subtitles, podcast transcripts, searchable meeting recordings — the OpenAI Whisper API is one of the fastest ways to get production-grade speech-to-text without babysitting a GPU. This guide walks through setup, real code, Docker-based deployment, and how to fold Whisper into a streaming media pipeline.

    What Is the OpenAI Whisper API

    Whisper started as an open-source model from OpenAI trained on 680,000 hours of multilingual audio. The hosted OpenAI Whisper API wraps that model behind a simple HTTPS endpoint, so you send an audio file and get back a transcript — no model weights, no GPU drivers, no CUDA version hell.

    It supports:

  • Transcription in the original spoken language
  • Translation directly into English
  • Multiple output formats (json, text, srt, vtt, verbose_json)
  • Word- and segment-level timestamps for caption generation
  • Automatic language detection across 90+ languages

    • For teams already running Docker-based infrastructure (see our Docker Compose guide for media servers), Whisper slots in as just another containerized service that calls out to an external API — no local inference workload required.

    How Whisper Differs from Self-Hosted Models

    The open-source Whisper model can be run locally with ffmpeg preprocessing and a Python inference script, but that means owning GPU costs, model loading time, and scaling logic yourself. The API version trades a per-minute usage fee for zero infrastructure management. If you’re processing a handful of files a day, the API is almost always cheaper than a standing GPU instance. If you’re transcribing thousands of hours monthly, self-hosting on a dedicated GPU box starts to make financial sense — more on that tradeoff later.

    Setting Up Your Environment

    You’ll need an OpenAI account with billing enabled and an API key. Store it as an environment variable — never hardcode it in source control.

    export OPENAI_API_KEY="sk-your-key-here"

    Installing Dependencies and Getting an API Key

    Grab your key from the OpenAI platform dashboard, then install the official Python SDK:

    python3 -m venv venv
source venv/bin/activate
pip install openai python-dotenv

    If you’re working with video files rather than raw audio, you’ll also need ffmpeg to extract the audio track first:

    sudo apt update && sudo apt install -y ffmpeg

    Making Your First Transcription Request

    Here’s a minimal script that transcribes an audio file and prints the text:

    import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

with open("episode-01.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        response_format="verbose_json",
        timestamp_granularities=["segment"]
    )

print(transcript.text)
for segment in transcript.segments:
    print(f"[{segment.start:.2f}s - {segment.end:.2f}s] {segment.text}")

    Using verbose_json gives you segment-level timestamps, which is exactly what you need to generate .srt or .vtt caption files for a video player.

    Generating SRT Captions Directly

    You can skip manual formatting entirely by requesting the srt format:

    with open("episode-01.mp3", "rb") as audio_file:
    srt_output = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        response_format="srt"
    )

with open("episode-01.srt", "w") as f:
    f.write(srt_output)

    That .srt file drops straight into Plex, Jellyfin, or any HTML5 video player with <track> support.

    Docker-Based Deployment for Production Pipelines

    For a repeatable production setup, wrap the transcription logic in a small containerized worker rather than running scripts by hand. This fits naturally alongside the rest of a Docker-based stack — if you haven’t containerized your media pipeline yet, our Docker Compose guide for media servers is a good starting point.

    FROM python:3.11-slim

RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY worker.py .

CMD ["python", "worker.py"]

    # docker-compose.yml
services:
  whisper-worker:
    build: .
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    volumes:
      - ./incoming:/app/incoming
      - ./captions:/app/captions
    restart: unless-stopped

    The worker polls an incoming/ directory, transcribes new files, and writes captions to captions/ — a pattern that’s easy to hook into any media ingestion pipeline.

    Handling Large Audio Files and Chunking

    The API caps uploads at 25 MB per request. For long-form content — full podcast episodes, lecture recordings, or livestream VODs — you need to split the audio first. Use ffmpeg to chunk into fixed-length segments before sending each one:

    ffmpeg -i full-episode.mp3 -f segment -segment_time 600 -c copy chunk_%03d.mp3

    Then loop over the chunks, transcribe each, and stitch the resulting text (or timestamped segments) back together, adjusting timestamps by the cumulative offset of each chunk.

    Adding Captions to Streaming Video Workflows

    If you’re running a self-hosted streaming setup — Jellyfin, Plex, or a custom player — auto-generated captions are one of the highest-value, lowest-effort additions you can make. A typical pipeline looks like:

    1. Extract audio from the source video with ffmpeg
    2. Send the audio to the Whisper API for transcription
    3. Save the returned .srt alongside the video file
    4. Let your media server auto-detect the sidecar subtitle file

    ffmpeg -i movie.mkv -vn -acodec mp3 movie-audio.mp3

    This is especially useful for archived home-theater libraries where original subtitle tracks were never included, or for creators who need burned-in captions for accessibility compliance.

    Cost Optimization and Rate Limits

    Whisper API pricing is billed per minute of audio processed, not per API call. A few practical ways to keep costs down:

  • Downsample audio to mono 16kHz before uploading — it doesn’t affect transcription quality but reduces file size and upload time
  • Batch process during off-peak hours if you’re running a queue-based worker
  • Cache transcripts — don’t re-transcribe the same file if it hasn’t changed
  • Use language hints when you know the source language, which slightly speeds up processing

    • Rate limits are account-tier dependent, so if you’re running a high-volume pipeline, implement exponential backoff on 429 responses rather than hammering retries.

    Self-Hosting vs API: When to Use Which

    Running open-source Whisper locally makes sense once your volume justifies dedicated GPU hardware. Below a certain threshold, the API is simpler and cheaper. As a rough rule of thumb:

  • Under ~50 hours/month of audio: use the API, skip the infrastructure
  • 50–500 hours/month: API is still usually cheaper unless you already have idle GPU capacity
  • 500+ hours/month: a dedicated GPU VPS running the open-source model often pays for itself within weeks

    • If you land in that last bucket, both DigitalOcean and Hetzner offer GPU-backed instances well suited to running Whisper locally at scale, and either is a solid choice if you’re already comfortable managing your own Docker stack — check out our best VPS providers for streaming servers breakdown for a deeper comparison.

    Deploying Your Transcription Service

    For the worker pattern described above, you don’t need anything exotic — a small 2-4 vCPU instance is plenty, since the actual transcription happens on OpenAI’s infrastructure, not yours. Your container just handles file movement, chunking, and API calls.

    If you’re standing up a new box specifically for this kind of media automation, DigitalOcean’s Droplets are quick to provision and integrate cleanly with Docker Compose out of the box. Hetzner tends to win on raw price-per-core if your workload is more CPU-bound (heavy ffmpeg transcoding alongside the transcription queue).

    Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Does the Whisper API support real-time streaming transcription?
    No. The hosted API is request/response only — you send a complete audio file and get a transcript back. For live/real-time captioning, you’d need to chunk a live stream into short segments and transcribe them in near-real-time, which introduces latency, or use a different real-time speech service.

    What audio formats does the API accept?
    MP3, MP4, MPEG, MPGA, M4A, WAV, and WEBM are all supported directly. If your source is a video container like MKV, extract the audio track with ffmpeg first.

    How accurate is Whisper compared to other transcription services?
    Whisper generally performs very well on clear English audio and holds up better than most competitors on accented speech and background noise, though accuracy on noisy or overlapping-speaker audio still degrades like any ASR system.

    Can I use the Whisper API for copyrighted movie or show audio?
    Technically yes for personal captioning of content you legally own, but redistributing generated captions for copyrighted material you don’t have rights to can raise legal issues — check your local regulations before publishing.

    Is there a free tier for the Whisper API?
    No dedicated free tier exists for the API itself, though new OpenAI accounts sometimes receive trial credits. Budget for per-minute usage costs in any production plan.

    What’s the maximum file size per request?
    25 MB per upload. Longer files need to be chunked with ffmpeg before sending, as covered earlier in this guide.

    Related articles:

  • AI Real Estate Agent: Self-Host One with Docker

    How to Build a Self-Hosted AI Real Estate Agent with Docker

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    Every SaaS vendor is now shipping an “AI real estate agent” bolt-on for lead qualification, listing summaries, and chat-based property search. Most of them are thin wrappers around GPT-4 with a markup attached. If you run infrastructure for a real estate brokerage, property management company, or proptech startup, you can build the same thing yourself — self-hosted, cheaper at scale, and fully under your control.

    This guide walks through the actual architecture: containerized LLM inference, a retrieval layer for property data, an agent framework to tie it together, and a production deployment on a cloud VPS. This is written for developers and sysadmins, not marketers — expect Docker Compose files and Python, not buzzwords.

    What Is an AI Real Estate Agent, Technically?

    Strip away the marketing and an AI real estate agent is a retrieval-augmented generation (RAG) pipeline with domain-specific tools bolted on. It typically does four things:

  • Answers natural-language questions about listings (“show me 3-bed homes under $450k near downtown with a garage”)
  • Summarizes property details, disclosures, and comps into digestible text
  • Qualifies leads by asking follow-up questions and scoring intent
  • Schedules showings or hands off to a human agent when confidence is low

    • None of this requires a proprietary platform. It requires an LLM, a vector store for your listing data, an orchestration layer, and a place to run it reliably. We’ve covered similar patterns before in our guide to self-hosting LLMs with Docker, and the same core pattern applies here — you’re just swapping the domain data.

    Core Components of the Stack

    A minimal production-grade AI real estate agent stack looks like this:

  • LLM inference server — Ollama or vLLM running an open-weight model (Llama 3, Mistral, or Qwen)
  • Vector database — Qdrant or pgvector for storing embedded listing descriptions
  • Orchestration layer — LangChain or LlamaIndex to handle retrieval and tool-calling
  • API layer — FastAPI to expose the agent to your website or CRM
  • Reverse proxy + TLS — Caddy or Nginx in front of everything

    • All five run comfortably as containers on a single mid-tier VPS for low-to-moderate traffic. At scale, you split inference onto a GPU node and keep the rest on CPU instances.

    Architecture: From LLM to Deployment

    The key architectural decision is whether to call a hosted API (OpenAI, Anthropic) or self-host the model. For a real estate agent handling sensitive client data — names, financials, property addresses — self-hosting avoids sending that data to a third party and gives you predictable costs at volume. We’ll build the self-hosted version.

    Choosing and Running Your LLM Backend

    Ollama is the fastest path to a running local LLM in a container. Here’s a docker-compose.yml that stands up the full stack:

    version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: agent-ollama
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  qdrant:
    image: qdrant/qdrant:latest
    container_name: agent-vectordb
    volumes:
      - qdrant_data:/qdrant/storage
    ports:
      - "6333:6333"

  agent-api:
    build: ./agent-api
    container_name: agent-api
    depends_on:
      - ollama
      - qdrant
    environment:
      - OLLAMA_HOST=http://ollama:11434
      - QDRANT_HOST=http://qdrant:6333
    ports:
      - "8000:8000"

volumes:
  ollama_data:
  qdrant_data:

    Pull a model once the containers are up:

    docker exec -it agent-ollama ollama pull llama3.1:8b

    An 8B parameter model runs acceptably on CPU for low traffic and comfortably on a single consumer GPU for anything higher volume.

    Building the Agent Logic with LangChain

    The agent-api service is where the real estate logic lives. It embeds listing data into Qdrant, retrieves relevant listings per query, and lets the LLM reason over them with tool access.

    # agent-api/main.py
from fastapi import FastAPI
from pydantic import BaseModel
from langchain_community.llms import Ollama
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import OllamaEmbeddings
from langchain.chains import RetrievalQA

app = FastAPI()

llm = Ollama(base_url="http://ollama:11434", model="llama3.1:8b")
embeddings = OllamaEmbeddings(base_url="http://ollama:11434", model="llama3.1:8b")

vectorstore = Qdrant(
    client=None,
    collection_name="listings",
    embeddings=embeddings,
    url="http://qdrant:6333",
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)

class Query(BaseModel):
    question: str

@app.post("/ask")
def ask(query: Query):
    result = qa_chain.invoke(query.question)
    return {"answer": result["result"]}

    This is intentionally minimal — production systems add conversation memory, lead-scoring logic, and a handoff tool that pings a human agent via Slack or CRM webhook when the model’s confidence drops below a threshold. LangChain’s tool-calling API is the cleanest way to wire that handoff in without writing a custom state machine.

    To populate the vector store, run listing data through the same embeddings model on ingestion — typically a nightly cron job pulling from your MLS feed or CRM export, chunking property descriptions, and upserting them into Qdrant.

    Deploying to Production

    A local Docker Compose setup is fine for development, but production needs TLS, monitoring, and a host that won’t fall over under a traffic spike from a listing going viral on social media. A few notes from running similar RAG stacks in production:

  • Put a 4-8 vCPU / 16GB+ RAM VPS behind the agent-api and Qdrant containers if you’re running CPU inference — Hetzner instances are the best price-per-core we’ve found for this workload, and DigitalOcean droplets are a solid alternative if you want a more polished managed-network experience.
  • Front the API with Cloudflare for DDoS protection and caching of static assets, and to hide your origin IP from scrapers targeting your listing data.
  • Set container resource limits (mem_limit, cpus) in Compose so a runaway inference request doesn’t starve the vector DB.
  • Back up the Qdrant volume on a schedule — losing embedded listing data means a full re-ingestion.

    • Our Docker Compose guide for beginners covers the fundamentals of multi-container orchestration if you’re newer to Compose specifically.

    For TLS termination, Caddy is the lowest-friction option — it handles Let’s Encrypt automatically:

    agent.yourdomain.com {
    reverse_proxy agent-api:8000
}

    Monitoring and Scaling the Agent

    Once the agent is live, you need visibility into latency, error rates, and inference cost per query — LLM calls are far more expensive per-request than typical API endpoints, so a silent bug that triggers repeated calls can blow through compute budget fast.

  • Track p95 latency on the /ask endpoint — anything over 3-4 seconds will feel broken to a user typing into a chat widget
  • Alert on GPU/CPU saturation on the inference host before it causes timeouts
  • Log every query and response for auditing — real estate has fair-housing compliance implications, and you need a paper trail of what the AI told a prospective buyer

    • BetterStack is a solid option for uptime and log monitoring across the whole stack without standing up your own Prometheus/Grafana pair — worth it if you don’t already have observability tooling running. If you do run your own stack, our container monitoring guide with Prometheus covers the metrics worth scraping from each service.

    At higher traffic, split the architecture: move Ollama to a dedicated GPU instance, keep the FastAPI layer on cheap CPU instances behind a load balancer, and scale the API horizontally since it’s stateless. Qdrant scales vertically well past what most single-brokerage deployments will ever need.

    Handling Fair Housing and Compliance

    This part matters more than the tech stack. Any AI system answering questions about housing in the US is subject to the Fair Housing Act — the model cannot discriminate or imply preference based on protected classes, even indirectly through phrasing. Bake explicit system-prompt constraints against discriminatory language into the LLM’s instructions, log every conversation, and have a human review process for edge cases. This isn’t optional legal boilerplate — it’s the single biggest risk in shipping this kind of system for a real brokerage.

    Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Do I need a GPU to run an AI real estate agent locally?
    No, but it helps. An 8B parameter model like Llama 3.1 runs on CPU at a few tokens per second, which is usable for low-traffic sites. For anything with real concurrent traffic, a single consumer GPU (RTX 4090 or similar) or a cloud GPU instance dramatically improves response time.

    Is self-hosting actually cheaper than using OpenAI’s API?
    At low volume, no — API calls are cheaper than running a dedicated server. Past a few thousand queries a day, self-hosting on a fixed-cost VPS typically wins, and you avoid sending client PII to a third party.

    Can this replace a human real estate agent?
    No, and it shouldn’t try to. It’s best used for lead qualification, initial questions, and listing search — freeing up human agents for negotiation, showings, and closing, where trust and local expertise matter most.

    What LLM should I use for real estate use cases?
    Llama 3.1 8B or Mistral 7B are good defaults — fast, cheap to run, and good enough at retrieval-augmented Q&A. Use a larger model (Llama 3.1 70B) if you need more nuanced reasoning over comps or disclosures and have the GPU budget for it.

    How do I keep the agent from giving legally risky answers?
    Constrain the system prompt explicitly against discriminatory language, log all conversations for audit, and route anything touching pricing negotiation or contract terms to a human. Treat the AI as a triage layer, not a decision-maker.

    Does this integrate with existing CRMs like Follow Up Boss or kvCORE?
    Yes, via webhook — have the FastAPI layer POST qualified leads and conversation summaries to your CRM’s API after each session, the same way you’d wire up any other lead source.

    Self-hosting an AI real estate agent isn’t harder than using a SaaS platform — it’s a Docker Compose file, an open-weight model, and a VPS. The tradeoff is you own the maintenance, but you also own the data, the cost curve, and the ability to actually understand why the thing answered the way it did.

    Related articles:

  • Agentic AI News: What DevOps Teams Need to Track

    Agentic AI News: A DevOps Guide to the Autonomous Agent Shift

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    If you’ve been half-following agentic AI news over the last few months, you’ve probably noticed the pattern: every week brings a new framework, a new “autonomous agent” demo, and a new set of claims about agents that can plan, execute, and self-correct without a human in the loop. For developers and sysadmins, the more useful question isn’t “is this hype?” — it’s “what do I actually need to change in my infrastructure to run this stuff safely?”

    This guide breaks down what’s actually new in agentic AI, why it matters operationally, and how to containerize, deploy, and monitor agent workloads on your own infrastructure without getting burned.

    What Counts as “Agentic AI” Right Now

    Agentic AI refers to systems that don’t just respond to a single prompt — they plan a sequence of steps, call tools or APIs, evaluate the results, and decide what to do next, often looping until a goal is met or a budget runs out. That’s a meaningful shift from the request/response chatbot pattern most teams are used to.

    From Chatbots to Autonomous Loops

    The practical difference shows up in three places:

  • State: agents maintain a working memory of what they’ve tried and what happened, not just the last message.
  • Tool use: agents call shell commands, APIs, databases, or other services as part of execution, not just to fetch context.
  • Control flow: agents decide their own next action, which means your infrastructure has to assume unpredictable, repeated calls rather than a single fixed request.

    • Frameworks like LangGraph and various open-source agent orchestrators have leaned into this pattern, and it’s why agentic AI news keeps circling back to the same operational concerns: cost control, sandboxing, and observability.

    Why This Matters for Infrastructure Teams

    The news cycle around agentic AI tends to focus on capability — what the agent can now do. The part that gets less coverage, and that actually matters for your job, is that agents run longer, call out more often, and fail in less predictable ways than a typical microservice. If you’re the one who gets paged when something misbehaves, you need guardrails before you need more capability.

    Recent Agentic AI News Themes Worth Tracking

    A few threads keep showing up across agentic AI news coverage:

    Open-Source Agent Frameworks Maturing

    Open-source agent orchestration tooling has moved from research demos to production-adjacent releases with retry logic, checkpointing, and human-in-the-loop approval steps. That maturity is good news operationally — it means fewer bespoke wrappers and more standard patterns you can containerize and monitor like any other service.

    Tool-Use Standardization

    There’s growing convergence around structured tool-calling interfaces (function calling, schema-based tool definitions) instead of agents parsing raw text to decide what to do. This matters because it makes agent actions auditable — you can log exactly which tool was called with which arguments, rather than trying to reverse-engineer intent from free text.

    Cost and Rate-Limit Pressure

    Because agents loop and re-call models multiple times per task, teams are reporting real cost surprises. This is pushing more agentic AI news toward practical topics: budget caps per task, step limits, and circuit breakers — the same patterns you’d already use for any retrying background job.

    Deploying Agentic AI Workloads with Docker

    Whether you’re running an open-source agent framework or a custom Python loop calling a model API, the deployment shape is similar: a long-running or on-demand worker process, a queue or trigger, and outbound calls to a model provider and whatever tools the agent is allowed to use.

    A Minimal Containerized Agent Worker

    Here’s a simple example of containerizing a Python-based agent worker that reads tasks off a queue and executes them with a bounded step count:

    # agent_worker.py
import os
import time

MAX_STEPS = int(os.getenv("AGENT_MAX_STEPS", "6"))

def run_agent_task(task: dict) -> dict:
    steps_taken = 0
    result = None

    while steps_taken < MAX_STEPS:
        action = decide_next_action(task, result)
        if action["type"] == "finish":
            return {"status": "done", "output": action["output"]}
        result = execute_tool(action)
        steps_taken += 1

    return {"status": "step_limit_reached", "last_result": result}

def decide_next_action(task, last_result):
    # Call your model provider here with task + last_result as context
    raise NotImplementedError

def execute_tool(action):
    # Dispatch to whitelisted tools only
    raise NotImplementedError

if __name__ == "__main__":
    while True:
        task = poll_queue()
        if task:
            run_agent_task(task)
        else:
            time.sleep(2)

    FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY agent_worker.py .

ENV AGENT_MAX_STEPS=6
USER 1000
CMD ["python", "agent_worker.py"]

    The AGENT_MAX_STEPS cap is the single most important line in this example. Without a hard ceiling, a misbehaving agent loop will happily keep calling your model provider — and your bill — indefinitely.

    Compose Stack: Agent Worker + Queue + Monitoring

    version: "3.9"
services:
  agent-worker:
    build: .
    restart: unless-stopped
    environment:
      - AGENT_MAX_STEPS=6
      - QUEUE_URL=redis://queue:6379/0
    depends_on:
      - queue
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M

  queue:
    image: redis:7-alpine
    restart: unless-stopped
    volumes:
      - queue-data:/data

  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

volumes:
  queue-data:

    Running this stack is not much different from any other worker/queue deployment you’ve already built — which is exactly the point. Treat agentic workloads as you would any bursty background job system, not as a special snowflake.

    For a broader look at running containerized services reliably, our Docker Compose monitoring stack guide walks through wiring Prometheus and Grafana into a stack like this one.

    Resource Limits Are Non-Negotiable

    Set CPU and memory limits on every agent container. Agents that call subprocesses, spawn browser automation, or shell out to CLI tools can consume resources far more aggressively than a typical API service, especially if a loop condition doesn’t terminate the way you expect.

    Monitoring and Observability for Agents

    Standard application metrics (latency, error rate, request count) aren’t enough for agentic workloads. You also need:

  • Step count per task — flags runaway loops before they become cost incidents.
  • Tool call logs — every tool invocation with its arguments, for auditability.
  • Token/cost per task — model API usage tends to be the real cost driver, not compute.
  • Timeout and abort rate — how often tasks hit your step or time limits instead of completing cleanly.

    • If you’re already running a BetterStack uptime and log monitoring setup, extending it to capture structured logs from your agent worker (one log line per tool call, with task ID and step number) gives you a searchable audit trail without much extra work. Alternatively, ship structured logs to your existing observability stack the same way you would for any other service — the format matters more than the destination.

    Setting Alerts Before You Need Them

    Don’t wait for an agentic AI news headline about a runaway agent bill to be your motivation. Set alerts on:

  • Step-limit-reached rate exceeding a threshold (signals prompt or tool-definition problems)
  • Hourly model API spend crossing a budget line
  • Tool call failure rate spiking (often the first sign of an upstream API change)

    • Security Considerations for Autonomous Agents

    Agentic AI news coverage often glosses over the security implications of giving a model the ability to call tools autonomously. A few practical rules:

  • Whitelist tools explicitly. Never let an agent construct arbitrary shell commands from model output; expose a fixed set of narrow, parameterized functions instead.
  • Run agents in isolated containers, not on hosts with broad filesystem or network access.
  • Scope credentials tightly. An agent’s API keys should have the minimum permissions needed for its specific tools — not your full admin credentials.
  • Log everything. If an agent takes an unexpected action, you need a full trace of what it decided and why.

    • If you’re evaluating where to run these workloads, a VPS with predictable CPU/memory limits makes it much easier to enforce container-level resource caps than a shared or bursty environment. Providers like DigitalOcean and Hetzner both offer straightforward Docker-friendly droplets/VMs where you can size instances precisely for agent worker workloads and scale horizontally as task volume grows.

    Testing Agent Behavior Before Production

    Before deploying any agent framework update, run it against a fixed set of test tasks with tool calls mocked out. This catches regressions in decision-making (an agent suddenly looping more, or choosing the wrong tool) before it hits production traffic and your budget.

    Keeping Up With Agentic AI News Without Drowning in It

    Given how fast this space moves, it’s not realistic to read every announcement. A more sustainable approach:

  • Follow release notes for the specific framework(s) you’ve adopted, not the entire ecosystem.
  • Skim agentic AI news weekly for security disclosures or breaking API changes — treat those as must-read.
  • Ignore benchmark-of-the-week posts unless they affect a model you’re actually running in production.

    • Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    What is agentic AI in simple terms?
    Agentic AI describes AI systems that plan multi-step actions, call tools or APIs, evaluate results, and decide their next step on their own, rather than just answering a single prompt.

    Is agentic AI the same as an AI agent framework like LangGraph?
    Not exactly — “agentic AI” describes the behavior pattern, while frameworks like LangGraph are specific tools for building and orchestrating that behavior. You can build an agentic system without using any particular framework.

    How do I stop an AI agent from running up my API bill?
    Set a hard step limit per task, a token/cost budget per task, and alerting on hourly spend. Treat these the same way you’d treat rate limits on any retrying background job.

    Do agentic AI workloads need special infrastructure?
    Not special, but they do need infrastructure that assumes bursty, repeated outbound calls and enforces per-container resource limits. A standard Docker worker/queue setup on a properly sized VPS handles this well.

    What’s the biggest security risk with autonomous agents?
    Letting an agent construct and execute arbitrary commands or API calls without a whitelist. Always scope tool access explicitly and log every tool invocation.

    Where should I host agent worker containers?
    Any Docker-friendly VPS with clear CPU/memory limits works. Predictable, non-shared resources make it far easier to reason about how an agent workload will behave under load.

    Agentic AI news will keep moving fast, but the infrastructure fundamentals don’t change much: containerize the workload, cap its steps and budget, log every action, and monitor it like you would any other background job system — just with tighter guardrails.

    Related articles:

  • Docker Compose Up Build: Full Guide & Best Practices

    Docker Compose Up Build: The Complete Guide to Rebuilding and Running Containers

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    If you’ve spent any time working with multi-container applications, you’ve almost certainly typed docker compose up build into your terminal, only to get a syntax error or unexpected behavior. The correct command is docker compose up --build, and understanding exactly what it does — and when to use it — will save you hours of debugging “why isn’t my code change showing up” mysteries.

    This guide covers the full mechanics of docker compose up --build, how it differs from running docker compose build separately, common pitfalls with caching, and practical workflows for local development, CI pipelines, and production-adjacent staging environments.

    What Does docker compose up –build Actually Do

    The docker compose up --build command tells Compose to rebuild any service images before starting the containers. Without the --build flag, Compose only builds an image if it doesn’t already exist locally — if you’ve made changes to your Dockerfile or the files it copies in, those changes will be silently ignored and you’ll keep running the old image.

    Here’s the basic syntax:

    docker compose up --build

    This single command does three things in sequence:

  • Rebuilds every service in your docker-compose.yml that has a build: directive
  • Recreates containers that depend on those rebuilt images
  • Starts (or restarts) all services defined in the compose file

    • If you only want to rebuild and start a single service, you can scope it:

    docker compose up --build web

    This rebuilds and starts only the web service, along with any services it depends on via depends_on.

    Why the Plain docker compose up Command Isn’t Enough

    A lot of confusion comes from assuming Compose always rebuilds on up. It doesn’t. Compose checks whether an image already exists for the service; if it does, up reuses it, even if your source code or Dockerfile has changed since the image was built.

    This is a common source of “phantom bugs” where a developer fixes an issue, runs docker compose up, and the bug is still there — because the container is running stale code baked into an old image layer. Running docker compose up --build forces Compose to re-evaluate the build context and rebuild as needed.

    The Difference Between docker compose build and docker compose up –build

    These two approaches look similar but serve different purposes:

    # Build images without starting containers
docker compose build

# Build images and immediately start containers
docker compose up --build

    docker compose build is useful in CI/CD pipelines where you want to build and push an image without running it locally. docker compose up --build is better suited for local development loops where you’re iterating quickly and want your changes reflected immediately.

    If you’re setting up a CI pipeline, it’s worth reading the official Docker Compose CLI reference for the full list of flags and how they interact with build caching.

    Common Flags and Options You’ll Actually Use

    Beyond the basic --build flag, there are several options that make the command far more useful in day-to-day work.

    Forcing a Full Rebuild with –no-cache

    Docker’s build cache is usually your friend — it speeds up rebuilds by reusing unchanged layers. But sometimes the cache causes problems, especially when a base image has been updated upstream or a RUN apt-get install step needs fresh package lists.

    docker compose build --no-cache
docker compose up --build --no-cache

    Note that --no-cache isn’t actually a flag on up directly in older Compose versions — if you hit an error, run the no-cache build separately first, then bring the stack up:

    docker compose build --no-cache && docker compose up

    Running in Detached Mode

    For background services, especially on a VPS or remote server, combine --build with -d:

    docker compose up --build -d

    This rebuilds images and runs containers in detached mode so your terminal session stays free. This is the pattern most people use when deploying to a DigitalOcean droplet or similar cloud VM — SSH in, pull the latest code, and run this one command to redeploy.

    Forcing Container Recreation

    Sometimes Compose decides a container doesn’t need to be recreated even after a rebuild, particularly if only environment variables changed. Force it with:

    docker compose up --build --force-recreate

    This guarantees old containers are torn down and replaced, not just restarted.

    Practical Workflow Examples

    Let’s walk through a realistic scenario. Say you have a simple Node.js API with the following docker-compose.yml:

    version: "3.9"
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=development
    volumes:
      - ./src:/app/src
  db:
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=devpassword
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

    And a Dockerfile like this:

    FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "src/index.js"]

    If you add a new npm dependency to package.json, running plain docker compose up won’t install it — the existing image already has node_modules baked in from the last build. You need:

    docker compose up --build

    This re-runs npm install inside the image build step, picking up the new dependency, then starts both the api and db services.

    Handling Multi-Service Rebuilds Efficiently

    In larger projects with multiple services (API, worker, frontend, reverse proxy), rebuilding everything every time is wasteful. Scope your builds:

    docker compose up --build api worker

    This rebuilds only api and worker, leaving db and any unchanged services untouched, which speeds up your iteration loop considerably.

    If you’re managing a growing compose file, it’s worth reviewing our guide to organizing multi-container Docker projects for patterns on splitting services across multiple compose files with docker compose -f.

    Troubleshooting Common docker compose up –build Issues

    A few issues come up repeatedly with this command, and most have simple fixes.

  • Changes not appearing after rebuild: Check whether you have a bind-mounted volume overwriting the container’s code directory with an outdated local copy. Volumes mounted in docker-compose.yml take precedence over what’s baked into the image.
  • Build succeeds but container immediately exits: Check your CMD or ENTRYPOINT — the container will exit if the foreground process crashes or completes. Run docker compose logs <service> to see the actual error.
  • “no space left on device” during build: Docker’s build cache and dangling images accumulate over time. Run docker system prune -a (carefully — this removes unused images) to reclaim space.
  • Environment variable changes not taking effect: Environment variables set in .env files or environment: blocks don’t always force a rebuild. Combine with --force-recreate to be safe.
  • Build context is too large and slow: Add a .dockerignore file to exclude node_modules, .git, and other large directories from being sent to the Docker daemon during build.

    • Speeding Up Builds with BuildKit

    Modern Docker installations use BuildKit by default, which parallelizes build steps and caches more intelligently. If you’re on an older setup, enable it explicitly:

    DOCKER_BUILDKIT=1 docker compose up --build

    BuildKit also supports cache mounts for package managers, which can dramatically speed up repeated npm install or pip install steps. See Docker’s own documentation on BuildKit for configuration details.

    Deploying with docker compose up –build on a Remote Server

    When deploying to a VPS, the workflow typically looks like this: SSH into the server, pull the latest code, and rebuild.

    ssh user@your-server-ip
cd /opt/myapp
git pull origin main
docker compose up --build -d

    For production deployments, consider adding a health check to your compose file so Compose (and any orchestration layer) knows when a service is actually ready:

    services:
  api:
    build: .
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3

    If you’re running this on a budget VPS provider, Hetzner offers solid price-to-performance ratios for small Docker Compose deployments, and their cloud consoles make it easy to spin up a fresh instance to test a deployment before pushing to your main server.

    For teams that need uptime monitoring on these deployed services, pairing your compose stack with BetterStack gives you alerting when a container-based service goes down, which is especially useful right after a --build deploy where a bad image can silently break a health check.

    If your compose-based app sits behind a public domain, routing DNS and basic DDoS protection through Cloudflare is a near-zero-cost way to harden a VPS-hosted deployment before you invest in a full load balancer setup.

    Automating Rebuilds in CI/CD

    Many teams wire docker compose up --build into a deployment script triggered by a git push. A minimal GitHub Actions step might look like:

    - name: Deploy via SSH
  run: |
    ssh -o StrictHostKeyChecking=no user@${{ secrets.SERVER_IP }} 
      "cd /opt/myapp && git pull && docker compose up --build -d"

    This pattern works well for small to medium projects. For anything with higher uptime requirements, you’ll want a proper CI pipeline that builds and tests images before they ever touch your production compose file — check out our beginner’s guide to Docker fundamentals if you’re still solidifying the basics before automating deployments.

    Best Practices Checklist

    Before you wire docker compose up --build into your regular workflow, keep these practices in mind:

  • Always add a .dockerignore file to keep build contexts small and fast
  • Use --build only when source files or Dockerfiles have changed — otherwise plain up is faster
  • Combine with -d for background services on remote servers
  • Use --force-recreate when environment variables or configs change without a corresponding image rebuild
  • Periodically run docker system prune to clean up unused build cache and dangling images
  • Pin base image versions (e.g., node:20-alpine instead of node:latest) to avoid unexpected behavior from upstream updates

    • Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Is “docker compose up build” a valid command?
    No. The correct syntax requires two dashes: docker compose up --build. Without the dashes, Docker will interpret build as a service name and likely return an error saying no such service exists.

    Does docker compose up –build always rebuild every service?
    Yes, by default it rebuilds all services in the compose file that have a build: directive. To limit the rebuild to specific services, list them after the command, like docker compose up --build api.

    What’s the difference between docker compose up –build and docker-compose up –build?
    Functionally they’re the same. docker compose (no hyphen) is the newer Compose V2 syntax integrated into the Docker CLI, while docker-compose is the older standalone Python-based tool. Compose V2 is faster and actively maintained, so it’s the recommended syntax going forward.

    Why isn’t my code change showing up even after running docker compose up –build?
    The most common cause is a bind-mounted volume overriding the container’s working directory with a stale local copy, or the build cache reusing a layer that should have been invalidated. Try docker compose build --no-cache followed by docker compose up --force-recreate.

    Does docker compose up –build slow down my workflow?
    It adds build time on every run, so it’s slower than plain up when nothing has changed. Use it selectively — only when you’ve modified your Dockerfile, dependencies, or files copied during the build step.

    Can I use docker compose up –build in production?
    It’s more common in development and simple deployment scripts. For production, most teams prefer building images in CI, pushing them to a registry, and having the production compose file reference a specific image tag rather than rebuilding on the server itself.

    Wrapping Up

    docker compose up --build is one of the most-used commands in a containerized development workflow, but its behavior around caching and image reuse trips up even experienced developers. The core rule to remember: if you’ve changed anything that affects the image (Dockerfile, dependencies, copied files), you need --build to see those changes reflected. Combine it with -d for background services, --force-recreate when configs change, and --no-cache when you suspect stale cache layers are the culprit, and you’ll avoid most of the common headaches associated with this command.

    Related articles:

  • AI Agent vs Agentic AI: Key Differences Explained

    AI Agent vs Agentic AI: What Developers Actually Need to Know

    Every vendor pitch deck in 2026 uses “agent” and “agentic” interchangeably, and it’s making architecture decisions harder than they need to be. If you’re a developer or sysadmin trying to figure out what to actually deploy on your infrastructure, the distinction matters — it changes your compute footprint, your orchestration layer, and your monitoring strategy.

    This article breaks down the ai agent vs agentic ai debate in concrete technical terms, with real deployment patterns you can run on your own servers.

    Quick Definitions Before We Go Deeper

  • AI agent: A single autonomous unit that perceives input, reasons over it, and takes an action — usually calling one or more tools or APIs to complete a bounded task.
  • Agentic AI: A system-level architecture where multiple agents (or agent instances) coordinate, delegate, and iterate toward a goal with minimal human intervention, often involving planning, memory, and self-correction loops.

    • In other words: an AI agent is a component. Agentic AI is the system built from those components, plus the orchestration logic that makes them work together.

    What Is an AI Agent?

    An AI agent is typically a single LLM-backed process wired to a fixed set of tools. Think of a Slack bot that reads a support ticket, queries a knowledge base, and drafts a reply. It has:

  • A defined input/output contract
  • A limited toolset (search, database query, API call)
  • No persistent long-term planning — it reacts, it doesn’t strategize across sessions
  • Usually a single LLM call or a short chain of calls per request

    • This is the pattern most teams are already running in production. It’s the same shape as a traditional microservice, just with an LLM doing the reasoning step instead of hardcoded business logic.

    What Is Agentic AI?

    Agentic AI describes a system where multiple agents — or a single agent operating across many iterative steps — collaborate to solve a multi-stage problem without a human approving each step. A DevOps example: an agentic pipeline that receives a vague ticket like “deploy is failing on staging,” then:

    1. Spins up a diagnostic agent to pull logs
    2. Spins up a root-cause agent to correlate logs against recent commits
    3. Spins up a remediation agent to propose (or apply) a fix
    4. Reports back with a summary and a rollback plan if the fix fails

    Each of those steps could itself be an AI agent. The agentic system is the orchestration layer that sequences them, maintains shared state, and decides when to loop back versus escalate to a human.

    Key Differences at a Glance

    | Dimension | AI Agent | Agentic AI |
    |—|—|—|
    | Scope | Single task | Multi-step goal |
    | State | Mostly stateless per request | Persistent memory across steps |
    | Autonomy | Reacts to input | Plans and self-corrects |
    | Failure mode | Returns an error | Retries, replans, or escalates |
    | Infra footprint | One process/container | Orchestrator + N worker agents |
    | Typical trigger | API call or webhook | Goal statement or ticket |

    If you’re deciding what to build first, start with the AI agent pattern. It’s cheaper to run, easier to debug, and easier to put behind rate limits. Only move to agentic AI once you have a workflow that genuinely needs multi-step planning — most “agentic” use cases people pitch you don’t actually need it.

    Architecture: How This Looks in Practice

    A single AI agent is straightforward to containerize. Here’s a minimal Python agent using a tool-calling pattern, wrapped for Docker deployment:

    # agent.py
import os
import requests
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def check_service_status(url: str) -> str:
    resp = requests.get(url, timeout=5)
    return f"{url} returned {resp.status_code}"

tools = [{
    "type": "function",
    "function": {
        "name": "check_service_status",
        "description": "Check HTTP status of a service URL",
        "parameters": {
            "type": "object",
            "properties": {"url": {"type": "string"}},
            "required": ["url"]
        }
    }
}]

def run_agent(prompt: str):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        tools=tools
    )
    return response.choices[0].message

    FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY agent.py .
CMD ["python", "agent.py"]

    That’s a full AI agent: one container, one job, one tool. Run it with:

    docker build -t status-check-agent .
docker run --env OPENAI_API_KEY=$OPENAI_API_KEY status-check-agent

    Deploying an Agentic AI System with Docker Compose

    Agentic AI needs an orchestrator plus multiple agent workers, usually with a shared message queue or state store between them. A minimal version:

    # docker-compose.yml
version: "3.9"
services:
  orchestrator:
    build: ./orchestrator
    environment:
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis

  diagnostic-agent:
    build: ./agents/diagnostic
    environment:
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis

  remediation-agent:
    build: ./agents/remediation
    environment:
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

    The orchestrator publishes tasks to Redis, each worker agent subscribes to its channel, and results flow back for the orchestrator to sequence the next step. This is roughly the same pattern used by frameworks like LangGraph and CrewAI, just stripped down to raw containers so you can see what’s actually happening under the hood.

    If you’re already running container workloads, this maps cleanly onto skills you have. Our Docker container monitoring guide covers how to keep tabs on multi-container stacks like this one, and if you haven’t settled on hosting yet, check our VPS hosting comparison for DevOps teams before committing infrastructure spend.

    Choosing the Right Pattern for Your Use Case

    Here’s a practical decision framework:

  • Use a single AI agent when the task is well-defined, has a clear input/output shape, and doesn’t require the system to decide what to do next — only how to answer this one thing.
  • Use agentic AI when the task involves ambiguity, multiple dependent steps, or requires the system to adapt its plan based on intermediate results.
  • Avoid agentic AI for anything latency-sensitive or cost-sensitive at scale — every planning loop adds LLM calls, and LLM calls add both latency and token cost.
  • Always add a human checkpoint before agentic systems take destructive or irreversible actions (deleting resources, pushing to production, sending customer-facing messages).

    • One operational reality worth flagging: agentic systems fail in messier ways than single agents. A single agent either returns a good answer or an error. An agentic system can get stuck in a planning loop, burn through your API budget, or take a plausible-but-wrong action three steps deep before anyone notices. Set hard iteration limits and cost ceilings from day one — don’t wait for a runaway loop to teach you the lesson.

    Monitoring and Guardrails

    Whatever you deploy, observability is non-negotiable. At minimum, track:

  • Token usage per agent, per task type
  • Tool call success/failure rates
  • Loop iteration counts for agentic workflows
  • Wall-clock time per completed task

    • Tools like Prometheus paired with Grafana work fine for this if you’re already running that stack — you’re just treating each agent container like any other service, with custom metrics exported for token spend and loop depth.

    If you’re deploying agentic pipelines that run continuously (not just on-demand), it’s worth putting real uptime and cost alerting in front of them rather than checking dashboards manually. A managed monitoring service saves you from finding out about a runaway agent loop from your cloud bill instead of an alert.

    Cost and Infrastructure Considerations

    Agentic AI systems are meaningfully more expensive to run than single agents, for two reasons: multiple LLM calls per task, and the standing infrastructure (queue, orchestrator, state store) needed to coordinate them. Before you commit to an agentic architecture, estimate:

  • Average number of LLM calls per completed workflow
  • Peak concurrent workflows your infra needs to support
  • Storage/retention requirements for agent memory and logs

    • For most small-to-mid-size deployments, a single mid-tier VPS running Docker Compose is enough to prototype an agentic system before you need Kubernetes. Scale the orchestration layer only once you’ve validated the workflow actually needs multiple coordinating agents — plenty of teams build a five-agent system for a job a single well-prompted agent could have handled.

    FAQ

    Is agentic AI just marketing for AI agents?
    Not entirely. There’s a real technical distinction: an AI agent handles one bounded task, while agentic AI coordinates multiple steps or multiple agents toward a broader goal with less human oversight. But yes, a lot of vendor marketing blurs the line to make simple agent wrappers sound more sophisticated than they are.

    Do I need a multi-agent framework to build agentic AI?
    No. You can build agentic behavior with a single agent that loops and re-plans based on its own intermediate outputs. Frameworks like LangGraph or CrewAI make multi-agent coordination easier, but they’re not required to get agentic behavior.

    Which is cheaper to run in production, an AI agent or agentic AI?
    A single AI agent is almost always cheaper — fewer LLM calls per task and a simpler infrastructure footprint. Agentic AI systems multiply both cost and complexity because of iterative planning and inter-agent coordination overhead.

    Can I containerize an agentic AI system the same way I containerize microservices?
    Yes. Each agent can run as its own container, coordinated through a message queue like Redis or RabbitMQ, similar to standard microservice patterns. The main addition is shared state/memory management between agents.

    What’s the biggest risk with agentic AI in production?
    Uncontrolled iteration loops and cost overruns. Without hard limits on planning steps and token budgets, an agentic system can spiral into repeated LLM calls without a human noticing until the bill or the logs show it.

    Should I start with an AI agent or jump straight to agentic AI?
    Start with a single AI agent. Validate that the task genuinely requires multi-step planning before adding orchestration complexity — most production use cases are solved by a well-scoped single agent.

    Bottom Line

    The ai agent vs agentic ai distinction isn’t academic — it directly determines your deployment shape, your monitoring needs, and your cost profile. Start narrow with a single agent wired to a fixed toolset, containerize it, and only reach for a full agentic architecture once you’ve proven the workflow actually needs multi-step, multi-agent coordination. Most teams over-build here; don’t be one of them.

    Related articles:

  • AI Phone Agents: Self-Hosting Guide with Docker

    AI Phone Agents: A Developer’s Guide to Self-Hosting Voice Automation

    Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

    AI phone agents — automated systems that answer calls, hold natural conversations, and route or resolve requests without a human on the line — have moved from novelty to production infrastructure. If you run servers for a living, you’ve probably already been asked to “just wire up an AI agent to the support line.” This guide walks through what that actually takes: the architecture, the Docker-based deployment, the telephony integration, and the operational tradeoffs of running AI phone agents on infrastructure you control instead of a black-box SaaS.

    What Are AI Phone Agents?

    An AI phone agent is a pipeline that converts an inbound or outbound phone call into a real-time conversation with a language model. Under the hood, it’s a chain of four systems working in near real time:

  • Telephony layer — receives/places the call and streams audio (Twilio, Telnyx, or SIP trunking)
  • Speech-to-text (STT) — transcribes caller audio to text as it arrives
  • LLM orchestration — generates a response, calls tools/APIs, manages conversation state
  • Text-to-speech (TTS) — converts the LLM’s reply back into audio streamed to the caller

    • Commercial platforms like Vapi or Retell bundle all of this behind an API key. That’s fine for a prototype, but once you’re handling real call volume, you hit the same wall every SaaS-wrapped-open-source product eventually hits: per-minute pricing that scales linearly with usage, vendor lock-in on voice models, and no control over where audio and transcripts are stored — a real problem if you’re in a regulated industry.

    Why Self-Host AI Phone Agents

    Running your own stack makes sense once you clear a few hundred call-minutes a day, or once data residency actually matters. The tradeoffs are concrete:

  • Cost at scale — self-hosted STT/TTS on a GPU box is often 5-10x cheaper per minute than metered API pricing once utilization is decent
  • Latency control — colocating your LLM inference and telephony bridge in the same region cuts round-trip latency, which matters a lot for conversational feel
  • Data ownership — call transcripts and audio never leave infrastructure you control, which simplifies compliance
  • Model flexibility — swap in any open-weight LLM or fine-tune on your own call logs

    • The cost is operational complexity. You’re now responsible for uptime, GPU capacity, and a real-time audio pipeline instead of a REST call.

    Core Architecture

    A minimal self-hosted stack looks like this:

    Caller --> Twilio Media Streams (WebSocket) --> Orchestrator (Python/Node)
                                                       |
                    +----------------------------------+----------------------------------+
                    |                                  |                                  |
              Whisper (STT)                     LLM (vLLM/Ollama)                  Piper/Coqui (TTS)

    Twilio (or an equivalent provider) opens a bidirectional WebSocket media stream for the call. Your orchestrator service buffers incoming audio chunks, feeds them to a streaming STT model, passes the resulting text to an LLM, and streams the LLM’s response through TTS back over the same socket. All of this needs to happen with well under a second of round-trip latency to feel like a real conversation.

    Docker Compose Setup

    Here’s a working docker-compose.yml for the core services. It assumes a GPU-enabled host for the STT and LLM containers.

    version: "3.9"

services:
  orchestrator:
    build: ./orchestrator
    ports:
      - "8080:8080"
    environment:
      - TWILIO_ACCOUNT_SID=${TWILIO_ACCOUNT_SID}
      - TWILIO_AUTH_TOKEN=${TWILIO_AUTH_TOKEN}
      - STT_URL=http://whisper:9000
      - LLM_URL=http://llm:11434
      - TTS_URL=http://tts:5002
    depends_on:
      - whisper
      - llm
      - tts

  whisper:
    image: onerahmet/openai-whisper-asr-webservice:latest
    environment:
      - ASR_MODEL=medium
      - ASR_ENGINE=faster_whisper
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  llm:
    image: ollama/ollama:latest
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  tts:
    image: ghcr.io/coqui-ai/tts:latest
    command: ["--model_name", "tts_models/en/vctk/vits"]

volumes:
  ollama_data:

    Bring it up with:

    docker compose up -d
docker compose logs -f orchestrator

    If you haven’t tuned a Docker Compose stack for GPU workloads before, our Docker Compose GPU deployment guide covers the NVIDIA Container Toolkit setup you’ll need before these containers will actually see the GPU.

    Setting Up Speech-to-Text and Text-to-Speech

    For STT, OpenAI’s Whisper run through faster-whisper gives near real-time transcription on a single consumer GPU. The medium model is a good balance of accuracy and latency for phone-quality (8kHz) audio; large-v3 is more accurate but adds 200-400ms of latency per chunk, which is often not worth it for a live call.

    For TTS, Coqui TTS and Piper are the two open-source options worth evaluating. Piper is CPU-friendly and extremely fast, which matters if you’re not running TTS on a GPU box — it can hit sub-200ms synthesis on modest hardware. Coqui’s VITS models sound more natural but need a GPU to stay fast enough for live conversation.

    A minimal orchestrator loop (Python, using websockets and httpx) looks like this:

    import asyncio
import httpx
import websockets

async def handle_call(websocket):
    async for message in websocket:
        audio_chunk = decode_media_stream(message)
        transcript = await transcribe(audio_chunk)
        if transcript:
            reply = await query_llm(transcript)
            audio_reply = await synthesize(reply)
            await websocket.send(encode_media_stream(audio_reply))

async def transcribe(chunk: bytes) -> str:
    async with httpx.AsyncClient() as client:
        resp = await client.post("http://whisper:9000/asr", content=chunk)
        return resp.json().get("text", "")

async def query_llm(text: str) -> str:
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://llm:11434/api/generate",
            json={"model": "llama3.1:8b", "prompt": text, "stream": False},
        )
        return resp.json()["response"]

async def synthesize(text: str) -> bytes:
    async with httpx.AsyncClient() as client:
        resp = await client.post("http://tts:5002/api/tts", json={"text": text})
        return resp.content

    This is intentionally bare — production versions need interrupt handling (so the caller can talk over the agent), silence detection to know when the caller has finished speaking, and streaming responses from the LLM rather than waiting for the full completion.

    Connecting to a Telephony Provider

    Twilio’s Media Streams API is the easiest on-ramp: you point a TwiML <Connect><Stream> verb at your orchestrator’s WebSocket URL and Twilio handles the PSTN side entirely. For higher call volume or lower per-minute cost, a SIP trunk into Asterisk or FreeSWITCH gives you more control but adds real telephony ops work — NAT traversal, codec negotiation, and carrier peering are not trivial.

    A basic TwiML response to bridge an inbound call into your stream:

    <Response>
  <Connect>
    <Stream url="wss://agents.example.com/media" />
  </Connect>
</Response>

    Scaling and Monitoring Your AI Phone Agent Infrastructure

    Once you’re past a handful of concurrent calls, a single Docker host won’t cut it. Each active call pins an STT stream, an LLM context, and a TTS session simultaneously, so concurrency scales GPU memory usage fast. Plan for:

  • Horizontal scaling of the orchestrator behind a load balancer, with sticky WebSocket routing per call
  • GPU pooling — batch multiple Whisper/TTS requests where the model server supports it (faster-whisper and vLLM both do)
  • Call queuing — a Redis-backed queue to hold calls when GPU capacity is saturated, rather than dropping them

    • Monitoring matters more here than in most services because failures are invisible until a caller hangs up frustrated. We run BetterStack for uptime checks on the orchestrator’s health endpoint and log-based alerting on STT/TTS error rates — worth setting up before you put a phone number in front of real customers. If you haven’t built out alerting for a service like this before, see our guide to self-hosted monitoring stacks for the Prometheus/Grafana side of things.

    For the hosting layer itself, GPU-backed instances are the main cost driver. DigitalOcean’s GPU droplets are a reasonable starting point if you want simple provisioning without committing to reserved capacity, and they scale down cleanly if call volume doesn’t materialize. If you’re running steady, predictable volume instead, Hetzner’s dedicated GPU servers come in significantly cheaper per hour for always-on workloads.

    Security Considerations

    Phone calls carry PII by default — names, account numbers, sometimes payment details read aloud. A few non-negotiables:

  • Encrypt call recordings and transcripts at rest, and set a retention policy instead of keeping everything indefinitely
  • Put the orchestrator’s public WebSocket endpoint behind TLS termination and IP allowlisting for your telephony provider’s signaling ranges
  • Never let the LLM’s tool-calling layer execute arbitrary actions (refunds, account changes) without a secondary confirmation step — prompt injection via a caller’s spoken input is a real attack surface, not a theoretical one
  • Front the public endpoints with Cloudflare for DDoS protection and WAF rules, since a public-facing voice endpoint is an easy target for automated abuse

    • If you’re exposing any of this stack to the public internet, our Linux server hardening checklist is a reasonable baseline before you take a first production call.

    Cost Comparison: Self-Hosted vs SaaS

    Rough numbers based on a single GPU instance handling moderate concurrent call volume:

    | Approach | Cost per minute | Setup effort | Data control |
    |—|—|—|—|
    | SaaS (Vapi/Retell) | $0.05-$0.15 | Low | None |
    | Self-hosted (cloud GPU) | $0.01-$0.03 | High | Full |
    | Self-hosted (dedicated GPU) | $0.005-$0.015 | High | Full |

    The breakeven point is usually somewhere around 5,000-10,000 call-minutes a month, depending on which models you’re running and how efficiently you batch inference. Below that volume, the engineering time to build and maintain the pipeline usually costs more than the SaaS markup.

    Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

    FAQ

    Do AI phone agents need a GPU to run?
    STT and TTS models can run on CPU, but latency suffers enough that conversations feel sluggish. For anything beyond a low-volume prototype, a GPU for Whisper and your LLM is effectively required to keep round-trip latency under a second.

    Can I use GPT-4 or Claude instead of a self-hosted LLM?
    Yes — nothing about this architecture requires an open-weight model. Swapping the query_llm function to call an API instead of Ollama works fine; you just give up the cost and data-residency benefits of full self-hosting for that piece of the pipeline.

    How do I handle interruptions when the caller talks over the agent?
    You need voice activity detection (VAD) running continuously on the incoming stream, and the orchestrator must be able to cancel an in-flight TTS response the moment new caller speech is detected. Libraries like webrtcvad or Silero VAD handle the detection; the cancellation logic is on you.

    What’s the biggest latency bottleneck in this stack?
    Usually the LLM generation step, especially if you wait for a full completion before starting TTS. Streaming tokens from the LLM into TTS as they arrive, rather than waiting for the full response, is the single biggest latency win available.

    Is this legal for handling customer calls?
    Generally yes, but call recording consent laws vary by state and country — some require two-party consent before recording. Check local requirements and play a disclosure message before recording begins.

    How many concurrent calls can one GPU handle?
    Depends heavily on model size and GPU memory, but a single mid-range GPU (e.g., an RTX 4090 or A10) running faster-whisper medium plus a small quantized LLM can typically handle 8-15 concurrent calls before latency degrades noticeably.

    Wrapping Up

    Self-hosting AI phone agents is a legitimate infrastructure project, not just a wrapper around an API key — you’re building a real-time audio pipeline with the same latency and reliability demands as VoIP itself. Start with the Docker Compose stack above on a single GPU box, get the conversation loop working end to end, and only invest in horizontal scaling once you have actual call volume to justify it.

    Related articles: