Building AI Agents From Scratch: A Practical DevOps Guide
Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.
Building AI agents from scratch is one of the most common requests DevOps and platform teams get once large language models move from a demo into something people actually want running in production. This guide walks through the architecture, infrastructure, and operational decisions involved in building AI agents from scratch — without relying on a heavyweight no-code platform — so you understand exactly what you’re deploying and why.
Most tutorials on this topic either hand-wave the infrastructure (“just call the API”) or over-engineer a toy example into something unmaintainable. This article splits the difference: it treats an agent as a normal piece of backend software with a scheduler, a state store, and a set of tools, then shows how to wire those pieces together and run them reliably on a VPS.
What “Building AI Agents From Scratch” Actually Means
Before writing code, it helps to define the term precisely. An AI agent, in the practical engineering sense, is a loop: it receives an input, decides on an action (call a tool, ask a clarifying question, or respond), executes that action, observes the result, and repeats until it reaches a stopping condition. Building AI agents from scratch means you own every part of that loop — the prompt construction, the tool-calling logic, the state management, and the deployment — rather than delegating it to a hosted agent-builder product.
This distinction matters operationally. A hosted platform trades control for convenience: you get a faster start but inherit its rate limits, its logging model, and its outage schedule. When you build AI agents from scratch, you decide how requests are queued, how failures are retried, and where the data lives. For teams already running Docker-based infrastructure, this is usually the more maintainable long-term choice.
Core Components of a From-Scratch Agent
At minimum, an agent built this way needs:
If you’re comparing this manual approach against a workflow-automation tool, it’s worth reading How to Build AI Agents With n8n first — n8n handles the orchestration loop and tool registry for you at the cost of some flexibility, which is a legitimate tradeoff for many teams.
Designing the Agent Loop
The orchestration loop is the part most tutorials skip past, but it’s where most production bugs live. A naive implementation calls the model, executes whatever tool it requests, and loops forever until the model stops requesting tools. In practice you need explicit guardrails.
Bounding Iterations and Cost
Every agent loop needs a hard iteration cap and a cost/token budget per task. Without this, a model that gets stuck in a reasoning loop (asking for the same tool repeatedly, or looping between two contradictory conclusions) will burn API credits indefinitely. A simple counter passed through the loop, combined with a maximum wall-clock timeout, catches the vast majority of runaway cases.
Structuring Tool Calls
Tool definitions should be strict — a JSON schema per tool, validated before execution, not just passed straight from the model’s output into a shell command or database query. This is the single most important security boundary when building AI agents from scratch: the model’s output is untrusted input, exactly like a form submission from a browser. Treat it accordingly — validate types, whitelist allowed operations, and never string-concatenate model output directly into a shell command or SQL query.
# Minimal example: validate a tool call before executing it
python3 - <<'EOF'
import json, subprocess, shlex
def run_tool(tool_name, args: dict):
ALLOWED_TOOLS = {"get_weather", "search_docs"}
if tool_name not in ALLOWED_TOOLS:
raise ValueError(f"Unregistered tool: {tool_name}")
if tool_name == "search_docs":
query = str(args.get("query", ""))[:200] # bound input length
return subprocess.run(
["grep", "-ri", query, "/data/docs"],
capture_output=True, text=True, timeout=5
).stdout
raise NotImplementedError(tool_name)
print(run_tool("search_docs", {"query": "deployment"}))
EOFHandling Partial Failures
Tool calls fail — APIs time out, databases lock, shell commands return nonzero exit codes. The loop needs to feed failures back to the model as observations rather than crashing the whole task, so the agent can retry, choose a different tool, or gracefully report that it couldn’t complete the request. This is different from typical request/response error handling because the “caller” here is the model itself, which needs the failure expressed in natural language it can reason about.
Choosing the Right Infrastructure
Once the loop is designed, the next question is where it runs. Building AI agents from scratch doesn’t require Kubernetes or a large cloud budget — a single VPS with Docker Compose is enough for most single-tenant or small-team agent deployments.
Containerizing the Agent Process
Package the agent runtime as a container with clearly defined environment variables for API keys, model selection, and tool endpoints. Keep the container stateless — persist conversation state and task status externally (Redis or Postgres), not on the container’s local filesystem, so the process can restart or scale without losing in-flight work.
If you’re already running Postgres for other services, this Postgres Docker Compose setup guide is a reasonable starting point for the state store, and Redis Docker Compose works well if you’d rather keep agent state in-memory with periodic persistence. For managing secrets like API keys across containers, see Docker Compose Secrets rather than baking keys into the image.
Scheduling and Triggering
Agents can be triggered synchronously (an HTTP request waits for the full loop to complete) or asynchronously (a webhook enqueues a task, a worker processes it, a callback or polling endpoint returns the result). For anything that might take more than a few seconds — which most multi-step agent tasks do — asynchronous triggering is the more reliable choice, since it avoids tying up an HTTP connection for the duration of an unpredictable model reasoning loop.
A minimal docker-compose.yml for a two-process setup (API + worker) might look like this:
version: "3.8"
services:
agent-api:
build: ./api
ports:
- "8080:8080"
environment:
- REDIS_URL=redis://redis:6379
depends_on:
- redis
agent-worker:
build: ./worker
environment:
- REDIS_URL=redis://redis:6379
- MODEL_API_KEY=${MODEL_API_KEY}
depends_on:
- redis
redis:
image: redis:7-alpine
volumes:
- redis-data:/data
volumes:
redis-data:Observability and Debugging
Agent behavior is harder to debug than typical request/response services because the “logic” isn’t fully expressed in your code — it emerges from the model’s interpretation of a prompt at runtime. Building AI agents from scratch means you’re also responsible for building the observability that makes this debuggable.
Logging Every Model Call and Tool Invocation
Log the full prompt sent to the model, the raw response, every tool call requested, and every tool result returned — not just a summary. When an agent produces an unexpected output days later, the only way to understand why is to replay exactly what the model saw at each step. Structured logging (JSON lines, one per model/tool interaction) makes this searchable later.
docker compose logs -f agent-worker | grep '"event":"tool_call"'If you’re running this stack alongside other Docker services, Docker Compose Logs covers the general debugging workflow this pattern builds on.
Tracking Cost and Latency Per Task
Each agent task should log total tokens consumed and wall-clock duration. Over time this data tells you which tasks are expensive outliers, which is essential for capacity planning and for deciding whether a given task should be routed to a smaller, cheaper model instead of your default.
Security Considerations Specific to Agents
An agent that can call tools is functionally similar to a service account with programmatic access to your systems — the model is deciding what to execute, but your infrastructure is executing it. Building AI agents from scratch puts this responsibility squarely on you rather than a platform vendor.
For a deeper walkthrough of the security-specific failure modes (prompt injection into tool arguments, excessive tool permissions, unbounded recursive tool calls), see AI Agent Security: A Practical Guide for DevOps.
Deployment and Hosting
A single agent process handling moderate traffic runs comfortably on a small VPS. If you’re setting up infrastructure specifically for this, a provider like DigitalOcean or Hetzner gives you a straightforward Docker-ready VPS without committing to a full managed Kubernetes bill before you know your actual traffic pattern. Start with a single droplet or server running Docker Compose, and only move to container orchestration once you have real usage data justifying the added complexity.
For the initial server setup itself, an unmanaged VPS hosting guide covers the baseline hardening (SSH keys, firewall rules, unattended upgrades) worth doing before you deploy anything that holds API credentials.
Zero-Downtime Restarts
Because agent tasks can run for tens of seconds to a few minutes, a naive docker compose restart during a deploy can kill in-flight tasks. Use a queue-based worker pattern (the worker pulls from Redis or a similar queue) so a graceful shutdown can finish the current task before exiting, and the orchestrator can spin up a replacement worker without losing queued work.
Recommended: Ready to put this into practice? DigitalOcean is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.
FAQ
Do I need a specialized “agent framework” to build AI agents from scratch?
No. Frameworks like LangChain or LlamaIndex can save time on boilerplate (prompt templates, tool-calling glue code), but they’re not required. Building AI agents from scratch with a plain HTTP client and a hand-written loop gives you more visibility into what’s actually happening at each step, which is often worth the extra initial code for production systems.
How is building AI agents from scratch different from using n8n or a similar automation tool?
n8n and similar tools provide the orchestration loop, tool registry, and scheduling out of the box through a visual workflow editor, trading some flexibility for a much faster setup. Building AI agents from scratch means writing that orchestration yourself, which gives finer control over retry logic, cost tracking, and custom tool behavior at the cost of more initial engineering time.
What’s the biggest infrastructure mistake teams make when building AI agents from scratch?
Running the agent loop synchronously inside an HTTP request handler with no timeout or iteration cap. This leads to hung connections, unpredictable API bills, and no way to observe what the agent is doing mid-task. A queue-based worker with logging at every step avoids all three problems.
Can I run building-AI-agents-from-scratch workloads on the same VPS as my other Docker services?
Yes, as long as you isolate resource usage — set memory and CPU limits on the agent worker container so a runaway task can’t starve your other services, and keep its state store (Redis or Postgres) properly backed up alongside the rest of your stack.
Conclusion
Building AI agents from scratch is a reasonable, well-scoped engineering task once you break it into its component parts: a bounded orchestration loop, a validated tool registry, an external state store, and a container runtime you already know how to operate. None of these pieces require exotic infrastructure — a single Docker Compose stack on a modest VPS is enough to get a reliable agent into production. The work that actually differentiates a good implementation from a fragile one is the same work that differentiates any backend service: careful input validation, structured logging, iteration and cost bounds, and a deployment process that doesn’t kill in-flight work. Get those right, and building AI agents from scratch becomes a maintainable, extensible part of your infrastructure rather than a one-off experiment. For further reading on the underlying model APIs, the OpenAI API documentation and Docker’s official documentation are both useful references while implementing the patterns described above.
Leave a Reply