How to Build a Self-Hosted AI Agents Directory with Docker

Disclosure: This post contains one or more links to providers we have a real, registered affiliate/referral relationship with. We may earn a commission at no extra cost to you if you sign up through them.

If you’ve tried to keep track of every autonomous agent, LangChain wrapper, and CLI-based assistant your team has spun up, you already know the problem: there’s no single source of truth. Spreadsheets rot. Slack threads get buried. A proper ai agents directory — a searchable, self-hosted catalog of every agent, its capabilities, endpoints, and owner — solves this, and you can stand one up in an afternoon with Docker.

This guide walks through the architecture, the Docker Compose stack, the database schema, and the operational concerns (backups, monitoring, hardening) you need before pushing this to production.

Why Build an AI Agents Directory in the First Place

Most teams don’t set out to build a directory — they back into needing one. First it’s two agents: a support-ticket triager and a changelog summarizer. Then it’s a dozen: deployment bots, code review assistants, data pipeline monitors, customer-facing chat agents. Without a catalog, you end up with duplicate agents solving the same problem, orphaned agents nobody remembers building, and zero visibility into which agents have access to which credentials.

A self-hosted directory gives you:

A single searchable index of every agent, tagged by capability, owner, and status

An audit trail of which agents have access to which APIs or secrets

A discovery layer so engineers stop reinventing agents that already exist

A place to document rate limits, cost per run, and failure modes

Public directories exist (browsing GitHub’s ai-agents topic is a good way to survey the landscape), but for internal, proprietary, or credential-bearing agents, self-hosting is non-negotiable.

The Problem With Scattered Agent Listings

Without a directory, agent metadata lives in README files, internal wikis, and tribal knowledge. When someone leaves the team, that knowledge often leaves with them. A directory forces structure: every agent needs a name, an owner, a status, and a description before it’s considered “registered.” That structure is what makes the catalog searchable and auditable instead of just another list.

Core Architecture for a Self-Hosted Directory

Keep the stack boring and operationally simple. You don’t need a microservices architecture for this — a single web app, a relational database, and a reverse proxy will comfortably handle thousands of agent records and moderate traffic.

Choosing Your Stack

A solid default:

App layer: Node.js (Express or Next.js) or Python (FastAPI) serving a REST API and a lightweight frontend

Database: PostgreSQL — relational, supports full-text search out of the box, and well-documented at postgresql.org

Reverse proxy: Nginx or Caddy for TLS termination

Container runtime: Docker Compose for single-host deployments; move to Swarm or Kubernetes only if you outgrow one VPS

If you haven’t settled on a proxy yet, our reverse proxy comparison guide breaks down when Caddy’s automatic HTTPS beats a hand-rolled Nginx config.

Database Schema for Agent Metadata

Start with a schema that captures the essentials without over-engineering:

CREATE TABLE agents (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL UNIQUE,
    slug VARCHAR(255) NOT NULL UNIQUE,
    description TEXT NOT NULL,
    owner_email VARCHAR(255) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'active', -- active, deprecated, experimental
    repo_url TEXT,
    endpoint_url TEXT,
    tags TEXT[] DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_agents_tags ON agents USING GIN (tags);
CREATE INDEX idx_agents_search ON agents USING GIN (to_tsvector('english', name || ' ' || description));

The GIN index on the tsvector column is what makes full-text search fast without bolting on Elasticsearch. For a directory of a few thousand entries, Postgres full-text search is plenty — don’t reach for a dedicated search engine until you actually need it.

Deploying the Stack with Docker Compose

Here’s a minimal but production-shaped Compose file: app, database, and reverse proxy, all networked together with named volumes for persistence.

docker-compose.yml Walkthrough

version: "3.9"

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: agents_directory
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./schema.sql:/docker-entrypoint-initdb.d/schema.sql:ro
    networks:
      - internal

  app:
    build: ./app
    restart: unless-stopped
    environment:
      DATABASE_URL: postgres://${DB_USER}:${DB_PASSWORD}@db:5432/agents_directory
      NODE_ENV: production
    depends_on:
      - db
    networks:
      - internal
      - web

  proxy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
    depends_on:
      - app
    networks:
      - web

volumes:
  pgdata:
  caddy_data:

networks:
  internal:
  web:

Notice the database sits only on the internal network — it’s never exposed to the web network or the host’s public interface. The app service bridges both networks, and only the proxy touches ports 80/443. This is the same segmentation pattern we cover in more depth in our Docker networking fundamentals guide.

Environment Variables and Secrets

Never bake credentials into the image. Use a .env file excluded from version control:

# .env
DB_USER=directory_admin
DB_PASSWORD=$(openssl rand -base64 24)

Generate the password once and store it in a secrets manager or your team’s password vault — not in Slack, not in a commit. Bring the stack up with:

docker compose --env-file .env up -d

Check that everything started cleanly:

docker compose ps
docker compose logs -f app

Adding Search, Tags, and Filtering

Once agents are in Postgres, exposing search is a single query using the tsvector index built earlier:

SELECT id, name, description, tags
FROM agents
WHERE to_tsvector('english', name || ' ' || description) @@ plainto_tsquery('english', $1)
ORDER BY updated_at DESC
LIMIT 20;

On the frontend, pair this with tag-based filtering (WHERE tags @> ARRAY[$1]) so users can narrow results to, say, deployment or customer-support agents. Keep the API simple: a GET /api/agents?search=&tag=&status= endpoint covers 90% of real usage without needing a query DSL.

Monitoring, Backups, and Uptime

A directory that goes down silently defeats the purpose — people will stop trusting it and fall back to spreadsheets. Two things matter here:

Uptime monitoring: A service like BetterStack will alert you the moment the directory or its API goes unreachable, before your team notices and starts asking questions in Slack.

Automated backups: Postgres data is the only thing that actually matters in this stack. Automate pg_dump on a cron schedule and ship the output somewhere off-host.

#!/usr/bin/env bash
set -euo pipefail

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
docker compose exec -T db pg_dump -U "$DB_USER" agents_directory | gzip > "backups/agents_directory_${TIMESTAMP}.sql.gz"

# Keep the last 14 days of backups
find backups/ -name "*.sql.gz" -mtime +14 -delete

Wire that into a nightly cron job and sync the backups/ directory to object storage. If you’re not already backing up your other self-hosted tools, our VPS backup strategy guide covers the same pattern for other services.

Hardening and Scaling in Production

For a single-team internal tool, one modest VPS is enough. DigitalOcean droplets and Hetzner cloud instances both work well here — Hetzner tends to win on raw price-per-core if your team is EU-based or latency-insensitive, while DigitalOcean’s managed database add-on is worth considering once you outgrow a single-container Postgres instance.

Basic hardening checklist before you expose this beyond localhost:

Put the directory behind your VPN or SSO proxy — don’t expose agent metadata (especially endpoint_url and owner emails) to the open internet

Enforce TLS via Caddy’s automatic HTTPS or a Let’s Encrypt cert in Nginx

Rotate the Postgres password and restrict pg_hba.conf to the internal Docker network only

Run docker scout or trivy against your app image periodically to catch known CVEs in base images

Rate-limit the search API to prevent it from being scraped wholesale

Once you have real traffic and want to track SEO performance for a public-facing version of the directory (some teams do publish a curated, non-sensitive subset), a tool like SE Ranking can help you monitor how discovery pages perform in search results.

Recommended: Ready to put this into practice? SE Ranking is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Q: Do I need a vector database to build an AI agents directory?
A: No. Unless you’re doing semantic search over thousands of long-form agent descriptions, Postgres full-text search with a GIN index is fast enough and much simpler to operate.

Q: Should the directory expose agent credentials or API keys directly?
A: No — store only references (e.g., a secrets-manager key name or vault path), never the actual secret value, in the directory database itself.

Q: Can I run this on a single small VPS?
A: Yes. A 2GB RAM droplet from DigitalOcean or a comparable Hetzner CX instance comfortably runs Postgres, the app, and Caddy for an internal directory serving a small-to-mid-size engineering team.

Q: How do I keep the directory from going stale?
A: Add a CI step that fails a pull request if a new agent is deployed without a corresponding directory entry — treat directory registration as a deployment requirement, not an afterthought.

Q: What’s the difference between this and a public directory like the GitHub topic page?
A: Public directories are for discovering third-party or open-source agents. A self-hosted directory tracks your own internal agents, including private endpoints, owners, and operational status that should never be public.

Q: Is Docker Compose enough, or do I need Kubernetes?
A: Compose is enough for the vast majority of internal tools like this. Move to Kubernetes only if you need multi-node redundancy or you’re already running K8s for everything else and don’t want a one-off Compose host.

Wrapping Up

A self-hosted AI agents directory doesn’t need to be complicated: Postgres for storage and search, a thin app layer for the API and UI, Caddy or Nginx for TLS, and a nightly backup job. The value isn’t in the tech stack — it’s in forcing every agent your team ships to have an owner, a description, and a status before it goes live. Start with the schema above, get it running with Docker Compose this week, and add search and tagging once real usage tells you what people actually search for.