Public directories exist (browsing GitHub’s ai-agents topic is a good way to survey the landscape), but for internal, proprietary, or credential-bearing agents, self-hosting is non-negotiable.

The Problem With Scattered Agent Listings

Without a directory, agent metadata lives in README files, internal wikis, and tribal knowledge. When someone leaves the team, that knowledge often leaves with them. A directory forces structure: every agent needs a name, an owner, a status, and a description before it’s considered “registered.” That structure is what makes the catalog searchable and auditable instead of just another list.

Core Architecture for a Self-Hosted Directory

Keep the stack boring and operationally simple. You don’t need a microservices architecture for this — a single web app, a relational database, and a reverse proxy will comfortably handle thousands of agent records and moderate traffic.

Choosing Your Stack

A solid default:

If you haven’t settled on a proxy yet, our reverse proxy comparison guide breaks down when Caddy’s automatic HTTPS beats a hand-rolled Nginx config.

Database Schema for Agent Metadata

Start with a schema that captures the essentials without over-engineering:

CREATE TABLE agents (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL UNIQUE,
    slug VARCHAR(255) NOT NULL UNIQUE,
    description TEXT NOT NULL,
    owner_email VARCHAR(255) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'active', -- active, deprecated, experimental
    repo_url TEXT,
    endpoint_url TEXT,
    tags TEXT[] DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_agents_tags ON agents USING GIN (tags);
CREATE INDEX idx_agents_search ON agents USING GIN (to_tsvector('english', name || ' ' || description));

The GIN index on the tsvector column is what makes full-text search fast without bolting on Elasticsearch. For a directory of a few thousand entries, Postgres full-text search is plenty — don’t reach for a dedicated search engine until you actually need it.

Deploying the Stack with Docker Compose

Here’s a minimal but production-shaped Compose file: app, database, and reverse proxy, all networked together with named volumes for persistence.

docker-compose.yml Walkthrough

version: "3.9"

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: agents_directory
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./schema.sql:/docker-entrypoint-initdb.d/schema.sql:ro
    networks:
      - internal

  app:
    build: ./app
    restart: unless-stopped
    environment:
      DATABASE_URL: postgres://${DB_USER}:${DB_PASSWORD}@db:5432/agents_directory
      NODE_ENV: production
    depends_on:
      - db
    networks:
      - internal
      - web

  proxy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
    depends_on:
      - app
    networks:
      - web

volumes:
  pgdata:
  caddy_data:

networks:
  internal:
  web:

Notice the database sits only on the internal network — it’s never exposed to the web network or the host’s public interface. The app service bridges both networks, and only the proxy touches ports 80/443. This is the same segmentation pattern we cover in more depth in our Docker networking fundamentals guide.

Environment Variables and Secrets

Never bake credentials into the image. Use a .env file excluded from version control:

# .env
DB_USER=directory_admin
DB_PASSWORD=$(openssl rand -base64 24)

Generate the password once and store it in a secrets manager or your team’s password vault — not in Slack, not in a commit. Bring the stack up with:

docker compose --env-file .env up -d

Check that everything started cleanly:

docker compose ps
docker compose logs -f app

Adding Search, Tags, and Filtering

Once agents are in Postgres, exposing search is a single query using the tsvector index built earlier:

SELECT id, name, description, tags
FROM agents
WHERE to_tsvector('english', name || ' ' || description) @@ plainto_tsquery('english', $1)
ORDER BY updated_at DESC
LIMIT 20;

On the frontend, pair this with tag-based filtering (WHERE tags @> ARRAY[$1]) so users can narrow results to, say, deployment or customer-support agents. Keep the API simple: a GET /api/agents?search=&tag=&status= endpoint covers 90% of real usage without needing a query DSL.

Monitoring, Backups, and Uptime

A directory that goes down silently defeats the purpose — people will stop trusting it and fall back to spreadsheets. Two things matter here:

#!/usr/bin/env bash
set -euo pipefail

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
docker compose exec -T db pg_dump -U "$DB_USER" agents_directory | gzip > "backups/agents_directory_${TIMESTAMP}.sql.gz"

# Keep the last 14 days of backups
find backups/ -name "*.sql.gz" -mtime +14 -delete

Wire that into a nightly cron job and sync the backups/ directory to object storage. If you’re not already backing up your other self-hosted tools, our VPS backup strategy guide covers the same pattern for other services.

Hardening and Scaling in Production

For a single-team internal tool, one modest VPS is enough. DigitalOcean droplets and Hetzner cloud instances both work well here — Hetzner tends to win on raw price-per-core if your team is EU-based or latency-insensitive, while DigitalOcean’s managed database add-on is worth considering once you outgrow a single-container Postgres instance.

Basic hardening checklist before you expose this beyond localhost:

Once you have real traffic and want to track SEO performance for a public-facing version of the directory (some teams do publish a curated, non-sensitive subset), a tool like SE Ranking can help you monitor how discovery pages perform in search results.

Recommended: Ready to put this into practice? SE Ranking is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Q: Do I need a vector database to build an AI agents directory?
A: No. Unless you’re doing semantic search over thousands of long-form agent descriptions, Postgres full-text search with a GIN index is fast enough and much simpler to operate.

Q: Should the directory expose agent credentials or API keys directly?
A: No — store only references (e.g., a secrets-manager key name or vault path), never the actual secret value, in the directory database itself.

Q: Can I run this on a single small VPS?
A: Yes. A 2GB RAM droplet from DigitalOcean or a comparable Hetzner CX instance comfortably runs Postgres, the app, and Caddy for an internal directory serving a small-to-mid-size engineering team.

Q: How do I keep the directory from going stale?
A: Add a CI step that fails a pull request if a new agent is deployed without a corresponding directory entry — treat directory registration as a deployment requirement, not an afterthought.

Q: What’s the difference between this and a public directory like the GitHub topic page?
A: Public directories are for discovering third-party or open-source agents. A self-hosted directory tracks your own internal agents, including private endpoints, owners, and operational status that should never be public.

Q: Is Docker Compose enough, or do I need Kubernetes?
A: Compose is enough for the vast majority of internal tools like this. Move to Kubernetes only if you need multi-node redundancy or you’re already running K8s for everything else and don’t want a one-off Compose host.

Wrapping Up

A self-hosted AI agents directory doesn’t need to be complicated: Postgres for storage and search, a thin app layer for the API and UI, Caddy or Nginx for TLS, and a nightly backup job. The value isn’t in the tech stack — it’s in forcing every agent your team ships to have an owner, a description, and a status before it goes live. Start with the schema above, get it running with Docker Compose this week, and add search and tagging once real usage tells you what people actually search for.