This isn’t a replacement for strategy — it’s a replacement for the grunt work that eats a strategist’s week.

Core Components of the Agent

A minimal but genuinely useful agent needs four pieces:

1. Crawler — fetches pages and extracts titles, meta tags, headings, and internal link structure.
2. Data enrichment layer — calls a rank-tracking or keyword API (we use SE Ranking in the example below, since it has a clean REST API and reasonable pricing for solo devs).
3. Reasoning layer — an LLM call that takes the crawl + rank data and returns structured JSON recommendations.
4. Output/action layer — writes results somewhere useful: a database, a Slack webhook, or a static report page.

We’ll build all four as a single Python service, then wrap it in Docker so it runs identically on your laptop and your production VPS.

Why Run It in Docker

You could run this as a bare cron job on your server, but you shouldn’t. Python dependency drift, mismatched requests/beautifulsoup4 versions, and “works on my machine” crawler bugs are exactly the class of problem Docker exists to kill. Packaging the agent as a container means:

If you’re new to container fundamentals, our Docker Compose guide for beginners covers the basics you’ll want before going further here.

Building the Agent

Step 1: The Crawler and Data Layer

Start with a Python module that crawls a small site and extracts the signals you care about. Keep it dependency-light — requests and BeautifulSoup are enough for a v1.

# crawler.py
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

def crawl_page(url):
    resp = requests.get(url, timeout=10, headers={"User-Agent": "SEOAgentBot/1.0"})
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")

    title = soup.title.string.strip() if soup.title and soup.title.string else ""
    meta_desc_tag = soup.find("meta", attrs={"name": "description"})
    meta_desc = meta_desc_tag["content"].strip() if meta_desc_tag else ""

    h1_tags = [h.get_text(strip=True) for h in soup.find_all("h1")]
    word_count = len(soup.get_text().split())

    links = set()
    domain = urlparse(url).netloc
    for a in soup.find_all("a", href=True):
        full_url = urljoin(url, a["href"])
        if urlparse(full_url).netloc == domain:
            links.add(full_url)

    return {
        "url": url,
        "title": title,
        "meta_description": meta_desc,
        "h1_count": len(h1_tags),
        "h1_tags": h1_tags,
        "word_count": word_count,
        "internal_links": list(links),
    }

This gives you a structured dict per page — enough to detect missing titles, duplicate H1s, and thin content before you even involve an LLM.

Step 2: The Reasoning Layer

Now feed the crawl output into an LLM call and force it to return structured JSON, not prose. This is the difference between a toy demo and something you can actually pipe into automation.

# analyzer.py
import json
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

SYSTEM_PROMPT = """You are an SEO auditing agent. Given crawl data for a single page,
return a JSON object with keys: issues (list of strings), severity ("low"|"medium"|"high"),
and suggested_title (string, only if the current title is weak or missing).
Be specific and terse. No fluff."""

def analyze_page(page_data: dict) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": json.dumps(page_data)},
        ],
    )
    return json.loads(response.choices[0].message.content)

Running this over every crawled page gives you a queue of prioritized fixes instead of a wall of raw data.

Step 3: Wiring It Together

# main.py
import json
import sys
from crawler import crawl_page
from analyzer import analyze_page

def run_agent(urls):
    report = []
    for url in urls:
        try:
            page = crawl_page(url)
            analysis = analyze_page(page)
            report.append({**page, "analysis": analysis})
        except Exception as e:
            report.append({"url": url, "error": str(e)})
    return report

if __name__ == "__main__":
    urls = sys.argv[1:] or ["https://thinkstreamtv.com"]
    result = run_agent(urls)
    print(json.dumps(result, indent=2))

At this point you have a working agent you can run with python main.py https://yoursite.com/page-1 https://yoursite.com/page-2.

Containerizing the Agent

Dockerfile and Compose Setup

Pin your Python version and dependencies so this runs the same everywhere:

# Dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

ENTRYPOINT ["python", "main.py"]

# requirements.txt
requests==2.32.3
beautifulsoup4==4.12.3
openai==1.40.0

Build and run it:

docker build -t seo-ai-agent .
docker run --rm -e OPENAI_API_KEY=$OPENAI_API_KEY 
  seo-ai-agent https://thinkstreamtv.com/some-article/

For scheduled runs against multiple sites, docker-compose with a .env file per project keeps things clean:

# docker-compose.yml
services:
  seo-agent:
    build: .
    env_file: .env
    command: ["https://thinkstreamtv.com", "https://thinkstreamtv.com/reviews/"]

Then trigger it from the host crontab:

# crontab -e
0 6 * * * cd /opt/seo-agent && docker compose run --rm seo-agent >> /var/log/seo-agent.log 2>&1

This runs the audit every morning at 6 AM and appends results to a log file you can pipe into Slack, a dashboard, or a nightly digest email.

Scheduling and Monitoring the Agent

A cron job that fails silently is worse than no automation at all. Two things are worth setting up immediately:

If you’re running this alongside other containerized services, our guide on monitoring Docker containers with uptime checks covers setting up alerting without a heavyweight observability stack.

Deploying to a Production VPS

Running this on your laptop is fine for testing, but a scheduled agent belongs on a server that’s always on. A $6-12/month VPS from DigitalOcean or Hetzner is plenty for a crawler hitting a handful of sites daily — you don’t need GPU compute since the heavy lifting happens via the OpenAI API, not locally.

Basic deployment checklist:

If your VPS is also hosting other Docker workloads, check our best VPS providers for Docker in 2026 comparison before committing to a plan size.

Security Considerations

A crawler with API keys is a juicy target if misconfigured. Keep the blast radius small:

Add USER appuser to the Dockerfile after creating a low-privilege user, and you’ve closed off most of the easy container-escape scenarios.

Wrapping Up

A self-hosted SEO AI agent isn’t magic — it’s a crawler, an API call, and an LLM prompt glued together, running on a schedule inside a container. The value isn’t in the AI being clever; it’s in never having to manually re-check meta descriptions across 200 pages again. Start small: one site, one daily cron run, and expand the analysis logic as you find gaps in what it catches.

Recommended: Ready to put this into practice? SE Ranking is a tool we use for exactly this, and we have a real, disclosed affiliate relationship with them.

FAQ

Do I need a paid LLM API to build an SEO AI agent?
No. You can start with a smaller, cheaper model like gpt-4o-mini or even a self-hosted open-weight model via Ollama for basic classification tasks. Reserve the more expensive models for final recommendation generation, not every crawl.

How is this different from tools like Screaming Frog or Ahrefs?
Those tools are excellent for raw crawling and rank data, and you can actually feed their exports into your agent’s reasoning layer instead of writing your own crawler. The agent’s value-add is the automated interpretation and prioritization step, not replacing the data source.

Can I run this without Docker?
Yes, but you’ll eventually hit dependency conflicts, especially if you run multiple Python projects on the same host. Docker isolates the agent’s environment so upgrades to one project don’t break another.

How often should the agent run?
Daily is reasonable for active sites; weekly is fine for smaller ones. Running it more frequently than your content actually changes just burns API credits for no new insight.

Is it safe to let the agent auto-publish changes?
Not recommended initially. Have it open a pull request or write to a review queue rather than pushing directly to production content, at least until you trust its output over a few months of runs.

What’s the cheapest way to host this?
A $6/month Hetzner or DigitalOcean droplet running Docker and a cron job is enough for most solo projects — you’re paying for API calls, not compute.