/ Self-Hosting
GitHub

Self-Hosting

Deployment·Docker · Production

CrewLayer is designed to self-host on any machine that runs Docker. This page covers local setup, environment configuration, and a production-ready setup with nginx and SSL.

Quick Setup (Local / Dev)

terminal
git clone https://github.com/GerardSole/CrewLayer
cd CrewLayer
cp .env.example .env    # then edit .env
docker compose up -d

# Verify
curl http://localhost:8000/health
# → {"status": "ok", "db": "ok", "redis": "ok"}

Services started by docker compose up -d:

The API runs database migrations via Alembic automatically on startup.

Environment Variables

VariableRequiredDescription
ANTHROPIC_API_KEY required Anthropic API key. Used for memory extraction (claude-opus-4-8) and embeddings.
DATABASE_URL required PostgreSQL async URL. Format: postgresql+asyncpg://user:pass@host/db. In Docker Compose this is pre-configured.
REDIS_URL required Redis connection URL. Default: redis://localhost:6379.
SECRET_KEY required Random secret for signing JWTs (dashboard auth). Generate with: python -c "import secrets; print(secrets.token_hex(32))"
EMBEDDING_PROVIDER optional anthropic (default) or local. Use local to run sentence-transformers locally without API cost.
METRICS_TOKEN optional Bearer token required to access /metrics from non-localhost IPs. Leave unset to block external access.
MCP_PORT optional Port for the MCP server in SSE mode. Default: 8001.
MCP_TRANSPORT optional stdio (default, for Claude Desktop/Code) or sse (for Docker/HTTP).
CREWLAYER_API_KEY optional API key the MCP server uses when calling the backend. Required only when running the MCP server.
MEMORY_DECAY_ENABLED optional true (default) or false. Controls whether the nightly memory decay job runs.

Observability Stack

CrewLayer ships a separate Docker Compose file for Prometheus and Grafana with a prebuilt dashboard:

terminal
docker compose -f docker-compose.observability.yml up -d

# Prometheus → http://localhost:9090
# Grafana    → http://localhost:3000 (admin/admin by default)

The Grafana dashboard at observability/grafana/dashboards/crewlayer.json shows active agents, memory counts, action rates, error rates, and Redis usage.

Production Setup

nginx + SSL

Place nginx in front of the API to handle SSL termination and serve the dashboard at a clean URL:

/etc/nginx/sites-available/crewlayer
server {
    listen 443 ssl http2;
    server_name api.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

    # Increase buffer for SSE (context subscribe, streaming)
    proxy_buffering off;
    proxy_read_timeout 3600s;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Redirect HTTP → HTTPS
server {
    listen 80;
    server_name api.yourdomain.com;
    return 301 https://$host$request_uri;
}
⚠️

Set proxy_buffering off — it's required for SSE endpoints (/v1/context/.../subscribe, /v1/streaming) to work correctly through nginx.

PostgreSQL Backups

terminal
# Manual dump
docker compose exec postgres pg_dump -U crewlayer crewlayer > backup_$(date +%Y%m%d).sql

# Restore
cat backup_20250614.sql | docker compose exec -T postgres psql -U crewlayer crewlayer

# Automated daily backup with cron (add to crontab -e)
0 2 * * * cd /path/to/CrewLayer && docker compose exec postgres pg_dump -U crewlayer crewlayer | gzip > /backups/crewlayer_$(date +\%Y\%m\%d).sql.gz

Recommended Resources

TierCPURAMDiskUse case
Dev2 vCPU4 GB20 GBLocal dev, testing
Small2 vCPU8 GB50 GB1–5 agents, low traffic
Medium4 vCPU16 GB100 GB SSD10–50 agents, moderate load
Large8 vCPU32 GB500 GB SSD100+ agents, high traffic

Upgrading to a New Version

terminal
# 1. Pull latest changes
git pull origin main

# 2. Rebuild the API image
docker compose build api

# 3. Run migrations (Alembic runs on startup automatically, but you can run manually)
docker compose run --rm api alembic upgrade head

# 4. Restart
docker compose up -d

# 5. Verify
curl http://localhost:8000/health
💡

Always take a PostgreSQL backup before upgrading, especially when there are new Alembic migrations.

Running Migrations Manually

terminal
# Apply all pending migrations
docker compose run --rm api alembic upgrade head

# Check current migration version
docker compose run --rm api alembic current

# Rollback one migration
docker compose run --rm api alembic downgrade -1

# Create a new migration after changing db/models.py
docker compose run --rm api alembic revision --autogenerate -m "add new table"