CTXD/LLM.txt

# CTXD — Context Dossier
# LLM Installation & Deployment Guide
# Feed this file to an LLM to assist with setup and deployment.

## What is CTXD?

CTXD (Context Dossier) is a single source of truth for multi-harness project context. It serves one canonical AGENTS.md per project to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP. It runs as two Docker containers: a PostgreSQL 16 database and a Python ASGI daemon.

## Prerequisites

- Docker 24+ and Docker Compose v2+
- A reverse proxy with TLS (Traefik recommended; Caddy or nginx also work)
- Linux host with ~1GB free RAM
- (Optional) Existing PostgreSQL 14+ instance if not using the bundled one

## Quick Deployment

1. Clone the repository:
   git clone ssh://git@code.cubecraftcreations.com:2288/CubeCraft-Creations/CTXD.git
   cd CTXD/app

2. Copy the environment template:
   cp .env.example .env

3. Generate secrets and fill in .env:
   python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # Use for CTXD_API_KEY
   python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # Use for OAUTH_APPROVAL_KEY

4. Edit .env with your values. Required variables:
   - DATABASE_URL — PostgreSQL connection string
   - POSTGRES_PASSWORD — Password for the bundled PG container
   - CTXD_API_KEY — Shared key for internal MCP and HTTP auth
   - OAUTH_ENABLED=true
   - OAUTH_ISSUER=https://your-domain.com
   - OAUTH_APPROVAL_KEY — Key for approving OAuth authorizations

5. Build and start:
   docker compose up -d

6. Wait for PostgreSQL to be healthy and the daemon to start:
   docker compose ps
   # Both ctxd-postgres and dossier should show "healthy" / "running"

7. Set the admin password:
   docker exec ctxd dossier user-set-password admin -p "your-admin-password"

8. Verify:
   curl http://localhost:9091/status
   # Expected: {"status": "ok", "db": "/data/ctxd.db"}

9. Access the Web UI:
   http://<server-ip>:9091/
   Sign in with admin / your-admin-password

## Environment Variables Reference

All config is driven by environment variables in .env. Precedence: env var > ctxd.yaml > defaults.

### Database
- DATABASE_URL — PostgreSQL connection string. If empty, falls back to SQLite.
- POSTGRES_USER — PostgreSQL user for bundled container (default: ctxd)
- POSTGRES_PASSWORD — PostgreSQL password for bundled container
- POSTGRES_DB — PostgreSQL database name (default: ctxd)

### Server
- CTXD_HOST — Bind address (default: 0.0.0.0)
- CTXD_PORT — Listen port (default: 9091)
- CTXD_HOME — Data directory inside container (default: /data)
- LOG_LEVEL — Uvicorn log level: debug, info, warning, error (default: info)

### Auth
- CTXD_AUTH_ENABLED — Enable authentication globally (default: false)
- CTXD_API_KEY — Shared API key for Hermes/internal MCP and HTTP auth
- CTXD_EXTERNAL_READONLY_KEY — Legacy ?key= on read-only MCP (migration only)

### OAuth
- OAUTH_ENABLED — Enable OAuth authorization server (default: false)
- OAUTH_ISSUER — Public URL used in OAuth discovery metadata
- OAUTH_APPROVAL_KEY — Fallback key for /oauth/authorize approval
- OAUTH_APPROVAL_USER_ID — User ID for attributing approvals (default: admin)
- OAUTH_ACCESS_TOKEN_TTL — Access token lifetime in seconds (default: 3600)
- OAUTH_REFRESH_TOKEN_TTL — Refresh token lifetime in seconds (default: 2592000)

### Web Sessions
- WEB_SESSION_TTL — Session cookie lifetime in seconds (default: 604800)

### Snapshots
- SNAPSHOT_MIN_KEEP — Minimum snapshots per project (default: 5)
- SNAPSHOT_MAX_KEEP — Maximum snapshots before rotation (default: 25)

## MCP Endpoints

CTXD exposes three MCP surfaces via Streamable HTTP:

1. /readonly/mcp — Read-only tools (OAuth ctxd.read or API key)
   Tools: list_projects, get_project_context, search_context, get_project_tags, list_files, get_file, get_client_guide

2. /write/mcp — Write tools (OAuth ctxd.write only)
   Tools: update_file, set_project_tags, sync_to_project, get_client_guide

3. /mcp — Internal full MCP (shared API key only, not exposed publicly)
   All read + write tools plus auto_generate_tags and get_user_profile

## Connecting LLM Clients

### Claude Desktop
Connector URL: https://your-domain.com/readonly/mcp
Claude auto-discovers OAuth metadata and registers via DCR.
For write access, use: https://your-domain.com/write/mcp

### Hermes Agent
Edit ~/.hermes/config.yaml:
  mcp_servers:
    dossier:
      url: http://<server-ip>:9091/mcp
      timeout: 30
      headers:
        Authorization: "Bearer your-api-key"
Restart Hermes after changing config.

### Other MCP Clients
1. Register an OAuth client: POST /oauth/register with your redirect URI
2. Authorize: GET /oauth/authorize (admin must approve)
3. Exchange code: POST /oauth/token
4. Connect to /readonly/mcp or /write/mcp with Bearer token

## Traefik Configuration

Route everything except the internal MCP endpoint:

  rule: Host(`ctxd.yourdomain.com`) && !Path(`/mcp`)

This exposes: landing page, dashboard, REST API, OAuth, read-only MCP, write MCP.
Only /mcp (internal, API key) is blocked.

## CLI Commands

All commands run inside the container:
  docker exec ctxd dossier <command>

Commands:
  init                                    Initialize database
  project-create <id> [--display-name N]  Create a project
  project-list                            List projects
  read <project_id>                       Print context to stdout
  edit <project_id>                       Open in $EDITOR
  file-list <project_id>                  List context files
  file-read <project_id> <file_path>      Read a single file
  sync <project_id> [path]               Sync AGENTS.md to project root
  search "query"                          Full-text search
  audit [--limit N]                       Show audit log
  user-list                               List users
  user-create <id> --display-name "Name" [--password "pw"]
  user-set-password <id> -p "password"
  oauth-client-create [-n "Name"] [--redirect-uri URI]
  oauth-client-list                       List OAuth clients
  oauth-client-revoke <client_id>         Revoke client and invalidate tokens
  import-vault <path>                     Import from Obsidian vault

## Admin UI

Sign in as admin, click "admin" in the masthead. Three tabs:
1. oauth clients — Create and revoke OAuth clients
2. users — List, create, edit, activate, inactivate, delete users
3. projects — List and delete projects (typed-name confirmation required)

## Project Files

Each project has multiple context files:
- CONTEXT.MD — Canonical overview (synced as AGENTS.md, cannot be deleted)
- DECISIONS.MD — Architecture decisions and rationale
- RUNBOOKS.MD — Deploy, troubleshoot, operate procedures
- PROMPTS.MD — Project-specific prompts for different harnesses
- GLOSSARY.MD — Project-specific terms and acronyms

The compiled view (get_project_context) concatenates all files with metadata header.

## Locked Files

- CONTEXT.MD — Cannot be deleted from any project (minimum required file)
- CONTEXT.MD and LLM-CLIENT.MD — Fully locked in the ctxd-docs project (cannot update or delete)

## Backups

PostgreSQL backup:
  docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup.sql

PostgreSQL restore:
  cat backup.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd

Snapshots are automatic (before each context update, rotated min 5 / max 25 per project).

## Migrating from SQLite to PostgreSQL

If you started with SQLite:
1. Start PostgreSQL: docker compose up -d postgres
2. Run migration: docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
3. Set DATABASE_URL in .env and restart: docker compose up -d dossier

## Using an External PostgreSQL

1. Create database and user on your external PG instance
2. Set DATABASE_URL in .env to point to it
3. Start only the app: docker compose up -d dossier
   (or scale postgres to 0: docker compose up -d --scale postgres=0)

## Troubleshooting

Login fails: Reset password with "docker exec ctxd dossier user-set-password admin -p newpass"
  Use double quotes if password contains single quotes. Use single quotes for $ ` ! characters.

MCP 404: Traefik is not routing /readonly/mcp. Check router rule includes it.
MCP 401: Auth is working. Check OAuth token scope and expiry.

PostgreSQL connection fails: Check DATABASE_URL password matches POSTGRES_PASSWORD.
  If PG data volume was initialized with a different password, reset it:
  docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'newpass'"
  Then update .env.

Container keeps restarting: Check "docker logs ctxd --tail 30".
  Common causes: wrong PG password, OAUTH_ENABLED=true but OAUTH_ISSUER empty, missing CTXD_API_KEY.

## File Structure

  CTXD/
  ├── .env                    Production environment (gitignored, contains secrets)
  ├── .env.example            Template with all documented variables
  ├── .gitignore
  ├── README.md               Full documentation
  ├── SKILL.md                LLM client guide (canonical source for LLM-CLIENT.MD)
  ├── data/                   Runtime data (gitignored)
  │   ├── ctxd.yaml           Fallback config (env vars take precedence)
  │   ├── oauth_state.json    OAuth clients, codes, tokens
  │   ├── web_sessions.json   Web UI sessions
  │   ├── snapshots/          Point-in-time context backups
  │   └── pg/                 PostgreSQL data volume
  └── app/                    Application source
      ├── docker-compose.yml
      ├── Dockerfile
      ├── pyproject.toml
      └── src/ctxd/
          ├── config.py        Env-driven config
          ├── db.py            Database layer (PostgreSQL + SQLite)
          ├── schema.sql       PostgreSQL schema
          ├── schema_sqlite.sql SQLite schema fallback
          ├── server.py        ASGI: HTTP + MCP + OAuth
          ├── cli.py           CLI commands
          ├── ui.html          Web UI dashboard
          ├── landing.html     Public landing page
          ├── auth_password.py PBKDF2 password hashing
          └── migrate_sqlite_to_pg.py  Migration script