CTXD — Context Dossier

A single source of truth for multi-harness project context. One canonical AGENTS.md per project, served to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP.

Overview

CTXD solves context sprawl: when you work across multiple LLM harnesses (Claude Desktop, Claude Code, Codex CLI, Cursor, Copilot, Hermes), each has its own context file convention (CLAUDE.md, .cursorrules, CODEX.md, etc.). Without a canonical source, these drift independently.

CTXD provides:

• Multi-file context per project — CONTEXT.MD, DECISIONS.MD, RUNBOOKS.MD, PROMPTS.MD, GLOSSARY.MD
• Compiled view — all files concatenated with metadata header, served as a single document to agents
• Sync to repos — writes CONTEXT.MD as AGENTS.md + symlinks (CLAUDE.md, .cursorrules, CODEX.md → AGENTS.md)
• Version-checked writes — optimistic concurrency with base_version to prevent silent overwrites
• OAuth 2.0 authorization server — DCR, Authorization Code + PKCE, ctxd.read and ctxd.write scopes
• Streamable HTTP MCP — single-endpoint transport for read-only and write surfaces
• Web UI — per-user password login, admin panel, project/file management
• PostgreSQL backend — with SQLite fallback for local dev
• Append-only audit log — every read, write, create, delete, sync, and search is logged
• Point-in-time snapshots — automatic version snapshots with rotation (min 5, max 25 per project)
• Full-text search — PostgreSQL tsvector with GIN index (FTS5 on SQLite fallback)

Architecture

Context Dossier (container: ctxd, 0.0.0.0:9091)
├── PostgreSQL 16 (container: ctxd-postgres)     # Primary DB
├── /data                                        # Config, OAuth state, web sessions
│   ├── ctxd.yaml                                # Fallback config (env vars take precedence)
│   ├── oauth_state.json                         # OAuth clients, codes, tokens
│   ├── web_sessions.json                        # Per-user web UI sessions
│   └── snapshots/                               # Point-in-time context backups
├── Streamable HTTP MCP:
│   ├── /mcp             (OAuth ctxd.read + ctxd.write; API key on LAN = full tools)
│   ├── /readonly/mcp    (alias → same OAuth behavior)
│   └── /oauth/mcp       (alias)
├── OAuth Authorization Server:
│   ├── /.well-known/oauth-authorization-server  # Discovery
│   ├── /.well-known/oauth-protected-resource     # Resource metadata
│   ├── /oauth/register                           # Dynamic Client Registration
│   ├── /oauth/authorize                          # Authorization + PKCE
│   └── /oauth/token                              # Token + refresh
├── Web UI + REST API (/)                        # Dashboard, admin, projects, files
└── Landing page (public host only)               # Themed login form

Quick Start

Prerequisites

• Docker and Docker Compose
• A reverse proxy with TLS (Traefik, Caddy, nginx) for public exposure
• (Optional) An existing PostgreSQL 14+ instance if not using the bundled one

1. Clone and configure

cd /mnt/ai-storage/projects/ctxd/app
cp .env.example .env

Edit .env with your values:

# Database
DATABASE_URL=postgresql://ctxd:your-password@postgres:5432/ctxd
POSTGRES_USER=ctxd
POSTGRES_PASSWORD=your-strong-password

# Server
CTXD_HOST=0.0.0.0
CTXD_PORT=9091
CTXD_HOME=/data

# Auth
CTXD_AUTH_ENABLED=true
CTXD_API_KEY=your-api-key-here          # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"

# OAuth
OAUTH_ENABLED=true
OAUTH_ISSUER=https://ctxd.yourdomain.com
OAUTH_APPROVAL_KEY=your-approval-key     # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"

2. Build and start

# Recommended: build + postgres + recreate ctxd (avoids 502 when PG was never started)
chmod +x scripts/deploy.sh
./scripts/deploy.sh

# Or manual (always include postgres when DATABASE_URL uses host "postgres"):
docker compose up -d

This starts:

• ctxd-postgres — PostgreSQL 16 (Alpine)
• ctxd — CTXD daemon (web UI + MCP + OAuth + REST API)

After code changes, use ./scripts/deploy.sh or docker compose up -d — not docker restart ctxd alone (old image) and not docker compose up -d --no-deps ctxd (skips postgres → crash loop → public 502).

3. Verify

# Health check
curl http://localhost:9091/status
# → {"status": "ok", "db": "/data/ctxd.db"}

# List projects (requires API key)
curl http://localhost:9091/projects -H "Authorization: Bearer your-api-key"

4. Set admin password

docker exec ctxd dossier user-set-password admin -p "your-admin-password"

Shell quoting: Use double quotes (") if the password contains single quotes ('). Use single quotes (') if it contains $, backticks, or !. The failure mode is shell expansion, not PBKDF2.

5. Access the Web UI

• LAN: http://<server-ip>:9091/
• Public (via Traefik): https://ctxd.yourdomain.com/

Sign in with admin / your password.

Configuration

Environment Variables

All config is driven by environment variables. A ctxd.yaml file in /data can override built-in defaults, but env vars always take precedence.

Precedence: env var > ctxd.yaml > built-in default

Variable	Default	Description
Database
DATABASE_URL	(empty)	PostgreSQL connection string. If empty, falls back to SQLite at $CTXD_HOME/ctxd.db
POSTGRES_USER	ctxd	PostgreSQL user (for bundled PG container)
POSTGRES_PASSWORD	ctxd_local_dev	PostgreSQL password (for bundled PG container)
POSTGRES_DB	ctxd	PostgreSQL database name (for bundled PG container)
Server
CTXD_HOST	0.0.0.0	Bind address
CTXD_PORT	9091	Listen port
CTXD_HOME	~/.ctx	Data directory (inside container: /data)
LOG_LEVEL	info	Uvicorn log level (debug, info, warning, error)
Auth
CTXD_AUTH_ENABLED	false	Enable authentication globally
CTXD_API_KEY	(empty)	Shared API key for Hermes/internal MCP + HTTP auth
CTXD_EXTERNAL_READONLY_KEY	(empty)	Legacy ?key= on read-only MCP (migration only)
OAuth
OAUTH_ENABLED	false	Enable OAuth authorization server
OAUTH_ISSUER	(empty)	Public URL (used in OAuth discovery metadata)
OAUTH_APPROVAL_KEY	(empty)	Fallback approval key for /oauth/authorize
OAUTH_APPROVAL_USER_ID	admin	Which user ID to attribute OAuth approvals to
OAUTH_ACCESS_TOKEN_TTL	3600	Access token lifetime in seconds
OAUTH_REFRESH_TOKEN_TTL	2592000	Refresh token lifetime in seconds (30 days)
PostgreSQL (container)
CTXD_PG_WAIT_SECONDS	120	Entrypoint: max wait for DB before exit (when DATABASE_URL set)
CTXD_PG_WAIT_INTERVAL	2	Seconds between connection attempts
Web Sessions
WEB_SESSION_TTL	604800	Session cookie lifetime in seconds (7 days)
Snapshots
SNAPSHOT_MIN_KEEP	5	Minimum snapshots retained per project
SNAPSHOT_MAX_KEEP	25	Maximum snapshots before rotation

Using an External PostgreSQL

To use an external PostgreSQL instead of the bundled container:

1. Create a database and user on your external PG instance
2. Set DATABASE_URL in .env to point to it
3. Start only the app (no bundled postgres): docker compose up -d --scale postgres=0 ctxd

Ensure DATABASE_URL points at your external host (not postgres). The entrypoint skips the compose-network wait when the URL is reachable.

Fallback to SQLite

If DATABASE_URL is empty or not set, CTXD falls back to SQLite at $CTXD_HOME/ctxd.db. This is useful for local development or single-user deployments that don't need PostgreSQL features.

MCP Surfaces

CTXD exposes MCP via Streamable HTTP on /mcp (single public connector):

Endpoint	Auth	Scope	Tools
/mcp	OAuth bearer	ctxd.read / ctxd.write	Scope-gated read + write tools
/mcp	Shared API key (LAN/Hermes)	(full)	All tools including get_user_profile, auto_generate_tags
/readonly/mcp, /oauth/mcp	OAuth (aliases)	same as /mcp	Backward-compatible URLs

Connecting an LLM Client

Claude Desktop / Claude Web:

Connector URL: https://ctxd.yourdomain.com/mcp

Claude auto-discovers OAuth metadata and registers via DCR. Request scope=ctxd.read ctxd.write for write access.

ChatGPT (MCP connector — recommended; no Custom GPT required):

ChatGPT can attach a remote MCP server in Developer mode (Plus/Pro and higher tiers for custom connectors). Use the same public URL as Claude — OAuth on /mcp, not REST Actions or a shared API key.

1. Server — deploy current CTXD and expose the public host (Traefik must route /mcp and /oauth/*; do not block /mcp):

   cd app
   ./scripts/deploy.sh
   

   Smoke: curl -sS https://ctxd.yourdomain.com/.well-known/oauth-authorization-server | head

2. ChatGPT — Settings → Connectors → enable Developer mode → Add connector (MCP / custom remote) → Server URL:

   https://ctxd.yourdomain.com/mcp
   

   Start OAuth and copy ChatGPT’s callback / redirect URL exactly.

3. CTXD admin — Web UI → admin → oauth clients:

   • After DCR, the client may appear automatically; otherwise create client with ChatGPT’s redirect URI.
   • Set allowed scopes to ctxd.read and ctxd.write (create form or scopes → save on an existing row).

4. Authorize — complete the browser approval (sign in as CTXD admin in that browser, or use the OAuth approval key). Tokens are capped by the client’s allowed scopes.

5. Chat — enable the CTXD connector for the conversation. Tools include list_projects, get_project_context, search_context, get_file, update_file, etc. Call get_client_guide first in a new session.

Optional CLI pre-register (if ChatGPT asks for client credentials before DCR):

docker exec ctxd dossier oauth-client-create -n "ChatGPT MCP" \
  --redirect-uri 'PASTE_CHATGPT_CALLBACK_URL' \
  --scope "ctxd.read ctxd.write"

If connect fails	Check
“does not implement OAuth”	Public host must return 200 JSON (not 502/404) for /.well-known/oauth-protected-resource/mcp and /.well-known/oauth-authorization-server. Traefik must proxy all paths to CTXD :9091. Set OAUTH_ENABLED=true and OAUTH_ISSUER=https://ctxd.cubecraftlabs.com.
502 on well-known URLs	Backend down or wrong Traefik service — fix routing before OAuth can work
404 on /mcp	Router not forwarding /mcp to CTXD (often a stale !Path(/mcp) rule)
401 on /mcp	Re-authorize; access token may have expired
Redirect mismatch	Callback URL in admin must match ChatGPT’s string exactly
invalid_request on authorize	CTXD requires PKCE S256 (code_challenge + code_challenge_method=S256)
Stale routes / old code	./scripts/deploy.sh or docker compose up -d --force-recreate ctxd after postgres is healthy (not restart alone; never --no-deps unless PG is already up)
No write tools	Client scopes include ctxd.write; re-authorize after changing scopes

Not required for ChatGPT MCP: Custom GPT, OpenAPI Actions, or CTXD_API_KEY in ChatGPT (public path is OAuth). Hermes continues to use LAN http://<server-ip>:9091/mcp with the API key.

Hermes Agent:

# ~/.hermes/config.yaml
mcp_servers:
  dossier:
    url: http://<server-ip>:9091/mcp
    timeout: 30
    headers:
      Authorization: "Bearer your-api-key"

Other MCP clients (Codex, Cursor, etc.):

• Register an OAuth client via POST /oauth/register with your redirect URI
• Connect to /mcp with scope=ctxd.read ctxd.write
• Use the access token as Authorization: Bearer <token>

MCP Tool Reference

Read tools (require ctxd.read on /mcp)

Tool	Args	Returns
get_client_guide	(none)	Locked LLM-CLIENT.MD guide — call this first
list_projects	(none)	All projects with version numbers
get_project_context	project_id	Compiled markdown of all context files
search_context	query, limit?	FTS results across all projects
get_project_tags	project_id	Metadata tags for a project
list_files	project_id	All context files in a project
get_file	project_id, file_path	Single file with metadata header

Write tools (require ctxd.write on /mcp)

Tool	Args	Returns
update_file	project_id, file_path, content, base_version	{"ok": true, "new_version": N} or conflict error
set_project_tags	project_id, tags[]	{"ok": true, "tags": [...]}
sync_to_project	project_id	Writes CONTEXT.MD as AGENTS.md + symlinks to project root

Locked Files

File	Scope	Protection
CONTEXT.MD	All projects	Cannot delete — minimum required file
CONTEXT.MD	ctxd-docs project only	Cannot update or delete
LLM-CLIENT.MD	ctxd-docs project only	Cannot update or delete

OAuth

Scopes

Scope	Grants
ctxd.read	Read-only MCP tools
ctxd.write	Write MCP tools (includes read)

Request both: scope=ctxd.read ctxd.write

Redirect URIs

Platform	Redirect URI
Claude Desktop	https://claude.ai/api/mcp/auth_callback
ChatGPT (MCP)	Paste callback URL from ChatGPT connector OAuth UI (per connector)
Claude Code	http://localhost:5555/oauth/callback
Codex CLI	http://localhost:7777/oauth/callback
Custom	Your app's documented OAuth callback

Managing OAuth Clients

Via Admin UI: http://<server-ip>:9091/ → sign in as admin → admin → oauth clients tab

Via CLI:

# Create
docker exec ctxd dossier oauth-client-create -n "Claude Desktop" --redirect-uri https://claude.ai/api/mcp/auth_callback

# List
docker exec ctxd dossier oauth-client-list

# Revoke (invalidates all tokens for that client)
docker exec ctxd dossier oauth-client-revoke ctxd_xxxxxxxx

Via API (admin session or API key):

# Create
curl -X POST http://localhost:9091/oauth/clients \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"client_name": "Claude Desktop", "redirect_uris": ["https://claude.ai/api/mcp/auth_callback"]}'

# List
curl http://localhost:9091/oauth/clients -H "Authorization: Bearer your-api-key"

# Revoke
curl -X DELETE http://localhost:9091/oauth/clients/ctxd_xxxxxxxx -H "Authorization: Bearer your-api-key"

Public Exposure (Traefik)

Router Rule

Route the public host to the backend (include /mcp — OAuth protects it):

rule: Host(`ctxd.yourdomain.com`)

This exposes:

• Landing page (GET /)
• Login (POST /auth/login, GET /auth/me)
• Full Web UI dashboard (all REST API endpoints)
• OAuth (/oauth/*, /.well-known/*)
• Public MCP (/mcp — OAuth read + write)

Hermes uses the same /mcp path on LAN with the shared API key (not exposed via public OAuth).

Landing Page Behavior

• Not signed in → themed landing page with login form
• Signed in (valid session cookie) → full dashboard
• Cookie: ctxd_session (SameSite=Lax, 7-day expiry)
• After login: cookie set + redirect to dashboard

A full Traefik template is at app/templates/traefik-ctxd-readonly-mcp.yaml in the project-context-management Hermes skill.

Web UI

Admin Panel

Sign in as admin → click admin in the masthead.

Tabs:

1. oauth clients — client list (revoke per row) + create client form
2. users — user list (id, name, role, active/inactive) + manage users (create, edit, activate, inactivate, delete)
3. projects — manage projects (list with remove button, typed-name confirmation for delete)

Project Files

Each project has multiple context files:

File	Purpose
CONTEXT.MD	Canonical project overview (synced as AGENTS.md to repos)
DECISIONS.MD	Architecture decisions, rationale
RUNBOOKS.MD	Deploy, troubleshoot, operate procedures
PROMPTS.MD	Project-specific prompts for different harnesses
GLOSSARY.MD	Project-specific terms, acronyms

The compiled view (get_project_context) concatenates all files with ## FILENAME headers and a single metadata block at the top.

CLI

All commands run inside the container:

docker exec ctxd dossier <command>

Commands

# Initialize (auto-runs on first container start)
dossier init

# Projects
dossier project-create <project_id> [--display-name "Name"] [--description "Desc"]
dossier project-list
dossier read <project_id>                          # Print context to stdout
dossier edit <project_id>                          # Open in $EDITOR

# Context files
dossier file-list <project_id>
dossier file-read <project_id> <file_path>

# Sync
dossier sync <project_id> [path]                   # Set sync path and/or sync AGENTS.md

# Search
dossier search "query"                             # FTS across all projects

# Audit
dossier audit [--limit N]

# Users
dossier user-list
dossier user-create <user_id> --display-name "Name" [--password "pw"]
dossier user-set-password <user_id> -p "password"

# OAuth
dossier oauth-client-create [-n "Name"] [--redirect-uri URI]
dossier oauth-client-list
dossier oauth-client-revoke <client_id>

# Import
dossier import-vault <path>                        # Import from Obsidian vault

REST API

All endpoints require Authorization: Bearer <api_key> or Authorization: Bearer <session_token> unless noted.

Method	Path	Description
GET	/	Web UI (LAN) or landing page (public host)
GET	/status	Health check (no auth)
POST	/auth/login	Web UI login → session token + cookie
POST	/auth/logout	Revoke session + clear cookie
GET	/auth/me	Current session identity
GET	/users	List users
POST	/users	Create user
PATCH	/users/<id>	Update user (admin)
DELETE	/users/<id>	Delete user (admin)
POST	/users/<id>/password	Set password (admin)
GET	/oauth/clients	List OAuth clients (admin)
POST	/oauth/clients	Register OAuth client (admin)
DELETE	/oauth/clients/<id>	Revoke OAuth client (admin)
GET	/projects	List all projects
POST	/projects	Create a project
DELETE	/projects/<id>	Delete a project (admin)
GET	/projects/<id>/context	Compiled context (all files)
POST	/projects/<id>/context	Update context (legacy single-file)
GET	/projects/<id>/files	List context files
GET	/projects/<id>/files/<name>	Read a single file
POST	/projects/<id>/files	Create a new file
PUT	/projects/<id>/files/<name>	Update a file (version-checked)
DELETE	/projects/<id>/files/<name>	Delete a file
POST	/projects/<id>/migrate-files	Migrate single-context to multi-file
GET	/projects/<id>/snapshots	List snapshots
GET	/projects/<id>/tags	Get metadata tags
POST	/projects/<id>/tags	Set metadata tags
POST	/projects/<id>/sync	Sync CONTEXT.MD as AGENTS.md
POST	/projects/<id>/import	Import raw text as context
GET	/search?q=...	Full-text search
GET	/audit?limit=N	Audit log
GET	/.well-known/oauth-authorization-server	OAuth discovery
GET	/.well-known/oauth-protected-resource	Resource metadata
POST	/oauth/register	Dynamic Client Registration
GET/POST	/oauth/authorize	Authorization endpoint
POST	/oauth/token	Token endpoint
POST/GET/DELETE	/readonly/mcp	Read-only MCP (Streamable HTTP)
POST/GET/DELETE	/write/mcp	Write MCP (Streamable HTTP)
POST/GET/DELETE	/mcp	Internal full MCP (API key only)

Backups

PostgreSQL

# Backup
docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup_$(date +%Y%m%d).sql

# Restore
cat backup_YYYYMMDD.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd

Snapshots

CTXD automatically takes point-in-time snapshots before each context update. Snapshots are stored as files in /data/snapshots/<project_id>/ and rotated (min 5, max 25 per project).

Migrating from SQLite to PostgreSQL

If you started with SQLite and want to move to PostgreSQL:

# 1. Start PostgreSQL
docker compose up -d postgres

# 2. Run the migration script (reads from /data/ctxd.db, writes to DATABASE_URL)
docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg

# 3. Set DATABASE_URL in .env and restart
docker compose up -d dossier

The migration handles all tables, rebuilds the FTS index, and skips orphaned rows with FK violations.

Project Structure

/mnt/ai-storage/projects/ctxd/
├── .env                              # Production environment (gitignored)
├── .env.example                      # Template (committed)
├── .gitignore
├── SKILL.md                         # LLM client guide (canonical source)
├── README.md                         # This file
├── data/                             # Runtime data (gitignored)
│   ├── ctxd.yaml                     # Fallback config (env vars take precedence)
│   ├── ctxd.db                       # SQLite DB (if no DATABASE_URL)
│   ├── pg/                           # PostgreSQL data volume
│   ├── oauth_state.json              # OAuth clients, codes, tokens
│   ├── web_sessions.json             # Web UI sessions
│   └── snapshots/                    # Point-in-time backups
└── app/                             # Application source
    ├── docker-compose.yml
    ├── Dockerfile
    ├── pyproject.toml
    ├── .env                          # Same as root .env (symlinked or copied)
    └── src/ctxd/
        ├── __init__.py
        ├── __main__.py               # CLI/daemon entry point
        ├── config.py                 # Env-driven config with yaml fallback
        ├── db.py                     # Database layer (PostgreSQL + SQLite)
        ├── schema.sql                # PostgreSQL schema
        ├── schema_sqlite.sql         # SQLite schema (fallback)
        ├── server.py                 # ASGI app: HTTP + MCP + OAuth
        ├── cli.py                    # CLI commands
        ├── ui.html                   # Web UI dashboard
        ├── landing.html              # Public landing page
        ├── auth_password.py          # PBKDF2 password hashing
        └── migrate_sqlite_to_pg.py   # One-time migration script

Development

Local Development (SQLite, no Docker)

cd app
pip install -e ".[mcp]"
export CTXD_HOME=./dev-data
python -m ctxd init
python -m ctxd
# → http://localhost:9091

Rebuilding After Code Changes

cd app
docker compose build
docker compose up -d --no-build
# Verify:
curl http://localhost:9091/status
docker logs ctxd --tail 20

Key Conventions

• Metadata headers are dynamically generated on read, never stored in the DB
• File paths are normalized to uppercase with .MD extension
• CONTEXT.MD is the minimum required file — cannot be deleted from any project
• Version checking uses base_version parameter — mismatches return 409 conflict
• Audit log is append-only at the application layer (every operation is logged)
• OAuth state (oauth_state.json) and web sessions (web_sessions.json) are file-based, not in PostgreSQL

Troubleshooting

Login fails with "invalid credentials"

# Reset admin password
docker exec ctxd dossier user-set-password admin -p "new-password"

If the password contains special characters, use the quoting that matches:

• ' in password → use double quotes (")
• $, `, ! in password → use single quotes (')

Login works on LAN but not on public host

The public host (https://ctxd.yourdomain.com) requires Traefik to route /auth/login and /auth/me. Check your Traefik router rule includes all paths (use !Path(/mcp) to block only the internal MCP).

MCP connection fails

1. Check OAuth discovery: curl https://ctxd.yourdomain.com/.well-known/oauth-authorization-server
2. Check MCP endpoint: curl -o /dev/null -w '%{http_code}' https://ctxd.yourdomain.com/readonly/mcp → should be 401 (not 404)
3. If 404: Traefik isn't routing /readonly/mcp — update the router rule
4. If 401: auth is working — check OAuth token scope and expiry

PostgreSQL connection fails

# Check PG is running
docker compose ps postgres

# Check connection
docker exec ctxd python3 -c "
import os
import psycopg
conn = psycopg.connect(os.environ['DATABASE_URL'])
print('Connected:', conn.info.server_version)
"

# If password mismatch (PG data volume initialized with different password):
docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'new-password'"
# Then update .env with the new password

Container keeps restarting / public site 502

docker compose ps -a
docker logs ctxd --tail 40

Symptom	Cause	Fix
failed to resolve host 'postgres'	ctxd-postgres not running (often after --no-deps ctxd)	cd app && docker compose up -d postgres ctxd or ./scripts/deploy.sh
Restarting (1) on ctxd only	Same — app up without DB on compose network	Start postgres first; wait for (healthy)
Cloudflare 502 on public URL	Traefik/backend has no healthy upstream on :9091	Fix local curl http://127.0.0.1:9091/status first

The container entrypoint waits up to 120s for PostgreSQL when DATABASE_URL is set (CTXD_PG_WAIT_SECONDS to override). If postgres never appears, logs print an explicit message instead of an immediate opaque crash.

Common other causes:

• DATABASE_URL password doesn't match what PG was initialized with
• OAUTH_ENABLED=true but OAUTH_ISSUER is empty
• Missing CTXD_API_KEY when CTXD_AUTH_ENABLED=true

License

MIT