2026-06-24 22:50:54 +00:00
# CTXD — Context Dossier
A single source of truth for multi-harness project context. One canonical `AGENTS.md` per project, served to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP.
## Overview
CTXD solves context sprawl: when you work across multiple LLM harnesses (Claude Desktop, Claude Code, Codex CLI, Cursor, Copilot, Hermes), each has its own context file convention (`CLAUDE.md` , `.cursorrules` , `CODEX.md` , etc.). Without a canonical source, these drift independently.
CTXD provides:
- **Multi-file context per project** — `CONTEXT.MD` , `DECISIONS.MD` , `RUNBOOKS.MD` , `PROMPTS.MD` , `GLOSSARY.MD`
- **Compiled view** — all files concatenated with metadata header, served as a single document to agents
- **Sync to repos** — writes `CONTEXT.MD` as `AGENTS.md` + symlinks (`CLAUDE.md` , `.cursorrules` , `CODEX.md` → `AGENTS.md` )
- **Version-checked writes** — optimistic concurrency with `base_version` to prevent silent overwrites
- **OAuth 2.0 authorization server** — DCR, Authorization Code + PKCE, `ctxd.read` and `ctxd.write` scopes
- **Streamable HTTP MCP** — single-endpoint transport for read-only and write surfaces
- **Web UI** — per-user password login, admin panel, project/file management
- **PostgreSQL backend** — with SQLite fallback for local dev
- **Append-only audit log** — every read, write, create, delete, sync, and search is logged
- **Point-in-time snapshots** — automatic version snapshots with rotation (min 5, max 25 per project)
- **Full-text search** — PostgreSQL `tsvector` with GIN index (FTS5 on SQLite fallback)
## Architecture
```
2026-06-24 23:19:08 +00:00
Context Dossier (container: ctxd, 0.0.0.0:9091)
2026-06-24 22:50:54 +00:00
├── PostgreSQL 16 (container: ctxd-postgres) # Primary DB
├── /data # Config, OAuth state, web sessions
│ ├── ctxd.yaml # Fallback config (env vars take precedence)
│ ├── oauth_state.json # OAuth clients, codes, tokens
│ ├── web_sessions.json # Per-user web UI sessions
│ └── snapshots/ # Point-in-time context backups
2026-06-25 12:45:59 +00:00
├── Streamable HTTP MCP:
│ ├── /mcp (OAuth ctxd.read + ctxd.write; API key on LAN = full tools)
│ ├── /readonly/mcp (alias → same OAuth behavior)
│ └── /oauth/mcp (alias)
2026-06-24 22:50:54 +00:00
├── OAuth Authorization Server:
│ ├── /.well-known/oauth-authorization-server # Discovery
│ ├── /.well-known/oauth-protected-resource # Resource metadata
│ ├── /oauth/register # Dynamic Client Registration
│ ├── /oauth/authorize # Authorization + PKCE
│ └── /oauth/token # Token + refresh
├── Web UI + REST API (/) # Dashboard, admin, projects, files
└── Landing page (public host only) # Themed login form
```
## Quick Start
### Prerequisites
- Docker and Docker Compose
- A reverse proxy with TLS (Traefik, Caddy, nginx) for public exposure
- (Optional) An existing PostgreSQL 14+ instance if not using the bundled one
### 1. Clone and configure
``` bash
cd /mnt/ai-storage/projects/ctxd/app
cp .env.example .env
```
Edit `.env` with your values:
``` bash
# Database
DATABASE_URL = postgresql://ctxd:your-password@postgres:5432/ctxd
POSTGRES_USER = ctxd
POSTGRES_PASSWORD = your-strong-password
# Server
CTXD_HOST = 0.0.0.0
CTXD_PORT = 9091
CTXD_HOME = /data
# Auth
CTXD_AUTH_ENABLED = true
CTXD_API_KEY = your-api-key-here # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
# OAuth
OAUTH_ENABLED = true
OAUTH_ISSUER = https://ctxd.yourdomain.com
OAUTH_APPROVAL_KEY = your-approval-key # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
```
### 2. Build and start
``` bash
2026-06-25 13:59:51 +00:00
# Recommended: build + postgres + recreate ctxd (avoids 502 when PG was never started)
chmod +x scripts/deploy.sh
./scripts/deploy.sh
# Or manual (always include postgres when DATABASE_URL uses host "postgres"):
2026-06-24 22:50:54 +00:00
docker compose up -d
```
This starts:
- `ctxd-postgres` — PostgreSQL 16 (Alpine)
2026-06-25 13:59:51 +00:00
- `ctxd` — CTXD daemon (web UI + MCP + OAuth + REST API)
**After code changes ** , use `./scripts/deploy.sh` or `docker compose up -d` — not `docker restart ctxd` alone (old image) and **not ** `docker compose up -d --no-deps ctxd` (skips postgres → crash loop → public **502 ** ).
2026-06-24 22:50:54 +00:00
### 3. Verify
``` bash
# Health check
curl http://localhost:9091/status
# → {"status": "ok", "db": "/data/ctxd.db"}
# List projects (requires API key)
curl http://localhost:9091/projects -H "Authorization: Bearer your-api-key"
```
### 4. Set admin password
``` bash
2026-06-24 23:19:08 +00:00
docker exec ctxd dossier user-set-password admin -p "your-admin-password"
2026-06-24 22:50:54 +00:00
```
> **Shell quoting:** Use double quotes (`"`) if the password contains single quotes (`'`). Use single quotes (`'`) if it contains `$`, backticks, or `!`. The failure mode is shell expansion, not PBKDF2.
### 5. Access the Web UI
- **LAN:** `http://<server-ip>:9091/`
- **Public (via Traefik):** `https://ctxd.yourdomain.com/`
Sign in with `admin` / your password.
## Configuration
### Environment Variables
All config is driven by environment variables. A `ctxd.yaml` file in `/data` can override built-in defaults, but env vars always take precedence.
**Precedence: ** env var > `ctxd.yaml` > built-in default
| Variable | Default | Description |
|----------|---------|-------------|
| **Database ** | | |
| `DATABASE_URL` | * (empty) * | PostgreSQL connection string. If empty, falls back to SQLite at `$CTXD_HOME/ctxd.db` |
| `POSTGRES_USER` | `ctxd` | PostgreSQL user (for bundled PG container) |
| `POSTGRES_PASSWORD` | `ctxd_local_dev` | PostgreSQL password (for bundled PG container) |
| `POSTGRES_DB` | `ctxd` | PostgreSQL database name (for bundled PG container) |
| **Server ** | | |
| `CTXD_HOST` | `0.0.0.0` | Bind address |
| `CTXD_PORT` | `9091` | Listen port |
| `CTXD_HOME` | `~/.ctx` | Data directory (inside container: `/data` ) |
| `LOG_LEVEL` | `info` | Uvicorn log level (`debug` , `info` , `warning` , `error` ) |
| **Auth ** | | |
| `CTXD_AUTH_ENABLED` | `false` | Enable authentication globally |
| `CTXD_API_KEY` | * (empty) * | Shared API key for Hermes/internal MCP + HTTP auth |
| `CTXD_EXTERNAL_READONLY_KEY` | * (empty) * | Legacy `?key=` on read-only MCP (migration only) |
| **OAuth ** | | |
| `OAUTH_ENABLED` | `false` | Enable OAuth authorization server |
| `OAUTH_ISSUER` | * (empty) * | Public URL (used in OAuth discovery metadata) |
| `OAUTH_APPROVAL_KEY` | * (empty) * | Fallback approval key for `/oauth/authorize` |
| `OAUTH_APPROVAL_USER_ID` | `admin` | Which user ID to attribute OAuth approvals to |
| `OAUTH_ACCESS_TOKEN_TTL` | `3600` | Access token lifetime in seconds |
| `OAUTH_REFRESH_TOKEN_TTL` | `2592000` | Refresh token lifetime in seconds (30 days) |
2026-06-25 13:59:51 +00:00
| **PostgreSQL (container) ** | | |
| `CTXD_PG_WAIT_SECONDS` | `120` | Entrypoint: max wait for DB before exit (when `DATABASE_URL` set) |
| `CTXD_PG_WAIT_INTERVAL` | `2` | Seconds between connection attempts |
2026-06-24 22:50:54 +00:00
| **Web Sessions ** | | |
| `WEB_SESSION_TTL` | `604800` | Session cookie lifetime in seconds (7 days) |
| **Snapshots ** | | |
| `SNAPSHOT_MIN_KEEP` | `5` | Minimum snapshots retained per project |
| `SNAPSHOT_MAX_KEEP` | `25` | Maximum snapshots before rotation |
### Using an External PostgreSQL
To use an external PostgreSQL instead of the bundled container:
1. Create a database and user on your external PG instance
2. Set `DATABASE_URL` in `.env` to point to it
2026-06-25 13:59:51 +00:00
3. Start only the app (no bundled postgres): `docker compose up -d --scale postgres=0 ctxd`
2026-06-24 22:50:54 +00:00
2026-06-25 13:59:51 +00:00
Ensure `DATABASE_URL` points at your external host (not `postgres` ). The entrypoint skips the compose-network wait when the URL is reachable.
2026-06-24 22:50:54 +00:00
### Fallback to SQLite
If `DATABASE_URL` is empty or not set, CTXD falls back to SQLite at `$CTXD_HOME/ctxd.db` . This is useful for local development or single-user deployments that don't need PostgreSQL features.
## MCP Surfaces
2026-06-25 12:45:59 +00:00
CTXD exposes MCP via Streamable HTTP on * * `/mcp` ** (single public connector):
2026-06-24 22:50:54 +00:00
| Endpoint | Auth | Scope | Tools |
|----------|------|-------|-------|
2026-06-25 12:45:59 +00:00
| `/mcp` | OAuth bearer | `ctxd.read` / `ctxd.write` | Scope-gated read + write tools |
| `/mcp` | Shared API key (LAN/Hermes) | * (full) * | All tools including `get_user_profile` , `auto_generate_tags` |
| `/readonly/mcp` , `/oauth/mcp` | OAuth (aliases) | same as `/mcp` | Backward-compatible URLs |
2026-06-24 22:50:54 +00:00
### Connecting an LLM Client
**Claude Desktop / Claude Web: **
```
2026-06-25 12:45:59 +00:00
Connector URL: https://ctxd.yourdomain.com/mcp
2026-06-24 22:50:54 +00:00
```
Claude auto-discovers OAuth metadata and registers via DCR. Request `scope=ctxd.read ctxd.write` for write access.
2026-06-25 13:46:22 +00:00
**ChatGPT (MCP connector — recommended; no Custom GPT required): **
ChatGPT can attach a **remote MCP server ** in **Developer mode ** (Plus/Pro and higher tiers for custom connectors). Use the same public URL as Claude — OAuth on `/mcp` , not REST Actions or a shared API key.
1. **Server ** — deploy current CTXD and expose the public host (Traefik must route `/mcp` and `/oauth/*` ; do not block `/mcp` ):
```bash
cd app
2026-06-25 13:59:51 +00:00
./scripts/deploy.sh
2026-06-25 13:46:22 +00:00
` ``
Smoke: ` curl -sS https://ctxd.yourdomain.com/.well-known/oauth-authorization-server | head`
2. **ChatGPT** — Settings → **Connectors** → enable **Developer mode** → **Add connector** (MCP / custom remote) → **Server URL:**
` ``
https://ctxd.yourdomain.com/mcp
` ``
Start OAuth and copy ChatGPT’ s **callback / redirect URL** exactly.
3. **CTXD admin** — Web UI → **admin** → **oauth clients**:
- After DCR, the client may appear automatically; otherwise **create client** with ChatGPT’ s redirect URI.
- Set **allowed scopes** to **` ctxd.read`** and **` ctxd.write`** (create form or **scopes** → save on an existing row).
4. **Authorize** — complete the browser approval (sign in as CTXD admin in that browser, or use the OAuth approval key). Tokens are capped by the client’ s allowed scopes.
5. **Chat** — enable the CTXD connector for the conversation. Tools include ` list_projects`, ` get_project_context`, ` search_context`, ` get_file`, ` update_file`, etc. Call **` get_client_guide`** first in a new session.
**Optional CLI pre-register** (if ChatGPT asks for client credentials before DCR):
` ``bash
docker exec ctxd dossier oauth-client-create -n "ChatGPT MCP" \
--redirect-uri 'PASTE_CHATGPT_CALLBACK_URL' \
--scope "ctxd.read ctxd.write"
` ``
| If connect fails | Check |
|------------------|--------|
2026-06-25 13:53:31 +00:00
| **“does not implement OAuth”** | Public host must return **200** JSON (not 502/404) for ` /.well-known/oauth-protected-resource/mcp` and ` /.well-known/oauth-authorization-server`. Traefik must proxy **all** paths to CTXD ` :9091`. Set ` OAUTH_ENABLED=true` and ` OAUTH_ISSUER=https://ctxd.cubecraftlabs.com`. |
| 502 on well-known URLs | Backend down or wrong Traefik service — fix routing before OAuth can work |
| 404 on ` /mcp` | Router not forwarding ` /mcp` to CTXD (often a stale ` !Path(/mcp)` rule) |
2026-06-25 13:46:22 +00:00
| 401 on ` /mcp` | Re-authorize; access token may have expired |
| Redirect mismatch | Callback URL in admin must match ChatGPT’ s string exactly |
| ` invalid_request` on authorize | CTXD requires PKCE **S256** (` code_challenge` + ` code_challenge_method=S256`) |
2026-06-25 13:59:51 +00:00
| Stale routes / old code | ` ./scripts/deploy.sh` or ` docker compose up -d --force-recreate ctxd` after postgres is healthy (not ` restart` alone; never ` --no-deps` unless PG is already up) |
2026-06-25 13:46:22 +00:00
| No write tools | Client scopes include ` ctxd.write`; re-authorize after changing scopes |
**Not required for ChatGPT MCP:** Custom GPT, OpenAPI Actions, or ` CTXD_API_KEY` in ChatGPT (public path is OAuth). Hermes continues to use LAN ` http://<server-ip>:9091/mcp` with the API key.
2026-06-24 22:50:54 +00:00
**Hermes Agent:**
` ``yaml
# ~/.hermes/config.yaml
mcp_servers:
dossier:
url: http://<server-ip>:9091/mcp
timeout: 30
headers:
Authorization: "Bearer your-api-key"
` ``
**Other MCP clients (Codex, Cursor, etc.):**
- Register an OAuth client via ` POST /oauth/register` with your redirect URI
2026-06-25 12:45:59 +00:00
- Connect to **` /mcp`** with ` scope=ctxd.read ctxd.write`
2026-06-24 22:50:54 +00:00
- Use the access token as ` Authorization: Bearer <token>`
### MCP Tool Reference
2026-06-25 12:45:59 +00:00
#### Read tools (require ` ctxd.read` on ` /mcp`)
2026-06-24 22:50:54 +00:00
| Tool | Args | Returns |
|------|------|---------|
| ` get_client_guide` | *(none)* | Locked ` LLM-CLIENT.MD` guide — **call this first** |
| ` list_projects` | *(none)* | All projects with version numbers |
| ` get_project_context` | ` project_id` | Compiled markdown of all context files |
| ` search_context` | ` query`, ` limit?` | FTS results across all projects |
| ` get_project_tags` | ` project_id` | Metadata tags for a project |
| ` list_files` | ` project_id` | All context files in a project |
| ` get_file` | ` project_id`, ` file_path` | Single file with metadata header |
2026-06-25 12:45:59 +00:00
#### Write tools (require ` ctxd.write` on ` /mcp`)
2026-06-24 22:50:54 +00:00
| Tool | Args | Returns |
|------|------|---------|
| ` update_file` | ` project_id`, ` file_path`, ` content`, ` base_version` | ` {"ok": true, "new_version": N}` or conflict error |
| ` set_project_tags` | ` project_id`, ` tags[]` | ` {"ok": true, "tags": [...]}` |
| ` sync_to_project` | ` project_id` | Writes ` CONTEXT.MD` as ` AGENTS.md` + symlinks to project root |
### Locked Files
| File | Scope | Protection |
|------|-------|------------|
| ` CONTEXT.MD` | All projects | Cannot delete — minimum required file |
| ` CONTEXT.MD` | ` ctxd-docs` project only | Cannot update or delete |
| ` LLM-CLIENT.MD` | ` ctxd-docs` project only | Cannot update or delete |
## OAuth
### Scopes
| Scope | Grants |
|-------|--------|
| ` ctxd.read` | Read-only MCP tools |
| ` ctxd.write` | Write MCP tools (includes read) |
Request both: ` scope=ctxd.read ctxd.write`
### Redirect URIs
| Platform | Redirect URI |
|----------|-------------|
| Claude Desktop | ` https://claude.ai/api/mcp/auth_callback` |
2026-06-25 13:46:22 +00:00
| ChatGPT (MCP) | Paste callback URL from ChatGPT connector OAuth UI (per connector) |
2026-06-24 22:50:54 +00:00
| Claude Code | ` http://localhost:5555/oauth/callback` |
| Codex CLI | ` http://localhost:7777/oauth/callback` |
| Custom | Your app's documented OAuth callback |
### Managing OAuth Clients
**Via Admin UI:** ` http://<server-ip>:9091/` → sign in as admin → **admin** → **oauth clients** tab
**Via CLI:**
` ``bash
# Create
2026-06-24 23:19:08 +00:00
docker exec ctxd dossier oauth-client-create -n "Claude Desktop" --redirect-uri https://claude.ai/api/mcp/auth_callback
2026-06-24 22:50:54 +00:00
# List
2026-06-24 23:19:08 +00:00
docker exec ctxd dossier oauth-client-list
2026-06-24 22:50:54 +00:00
# Revoke (invalidates all tokens for that client)
2026-06-24 23:19:08 +00:00
docker exec ctxd dossier oauth-client-revoke ctxd_xxxxxxxx
2026-06-24 22:50:54 +00:00
` ``
**Via API (admin session or API key):**
` ``bash
# Create
curl -X POST http://localhost:9091/oauth/clients \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d '{"client_name": "Claude Desktop", "redirect_uris": ["https://claude.ai/api/mcp/auth_callback"]}'
# List
curl http://localhost:9091/oauth/clients -H "Authorization: Bearer your-api-key"
# Revoke
curl -X DELETE http://localhost:9091/oauth/clients/ctxd_xxxxxxxx -H "Authorization: Bearer your-api-key"
` ``
## Public Exposure (Traefik)
### Router Rule
2026-06-25 12:45:59 +00:00
Route the public host to the backend (include ` /mcp` — OAuth protects it):
2026-06-24 22:50:54 +00:00
` ``yaml
2026-06-25 12:45:59 +00:00
rule: Host(` ctxd.yourdomain.com`)
2026-06-24 22:50:54 +00:00
` ``
This exposes:
- Landing page (` GET /`)
- Login (` POST /auth/login`, ` GET /auth/me`)
- Full Web UI dashboard (all REST API endpoints)
- OAuth (` /oauth/*`, ` /.well-known/*`)
2026-06-25 12:45:59 +00:00
- Public MCP (` /mcp` — OAuth read + write)
2026-06-24 22:50:54 +00:00
2026-06-25 12:45:59 +00:00
Hermes uses the same ` /mcp` path on LAN with the shared API key (not exposed via public OAuth).
2026-06-24 22:50:54 +00:00
### Landing Page Behavior
- **Not signed in** → themed landing page with login form
- **Signed in** (valid session cookie) → full dashboard
- Cookie: ` ctxd_session` (SameSite=Lax, 7-day expiry)
- After login: cookie set + redirect to dashboard
A full Traefik template is at ` app/templates/traefik-ctxd-readonly-mcp.yaml` in the ` project-context-management` Hermes skill.
## Web UI
### Admin Panel
Sign in as admin → click **admin** in the masthead.
**Tabs:**
1. **oauth clients** — client list (revoke per row) + create client form
2. **users** — user list (id, name, role, active/inactive) + manage users (create, edit, activate, inactivate, delete)
3. **projects** — manage projects (list with remove button, typed-name confirmation for delete)
### Project Files
Each project has multiple context files:
| File | Purpose |
|------|---------|
| ` CONTEXT.MD` | Canonical project overview (synced as ` AGENTS.md` to repos) |
| ` DECISIONS.MD` | Architecture decisions, rationale |
| ` RUNBOOKS.MD` | Deploy, troubleshoot, operate procedures |
| ` PROMPTS.MD` | Project-specific prompts for different harnesses |
| ` GLOSSARY.MD` | Project-specific terms, acronyms |
The compiled view (` get_project_context`) concatenates all files with ` ## FILENAME` headers and a single metadata block at the top.
## CLI
All commands run inside the container:
` ``bash
2026-06-24 23:19:08 +00:00
docker exec ctxd dossier <command>
2026-06-24 22:50:54 +00:00
` ``
### Commands
` ``bash
# Initialize (auto-runs on first container start)
dossier init
# Projects
dossier project-create <project_id> [--display-name "Name"] [--description "Desc"]
dossier project-list
dossier read <project_id> # Print context to stdout
dossier edit <project_id> # Open in $EDITOR
# Context files
dossier file-list <project_id>
dossier file-read <project_id> <file_path>
# Sync
dossier sync <project_id> [path] # Set sync path and/or sync AGENTS.md
# Search
dossier search "query" # FTS across all projects
# Audit
dossier audit [--limit N]
# Users
dossier user-list
dossier user-create <user_id> --display-name "Name" [--password "pw"]
dossier user-set-password <user_id> -p "password"
# OAuth
dossier oauth-client-create [-n "Name"] [--redirect-uri URI]
dossier oauth-client-list
dossier oauth-client-revoke <client_id>
# Import
dossier import-vault <path> # Import from Obsidian vault
` ``
## REST API
All endpoints require ` Authorization: Bearer <api_key>` or ` Authorization: Bearer <session_token>` unless noted.
| Method | Path | Description |
|--------|------|-------------|
| ` GET` | ` /` | Web UI (LAN) or landing page (public host) |
| ` GET` | ` /status` | Health check (no auth) |
| ` POST` | ` /auth/login` | Web UI login → session token + cookie |
| ` POST` | ` /auth/logout` | Revoke session + clear cookie |
| ` GET` | ` /auth/me` | Current session identity |
| ` GET` | ` /users` | List users |
| ` POST` | ` /users` | Create user |
| ` PATCH` | ` /users/<id>` | Update user (admin) |
| ` DELETE` | ` /users/<id>` | Delete user (admin) |
| ` POST` | ` /users/<id>/password` | Set password (admin) |
| ` GET` | ` /oauth/clients` | List OAuth clients (admin) |
| ` POST` | ` /oauth/clients` | Register OAuth client (admin) |
| ` DELETE` | ` /oauth/clients/<id>` | Revoke OAuth client (admin) |
| ` GET` | ` /projects` | List all projects |
| ` POST` | ` /projects` | Create a project |
| ` DELETE` | ` /projects/<id>` | Delete a project (admin) |
| ` GET` | ` /projects/<id>/context` | Compiled context (all files) |
| ` POST` | ` /projects/<id>/context` | Update context (legacy single-file) |
| ` GET` | ` /projects/<id>/files` | List context files |
| ` GET` | ` /projects/<id>/files/<name>` | Read a single file |
| ` POST` | ` /projects/<id>/files` | Create a new file |
| ` PUT` | ` /projects/<id>/files/<name>` | Update a file (version-checked) |
| ` DELETE` | ` /projects/<id>/files/<name>` | Delete a file |
| ` POST` | ` /projects/<id>/migrate-files` | Migrate single-context to multi-file |
| ` GET` | ` /projects/<id>/snapshots` | List snapshots |
| ` GET` | ` /projects/<id>/tags` | Get metadata tags |
| ` POST` | ` /projects/<id>/tags` | Set metadata tags |
| ` POST` | ` /projects/<id>/sync` | Sync CONTEXT.MD as AGENTS.md |
| ` POST` | ` /projects/<id>/import` | Import raw text as context |
| ` GET` | ` /search?q=...` | Full-text search |
| ` GET` | ` /audit?limit=N` | Audit log |
| ` GET` | ` /.well-known/oauth-authorization-server` | OAuth discovery |
| ` GET` | ` /.well-known/oauth-protected-resource` | Resource metadata |
| ` POST` | ` /oauth/register` | Dynamic Client Registration |
| ` GET/POST` | ` /oauth/authorize` | Authorization endpoint |
| ` POST` | ` /oauth/token` | Token endpoint |
| ` POST/GET/DELETE` | ` /readonly/mcp` | Read-only MCP (Streamable HTTP) |
| ` POST/GET/DELETE` | ` /write/mcp` | Write MCP (Streamable HTTP) |
| ` POST/GET/DELETE` | ` /mcp` | Internal full MCP (API key only) |
## Backups
### PostgreSQL
` ``bash
# Backup
docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup_$(date +%Y%m%d).sql
# Restore
cat backup_YYYYMMDD.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd
` ``
### Snapshots
CTXD automatically takes point-in-time snapshots before each context update. Snapshots are stored as files in ` /data/snapshots/<project_id>/` and rotated (min 5, max 25 per project).
## Migrating from SQLite to PostgreSQL
If you started with SQLite and want to move to PostgreSQL:
` ``bash
# 1. Start PostgreSQL
docker compose up -d postgres
# 2. Run the migration script (reads from /data/ctxd.db, writes to DATABASE_URL)
2026-06-24 23:19:08 +00:00
docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
2026-06-24 22:50:54 +00:00
# 3. Set DATABASE_URL in .env and restart
docker compose up -d dossier
` ``
The migration handles all tables, rebuilds the FTS index, and skips orphaned rows with FK violations.
## Project Structure
` ``
/mnt/ai-storage/projects/ctxd/
├── .env # Production environment (gitignored)
├── .env.example # Template (committed)
├── .gitignore
├── SKILL.md # LLM client guide (canonical source)
├── README.md # This file
├── data/ # Runtime data (gitignored)
│ ├── ctxd.yaml # Fallback config (env vars take precedence)
│ ├── ctxd.db # SQLite DB (if no DATABASE_URL)
│ ├── pg/ # PostgreSQL data volume
│ ├── oauth_state.json # OAuth clients, codes, tokens
│ ├── web_sessions.json # Web UI sessions
│ └── snapshots/ # Point-in-time backups
└── app/ # Application source
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
├── .env # Same as root .env (symlinked or copied)
└── src/ctxd/
├── __init__.py
├── __main__.py # CLI/daemon entry point
├── config.py # Env-driven config with yaml fallback
├── db.py # Database layer (PostgreSQL + SQLite)
├── schema.sql # PostgreSQL schema
├── schema_sqlite.sql # SQLite schema (fallback)
├── server.py # ASGI app: HTTP + MCP + OAuth
├── cli.py # CLI commands
├── ui.html # Web UI dashboard
├── landing.html # Public landing page
├── auth_password.py # PBKDF2 password hashing
└── migrate_sqlite_to_pg.py # One-time migration script
` ``
## Development
### Local Development (SQLite, no Docker)
` ``bash
cd app
pip install -e ".[mcp]"
export CTXD_HOME=./dev-data
python -m ctxd init
python -m ctxd
# → http://localhost:9091
` ``
### Rebuilding After Code Changes
` ``bash
cd app
docker compose build
docker compose up -d --no-build
# Verify:
curl http://localhost:9091/status
2026-06-24 23:19:08 +00:00
docker logs ctxd --tail 20
2026-06-24 22:50:54 +00:00
` ``
### Key Conventions
- **Metadata headers** are dynamically generated on read, never stored in the DB
- **File paths** are normalized to uppercase with ` .MD` extension
- **` CONTEXT.MD`** is the minimum required file — cannot be deleted from any project
- **Version checking** uses ` base_version` parameter — mismatches return ` 409 conflict`
- **Audit log** is append-only at the application layer (every operation is logged)
- **OAuth state** (` oauth_state.json`) and **web sessions** (` web_sessions.json`) are file-based, not in PostgreSQL
## Troubleshooting
### Login fails with "invalid credentials"
` ``bash
# Reset admin password
2026-06-24 23:19:08 +00:00
docker exec ctxd dossier user-set-password admin -p "new-password"
2026-06-24 22:50:54 +00:00
` ``
If the password contains special characters, use the quoting that matches:
- ` '` in password → use double quotes (` "`)
- ` $`, ` ` ` ``, ` !` in password → use single quotes (` '`)
### Login works on LAN but not on public host
The public host (` https://ctxd.yourdomain.com`) requires Traefik to route ` /auth/login` and ` /auth/me`. Check your Traefik router rule includes all paths (use ` !Path(`/mcp` )` to block only the internal MCP).
### MCP connection fails
1. Check OAuth discovery: ` curl https://ctxd.yourdomain.com/.well-known/oauth-authorization-server`
2. Check MCP endpoint: ` curl -o /dev/null -w '%{http_code}' https://ctxd.yourdomain.com/readonly/mcp` → should be ` 401` (not ` 404`)
3. If ` 404`: Traefik isn't routing ` /readonly/mcp` — update the router rule
4. If ` 401`: auth is working — check OAuth token scope and expiry
### PostgreSQL connection fails
` ``bash
# Check PG is running
docker compose ps postgres
# Check connection
2026-06-24 23:19:08 +00:00
docker exec ctxd python3 -c "
2026-06-24 22:50:54 +00:00
import os
import psycopg
conn = psycopg.connect(os.environ['DATABASE_URL'])
print('Connected:', conn.info.server_version)
"
# If password mismatch (PG data volume initialized with different password):
docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'new-password'"
# Then update .env with the new password
` ``
2026-06-25 13:59:51 +00:00
### Container keeps restarting / public site 502
2026-06-24 22:50:54 +00:00
` ``bash
2026-06-25 13:59:51 +00:00
docker compose ps -a
docker logs ctxd --tail 40
2026-06-24 22:50:54 +00:00
` ``
2026-06-25 13:59:51 +00:00
| Symptom | Cause | Fix |
|---------|--------|-----|
| ` failed to resolve host 'postgres'` | ` ctxd-postgres` not running (often after ` --no-deps ctxd`) | ` cd app && docker compose up -d postgres ctxd` or ` ./scripts/deploy.sh` |
| ` Restarting (1)` on ` ctxd` only | Same — app up without DB on compose network | Start postgres first; wait for **(healthy)** |
| Cloudflare **502** on public URL | Traefik/backend has no healthy upstream on ` :9091` | Fix local ` curl http://127.0.0.1:9091/status` first |
The container **entrypoint waits up to 120s** for PostgreSQL when ` DATABASE_URL` is set (` CTXD_PG_WAIT_SECONDS` to override). If postgres never appears, logs print an explicit message instead of an immediate opaque crash.
Common other causes:
2026-06-24 22:50:54 +00:00
- ` DATABASE_URL` password doesn't match what PG was initialized with
- ` OAUTH_ENABLED=true` but ` OAUTH_ISSUER` is empty
- Missing ` CTXD_API_KEY` when ` CTXD_AUTH_ENABLED=true`
## License
MIT