9e85c1b8ec
- entrypoint: wait for DATABASE_URL (CTXD_PG_WAIT_SECONDS) with clear fatal message - scripts/deploy.sh: postgres healthy then force-recreate ctxd - compose + README: ban --no-deps ctxd as default; 502 troubleshooting
661 lines
26 KiB
Markdown
661 lines
26 KiB
Markdown
# CTXD — Context Dossier
|
||
|
||
A single source of truth for multi-harness project context. One canonical `AGENTS.md` per project, served to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP.
|
||
|
||
## Overview
|
||
|
||
CTXD solves context sprawl: when you work across multiple LLM harnesses (Claude Desktop, Claude Code, Codex CLI, Cursor, Copilot, Hermes), each has its own context file convention (`CLAUDE.md`, `.cursorrules`, `CODEX.md`, etc.). Without a canonical source, these drift independently.
|
||
|
||
CTXD provides:
|
||
- **Multi-file context per project** — `CONTEXT.MD`, `DECISIONS.MD`, `RUNBOOKS.MD`, `PROMPTS.MD`, `GLOSSARY.MD`
|
||
- **Compiled view** — all files concatenated with metadata header, served as a single document to agents
|
||
- **Sync to repos** — writes `CONTEXT.MD` as `AGENTS.md` + symlinks (`CLAUDE.md`, `.cursorrules`, `CODEX.md` → `AGENTS.md`)
|
||
- **Version-checked writes** — optimistic concurrency with `base_version` to prevent silent overwrites
|
||
- **OAuth 2.0 authorization server** — DCR, Authorization Code + PKCE, `ctxd.read` and `ctxd.write` scopes
|
||
- **Streamable HTTP MCP** — single-endpoint transport for read-only and write surfaces
|
||
- **Web UI** — per-user password login, admin panel, project/file management
|
||
- **PostgreSQL backend** — with SQLite fallback for local dev
|
||
- **Append-only audit log** — every read, write, create, delete, sync, and search is logged
|
||
- **Point-in-time snapshots** — automatic version snapshots with rotation (min 5, max 25 per project)
|
||
- **Full-text search** — PostgreSQL `tsvector` with GIN index (FTS5 on SQLite fallback)
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Context Dossier (container: ctxd, 0.0.0.0:9091)
|
||
├── PostgreSQL 16 (container: ctxd-postgres) # Primary DB
|
||
├── /data # Config, OAuth state, web sessions
|
||
│ ├── ctxd.yaml # Fallback config (env vars take precedence)
|
||
│ ├── oauth_state.json # OAuth clients, codes, tokens
|
||
│ ├── web_sessions.json # Per-user web UI sessions
|
||
│ └── snapshots/ # Point-in-time context backups
|
||
├── Streamable HTTP MCP:
|
||
│ ├── /mcp (OAuth ctxd.read + ctxd.write; API key on LAN = full tools)
|
||
│ ├── /readonly/mcp (alias → same OAuth behavior)
|
||
│ └── /oauth/mcp (alias)
|
||
├── OAuth Authorization Server:
|
||
│ ├── /.well-known/oauth-authorization-server # Discovery
|
||
│ ├── /.well-known/oauth-protected-resource # Resource metadata
|
||
│ ├── /oauth/register # Dynamic Client Registration
|
||
│ ├── /oauth/authorize # Authorization + PKCE
|
||
│ └── /oauth/token # Token + refresh
|
||
├── Web UI + REST API (/) # Dashboard, admin, projects, files
|
||
└── Landing page (public host only) # Themed login form
|
||
```
|
||
|
||
## Quick Start
|
||
|
||
### Prerequisites
|
||
|
||
- Docker and Docker Compose
|
||
- A reverse proxy with TLS (Traefik, Caddy, nginx) for public exposure
|
||
- (Optional) An existing PostgreSQL 14+ instance if not using the bundled one
|
||
|
||
### 1. Clone and configure
|
||
|
||
```bash
|
||
cd /mnt/ai-storage/projects/ctxd/app
|
||
cp .env.example .env
|
||
```
|
||
|
||
Edit `.env` with your values:
|
||
|
||
```bash
|
||
# Database
|
||
DATABASE_URL=postgresql://ctxd:your-password@postgres:5432/ctxd
|
||
POSTGRES_USER=ctxd
|
||
POSTGRES_PASSWORD=your-strong-password
|
||
|
||
# Server
|
||
CTXD_HOST=0.0.0.0
|
||
CTXD_PORT=9091
|
||
CTXD_HOME=/data
|
||
|
||
# Auth
|
||
CTXD_AUTH_ENABLED=true
|
||
CTXD_API_KEY=your-api-key-here # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
||
|
||
# OAuth
|
||
OAUTH_ENABLED=true
|
||
OAUTH_ISSUER=https://ctxd.yourdomain.com
|
||
OAUTH_APPROVAL_KEY=your-approval-key # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
||
```
|
||
|
||
### 2. Build and start
|
||
|
||
```bash
|
||
# Recommended: build + postgres + recreate ctxd (avoids 502 when PG was never started)
|
||
chmod +x scripts/deploy.sh
|
||
./scripts/deploy.sh
|
||
|
||
# Or manual (always include postgres when DATABASE_URL uses host "postgres"):
|
||
docker compose up -d
|
||
```
|
||
|
||
This starts:
|
||
- `ctxd-postgres` — PostgreSQL 16 (Alpine)
|
||
- `ctxd` — CTXD daemon (web UI + MCP + OAuth + REST API)
|
||
|
||
**After code changes**, use `./scripts/deploy.sh` or `docker compose up -d` — not `docker restart ctxd` alone (old image) and **not** `docker compose up -d --no-deps ctxd` (skips postgres → crash loop → public **502**).
|
||
|
||
### 3. Verify
|
||
|
||
```bash
|
||
# Health check
|
||
curl http://localhost:9091/status
|
||
# → {"status": "ok", "db": "/data/ctxd.db"}
|
||
|
||
# List projects (requires API key)
|
||
curl http://localhost:9091/projects -H "Authorization: Bearer your-api-key"
|
||
```
|
||
|
||
### 4. Set admin password
|
||
|
||
```bash
|
||
docker exec ctxd dossier user-set-password admin -p "your-admin-password"
|
||
```
|
||
|
||
> **Shell quoting:** Use double quotes (`"`) if the password contains single quotes (`'`). Use single quotes (`'`) if it contains `$`, backticks, or `!`. The failure mode is shell expansion, not PBKDF2.
|
||
|
||
### 5. Access the Web UI
|
||
|
||
- **LAN:** `http://<server-ip>:9091/`
|
||
- **Public (via Traefik):** `https://ctxd.yourdomain.com/`
|
||
|
||
Sign in with `admin` / your password.
|
||
|
||
## Configuration
|
||
|
||
### Environment Variables
|
||
|
||
All config is driven by environment variables. A `ctxd.yaml` file in `/data` can override built-in defaults, but env vars always take precedence.
|
||
|
||
**Precedence:** env var > `ctxd.yaml` > built-in default
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| **Database** | | |
|
||
| `DATABASE_URL` | *(empty)* | PostgreSQL connection string. If empty, falls back to SQLite at `$CTXD_HOME/ctxd.db` |
|
||
| `POSTGRES_USER` | `ctxd` | PostgreSQL user (for bundled PG container) |
|
||
| `POSTGRES_PASSWORD` | `ctxd_local_dev` | PostgreSQL password (for bundled PG container) |
|
||
| `POSTGRES_DB` | `ctxd` | PostgreSQL database name (for bundled PG container) |
|
||
| **Server** | | |
|
||
| `CTXD_HOST` | `0.0.0.0` | Bind address |
|
||
| `CTXD_PORT` | `9091` | Listen port |
|
||
| `CTXD_HOME` | `~/.ctx` | Data directory (inside container: `/data`) |
|
||
| `LOG_LEVEL` | `info` | Uvicorn log level (`debug`, `info`, `warning`, `error`) |
|
||
| **Auth** | | |
|
||
| `CTXD_AUTH_ENABLED` | `false` | Enable authentication globally |
|
||
| `CTXD_API_KEY` | *(empty)* | Shared API key for Hermes/internal MCP + HTTP auth |
|
||
| `CTXD_EXTERNAL_READONLY_KEY` | *(empty)* | Legacy `?key=` on read-only MCP (migration only) |
|
||
| **OAuth** | | |
|
||
| `OAUTH_ENABLED` | `false` | Enable OAuth authorization server |
|
||
| `OAUTH_ISSUER` | *(empty)* | Public URL (used in OAuth discovery metadata) |
|
||
| `OAUTH_APPROVAL_KEY` | *(empty)* | Fallback approval key for `/oauth/authorize` |
|
||
| `OAUTH_APPROVAL_USER_ID` | `admin` | Which user ID to attribute OAuth approvals to |
|
||
| `OAUTH_ACCESS_TOKEN_TTL` | `3600` | Access token lifetime in seconds |
|
||
| `OAUTH_REFRESH_TOKEN_TTL` | `2592000` | Refresh token lifetime in seconds (30 days) |
|
||
| **PostgreSQL (container)** | | |
|
||
| `CTXD_PG_WAIT_SECONDS` | `120` | Entrypoint: max wait for DB before exit (when `DATABASE_URL` set) |
|
||
| `CTXD_PG_WAIT_INTERVAL` | `2` | Seconds between connection attempts |
|
||
| **Web Sessions** | | |
|
||
| `WEB_SESSION_TTL` | `604800` | Session cookie lifetime in seconds (7 days) |
|
||
| **Snapshots** | | |
|
||
| `SNAPSHOT_MIN_KEEP` | `5` | Minimum snapshots retained per project |
|
||
| `SNAPSHOT_MAX_KEEP` | `25` | Maximum snapshots before rotation |
|
||
|
||
### Using an External PostgreSQL
|
||
|
||
To use an external PostgreSQL instead of the bundled container:
|
||
|
||
1. Create a database and user on your external PG instance
|
||
2. Set `DATABASE_URL` in `.env` to point to it
|
||
3. Start only the app (no bundled postgres): `docker compose up -d --scale postgres=0 ctxd`
|
||
|
||
Ensure `DATABASE_URL` points at your external host (not `postgres`). The entrypoint skips the compose-network wait when the URL is reachable.
|
||
|
||
### Fallback to SQLite
|
||
|
||
If `DATABASE_URL` is empty or not set, CTXD falls back to SQLite at `$CTXD_HOME/ctxd.db`. This is useful for local development or single-user deployments that don't need PostgreSQL features.
|
||
|
||
## MCP Surfaces
|
||
|
||
CTXD exposes MCP via Streamable HTTP on **`/mcp`** (single public connector):
|
||
|
||
| Endpoint | Auth | Scope | Tools |
|
||
|----------|------|-------|-------|
|
||
| `/mcp` | OAuth bearer | `ctxd.read` / `ctxd.write` | Scope-gated read + write tools |
|
||
| `/mcp` | Shared API key (LAN/Hermes) | *(full)* | All tools including `get_user_profile`, `auto_generate_tags` |
|
||
| `/readonly/mcp`, `/oauth/mcp` | OAuth (aliases) | same as `/mcp` | Backward-compatible URLs |
|
||
|
||
### Connecting an LLM Client
|
||
|
||
**Claude Desktop / Claude Web:**
|
||
```
|
||
Connector URL: https://ctxd.yourdomain.com/mcp
|
||
```
|
||
Claude auto-discovers OAuth metadata and registers via DCR. Request `scope=ctxd.read ctxd.write` for write access.
|
||
|
||
**ChatGPT (MCP connector — recommended; no Custom GPT required):**
|
||
|
||
ChatGPT can attach a **remote MCP server** in **Developer mode** (Plus/Pro and higher tiers for custom connectors). Use the same public URL as Claude — OAuth on `/mcp`, not REST Actions or a shared API key.
|
||
|
||
1. **Server** — deploy current CTXD and expose the public host (Traefik must route `/mcp` and `/oauth/*`; do not block `/mcp`):
|
||
|
||
```bash
|
||
cd app
|
||
./scripts/deploy.sh
|
||
```
|
||
|
||
Smoke: `curl -sS https://ctxd.yourdomain.com/.well-known/oauth-authorization-server | head`
|
||
|
||
2. **ChatGPT** — Settings → **Connectors** → enable **Developer mode** → **Add connector** (MCP / custom remote) → **Server URL:**
|
||
|
||
```
|
||
https://ctxd.yourdomain.com/mcp
|
||
```
|
||
|
||
Start OAuth and copy ChatGPT’s **callback / redirect URL** exactly.
|
||
|
||
3. **CTXD admin** — Web UI → **admin** → **oauth clients**:
|
||
|
||
- After DCR, the client may appear automatically; otherwise **create client** with ChatGPT’s redirect URI.
|
||
- Set **allowed scopes** to **`ctxd.read`** and **`ctxd.write`** (create form or **scopes** → save on an existing row).
|
||
|
||
4. **Authorize** — complete the browser approval (sign in as CTXD admin in that browser, or use the OAuth approval key). Tokens are capped by the client’s allowed scopes.
|
||
|
||
5. **Chat** — enable the CTXD connector for the conversation. Tools include `list_projects`, `get_project_context`, `search_context`, `get_file`, `update_file`, etc. Call **`get_client_guide`** first in a new session.
|
||
|
||
**Optional CLI pre-register** (if ChatGPT asks for client credentials before DCR):
|
||
|
||
```bash
|
||
docker exec ctxd dossier oauth-client-create -n "ChatGPT MCP" \
|
||
--redirect-uri 'PASTE_CHATGPT_CALLBACK_URL' \
|
||
--scope "ctxd.read ctxd.write"
|
||
```
|
||
|
||
| If connect fails | Check |
|
||
|------------------|--------|
|
||
| **“does not implement OAuth”** | Public host must return **200** JSON (not 502/404) for `/.well-known/oauth-protected-resource/mcp` and `/.well-known/oauth-authorization-server`. Traefik must proxy **all** paths to CTXD `:9091`. Set `OAUTH_ENABLED=true` and `OAUTH_ISSUER=https://ctxd.cubecraftlabs.com`. |
|
||
| 502 on well-known URLs | Backend down or wrong Traefik service — fix routing before OAuth can work |
|
||
| 404 on `/mcp` | Router not forwarding `/mcp` to CTXD (often a stale `!Path(/mcp)` rule) |
|
||
| 401 on `/mcp` | Re-authorize; access token may have expired |
|
||
| Redirect mismatch | Callback URL in admin must match ChatGPT’s string exactly |
|
||
| `invalid_request` on authorize | CTXD requires PKCE **S256** (`code_challenge` + `code_challenge_method=S256`) |
|
||
| Stale routes / old code | `./scripts/deploy.sh` or `docker compose up -d --force-recreate ctxd` after postgres is healthy (not `restart` alone; never `--no-deps` unless PG is already up) |
|
||
| No write tools | Client scopes include `ctxd.write`; re-authorize after changing scopes |
|
||
|
||
**Not required for ChatGPT MCP:** Custom GPT, OpenAPI Actions, or `CTXD_API_KEY` in ChatGPT (public path is OAuth). Hermes continues to use LAN `http://<server-ip>:9091/mcp` with the API key.
|
||
|
||
**Hermes Agent:**
|
||
```yaml
|
||
# ~/.hermes/config.yaml
|
||
mcp_servers:
|
||
dossier:
|
||
url: http://<server-ip>:9091/mcp
|
||
timeout: 30
|
||
headers:
|
||
Authorization: "Bearer your-api-key"
|
||
```
|
||
|
||
**Other MCP clients (Codex, Cursor, etc.):**
|
||
- Register an OAuth client via `POST /oauth/register` with your redirect URI
|
||
- Connect to **`/mcp`** with `scope=ctxd.read ctxd.write`
|
||
- Use the access token as `Authorization: Bearer <token>`
|
||
|
||
### MCP Tool Reference
|
||
|
||
#### Read tools (require `ctxd.read` on `/mcp`)
|
||
|
||
| Tool | Args | Returns |
|
||
|------|------|---------|
|
||
| `get_client_guide` | *(none)* | Locked `LLM-CLIENT.MD` guide — **call this first** |
|
||
| `list_projects` | *(none)* | All projects with version numbers |
|
||
| `get_project_context` | `project_id` | Compiled markdown of all context files |
|
||
| `search_context` | `query`, `limit?` | FTS results across all projects |
|
||
| `get_project_tags` | `project_id` | Metadata tags for a project |
|
||
| `list_files` | `project_id` | All context files in a project |
|
||
| `get_file` | `project_id`, `file_path` | Single file with metadata header |
|
||
|
||
#### Write tools (require `ctxd.write` on `/mcp`)
|
||
|
||
| Tool | Args | Returns |
|
||
|------|------|---------|
|
||
| `update_file` | `project_id`, `file_path`, `content`, `base_version` | `{"ok": true, "new_version": N}` or conflict error |
|
||
| `set_project_tags` | `project_id`, `tags[]` | `{"ok": true, "tags": [...]}` |
|
||
| `sync_to_project` | `project_id` | Writes `CONTEXT.MD` as `AGENTS.md` + symlinks to project root |
|
||
|
||
### Locked Files
|
||
|
||
| File | Scope | Protection |
|
||
|------|-------|------------|
|
||
| `CONTEXT.MD` | All projects | Cannot delete — minimum required file |
|
||
| `CONTEXT.MD` | `ctxd-docs` project only | Cannot update or delete |
|
||
| `LLM-CLIENT.MD` | `ctxd-docs` project only | Cannot update or delete |
|
||
|
||
## OAuth
|
||
|
||
### Scopes
|
||
|
||
| Scope | Grants |
|
||
|-------|--------|
|
||
| `ctxd.read` | Read-only MCP tools |
|
||
| `ctxd.write` | Write MCP tools (includes read) |
|
||
|
||
Request both: `scope=ctxd.read ctxd.write`
|
||
|
||
### Redirect URIs
|
||
|
||
| Platform | Redirect URI |
|
||
|----------|-------------|
|
||
| Claude Desktop | `https://claude.ai/api/mcp/auth_callback` |
|
||
| ChatGPT (MCP) | Paste callback URL from ChatGPT connector OAuth UI (per connector) |
|
||
| Claude Code | `http://localhost:5555/oauth/callback` |
|
||
| Codex CLI | `http://localhost:7777/oauth/callback` |
|
||
| Custom | Your app's documented OAuth callback |
|
||
|
||
### Managing OAuth Clients
|
||
|
||
**Via Admin UI:** `http://<server-ip>:9091/` → sign in as admin → **admin** → **oauth clients** tab
|
||
|
||
**Via CLI:**
|
||
```bash
|
||
# Create
|
||
docker exec ctxd dossier oauth-client-create -n "Claude Desktop" --redirect-uri https://claude.ai/api/mcp/auth_callback
|
||
|
||
# List
|
||
docker exec ctxd dossier oauth-client-list
|
||
|
||
# Revoke (invalidates all tokens for that client)
|
||
docker exec ctxd dossier oauth-client-revoke ctxd_xxxxxxxx
|
||
```
|
||
|
||
**Via API (admin session or API key):**
|
||
```bash
|
||
# Create
|
||
curl -X POST http://localhost:9091/oauth/clients \
|
||
-H "Authorization: Bearer your-api-key" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"client_name": "Claude Desktop", "redirect_uris": ["https://claude.ai/api/mcp/auth_callback"]}'
|
||
|
||
# List
|
||
curl http://localhost:9091/oauth/clients -H "Authorization: Bearer your-api-key"
|
||
|
||
# Revoke
|
||
curl -X DELETE http://localhost:9091/oauth/clients/ctxd_xxxxxxxx -H "Authorization: Bearer your-api-key"
|
||
```
|
||
|
||
## Public Exposure (Traefik)
|
||
|
||
### Router Rule
|
||
|
||
Route the public host to the backend (include `/mcp` — OAuth protects it):
|
||
|
||
```yaml
|
||
rule: Host(`ctxd.yourdomain.com`)
|
||
```
|
||
|
||
This exposes:
|
||
- Landing page (`GET /`)
|
||
- Login (`POST /auth/login`, `GET /auth/me`)
|
||
- Full Web UI dashboard (all REST API endpoints)
|
||
- OAuth (`/oauth/*`, `/.well-known/*`)
|
||
- Public MCP (`/mcp` — OAuth read + write)
|
||
|
||
Hermes uses the same `/mcp` path on LAN with the shared API key (not exposed via public OAuth).
|
||
|
||
### Landing Page Behavior
|
||
|
||
- **Not signed in** → themed landing page with login form
|
||
- **Signed in** (valid session cookie) → full dashboard
|
||
- Cookie: `ctxd_session` (SameSite=Lax, 7-day expiry)
|
||
- After login: cookie set + redirect to dashboard
|
||
|
||
A full Traefik template is at `app/templates/traefik-ctxd-readonly-mcp.yaml` in the `project-context-management` Hermes skill.
|
||
|
||
## Web UI
|
||
|
||
### Admin Panel
|
||
|
||
Sign in as admin → click **admin** in the masthead.
|
||
|
||
**Tabs:**
|
||
1. **oauth clients** — client list (revoke per row) + create client form
|
||
2. **users** — user list (id, name, role, active/inactive) + manage users (create, edit, activate, inactivate, delete)
|
||
3. **projects** — manage projects (list with remove button, typed-name confirmation for delete)
|
||
|
||
### Project Files
|
||
|
||
Each project has multiple context files:
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `CONTEXT.MD` | Canonical project overview (synced as `AGENTS.md` to repos) |
|
||
| `DECISIONS.MD` | Architecture decisions, rationale |
|
||
| `RUNBOOKS.MD` | Deploy, troubleshoot, operate procedures |
|
||
| `PROMPTS.MD` | Project-specific prompts for different harnesses |
|
||
| `GLOSSARY.MD` | Project-specific terms, acronyms |
|
||
|
||
The compiled view (`get_project_context`) concatenates all files with `## FILENAME` headers and a single metadata block at the top.
|
||
|
||
## CLI
|
||
|
||
All commands run inside the container:
|
||
|
||
```bash
|
||
docker exec ctxd dossier <command>
|
||
```
|
||
|
||
### Commands
|
||
|
||
```bash
|
||
# Initialize (auto-runs on first container start)
|
||
dossier init
|
||
|
||
# Projects
|
||
dossier project-create <project_id> [--display-name "Name"] [--description "Desc"]
|
||
dossier project-list
|
||
dossier read <project_id> # Print context to stdout
|
||
dossier edit <project_id> # Open in $EDITOR
|
||
|
||
# Context files
|
||
dossier file-list <project_id>
|
||
dossier file-read <project_id> <file_path>
|
||
|
||
# Sync
|
||
dossier sync <project_id> [path] # Set sync path and/or sync AGENTS.md
|
||
|
||
# Search
|
||
dossier search "query" # FTS across all projects
|
||
|
||
# Audit
|
||
dossier audit [--limit N]
|
||
|
||
# Users
|
||
dossier user-list
|
||
dossier user-create <user_id> --display-name "Name" [--password "pw"]
|
||
dossier user-set-password <user_id> -p "password"
|
||
|
||
# OAuth
|
||
dossier oauth-client-create [-n "Name"] [--redirect-uri URI]
|
||
dossier oauth-client-list
|
||
dossier oauth-client-revoke <client_id>
|
||
|
||
# Import
|
||
dossier import-vault <path> # Import from Obsidian vault
|
||
```
|
||
|
||
## REST API
|
||
|
||
All endpoints require `Authorization: Bearer <api_key>` or `Authorization: Bearer <session_token>` unless noted.
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `GET` | `/` | Web UI (LAN) or landing page (public host) |
|
||
| `GET` | `/status` | Health check (no auth) |
|
||
| `POST` | `/auth/login` | Web UI login → session token + cookie |
|
||
| `POST` | `/auth/logout` | Revoke session + clear cookie |
|
||
| `GET` | `/auth/me` | Current session identity |
|
||
| `GET` | `/users` | List users |
|
||
| `POST` | `/users` | Create user |
|
||
| `PATCH` | `/users/<id>` | Update user (admin) |
|
||
| `DELETE` | `/users/<id>` | Delete user (admin) |
|
||
| `POST` | `/users/<id>/password` | Set password (admin) |
|
||
| `GET` | `/oauth/clients` | List OAuth clients (admin) |
|
||
| `POST` | `/oauth/clients` | Register OAuth client (admin) |
|
||
| `DELETE` | `/oauth/clients/<id>` | Revoke OAuth client (admin) |
|
||
| `GET` | `/projects` | List all projects |
|
||
| `POST` | `/projects` | Create a project |
|
||
| `DELETE` | `/projects/<id>` | Delete a project (admin) |
|
||
| `GET` | `/projects/<id>/context` | Compiled context (all files) |
|
||
| `POST` | `/projects/<id>/context` | Update context (legacy single-file) |
|
||
| `GET` | `/projects/<id>/files` | List context files |
|
||
| `GET` | `/projects/<id>/files/<name>` | Read a single file |
|
||
| `POST` | `/projects/<id>/files` | Create a new file |
|
||
| `PUT` | `/projects/<id>/files/<name>` | Update a file (version-checked) |
|
||
| `DELETE` | `/projects/<id>/files/<name>` | Delete a file |
|
||
| `POST` | `/projects/<id>/migrate-files` | Migrate single-context to multi-file |
|
||
| `GET` | `/projects/<id>/snapshots` | List snapshots |
|
||
| `GET` | `/projects/<id>/tags` | Get metadata tags |
|
||
| `POST` | `/projects/<id>/tags` | Set metadata tags |
|
||
| `POST` | `/projects/<id>/sync` | Sync CONTEXT.MD as AGENTS.md |
|
||
| `POST` | `/projects/<id>/import` | Import raw text as context |
|
||
| `GET` | `/search?q=...` | Full-text search |
|
||
| `GET` | `/audit?limit=N` | Audit log |
|
||
| `GET` | `/.well-known/oauth-authorization-server` | OAuth discovery |
|
||
| `GET` | `/.well-known/oauth-protected-resource` | Resource metadata |
|
||
| `POST` | `/oauth/register` | Dynamic Client Registration |
|
||
| `GET/POST` | `/oauth/authorize` | Authorization endpoint |
|
||
| `POST` | `/oauth/token` | Token endpoint |
|
||
| `POST/GET/DELETE` | `/readonly/mcp` | Read-only MCP (Streamable HTTP) |
|
||
| `POST/GET/DELETE` | `/write/mcp` | Write MCP (Streamable HTTP) |
|
||
| `POST/GET/DELETE` | `/mcp` | Internal full MCP (API key only) |
|
||
|
||
## Backups
|
||
|
||
### PostgreSQL
|
||
|
||
```bash
|
||
# Backup
|
||
docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup_$(date +%Y%m%d).sql
|
||
|
||
# Restore
|
||
cat backup_YYYYMMDD.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd
|
||
```
|
||
|
||
### Snapshots
|
||
|
||
CTXD automatically takes point-in-time snapshots before each context update. Snapshots are stored as files in `/data/snapshots/<project_id>/` and rotated (min 5, max 25 per project).
|
||
|
||
## Migrating from SQLite to PostgreSQL
|
||
|
||
If you started with SQLite and want to move to PostgreSQL:
|
||
|
||
```bash
|
||
# 1. Start PostgreSQL
|
||
docker compose up -d postgres
|
||
|
||
# 2. Run the migration script (reads from /data/ctxd.db, writes to DATABASE_URL)
|
||
docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
|
||
|
||
# 3. Set DATABASE_URL in .env and restart
|
||
docker compose up -d dossier
|
||
```
|
||
|
||
The migration handles all tables, rebuilds the FTS index, and skips orphaned rows with FK violations.
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
/mnt/ai-storage/projects/ctxd/
|
||
├── .env # Production environment (gitignored)
|
||
├── .env.example # Template (committed)
|
||
├── .gitignore
|
||
├── SKILL.md # LLM client guide (canonical source)
|
||
├── README.md # This file
|
||
├── data/ # Runtime data (gitignored)
|
||
│ ├── ctxd.yaml # Fallback config (env vars take precedence)
|
||
│ ├── ctxd.db # SQLite DB (if no DATABASE_URL)
|
||
│ ├── pg/ # PostgreSQL data volume
|
||
│ ├── oauth_state.json # OAuth clients, codes, tokens
|
||
│ ├── web_sessions.json # Web UI sessions
|
||
│ └── snapshots/ # Point-in-time backups
|
||
└── app/ # Application source
|
||
├── docker-compose.yml
|
||
├── Dockerfile
|
||
├── pyproject.toml
|
||
├── .env # Same as root .env (symlinked or copied)
|
||
└── src/ctxd/
|
||
├── __init__.py
|
||
├── __main__.py # CLI/daemon entry point
|
||
├── config.py # Env-driven config with yaml fallback
|
||
├── db.py # Database layer (PostgreSQL + SQLite)
|
||
├── schema.sql # PostgreSQL schema
|
||
├── schema_sqlite.sql # SQLite schema (fallback)
|
||
├── server.py # ASGI app: HTTP + MCP + OAuth
|
||
├── cli.py # CLI commands
|
||
├── ui.html # Web UI dashboard
|
||
├── landing.html # Public landing page
|
||
├── auth_password.py # PBKDF2 password hashing
|
||
└── migrate_sqlite_to_pg.py # One-time migration script
|
||
```
|
||
|
||
## Development
|
||
|
||
### Local Development (SQLite, no Docker)
|
||
|
||
```bash
|
||
cd app
|
||
pip install -e ".[mcp]"
|
||
export CTXD_HOME=./dev-data
|
||
python -m ctxd init
|
||
python -m ctxd
|
||
# → http://localhost:9091
|
||
```
|
||
|
||
### Rebuilding After Code Changes
|
||
|
||
```bash
|
||
cd app
|
||
docker compose build
|
||
docker compose up -d --no-build
|
||
# Verify:
|
||
curl http://localhost:9091/status
|
||
docker logs ctxd --tail 20
|
||
```
|
||
|
||
### Key Conventions
|
||
|
||
- **Metadata headers** are dynamically generated on read, never stored in the DB
|
||
- **File paths** are normalized to uppercase with `.MD` extension
|
||
- **`CONTEXT.MD`** is the minimum required file — cannot be deleted from any project
|
||
- **Version checking** uses `base_version` parameter — mismatches return `409 conflict`
|
||
- **Audit log** is append-only at the application layer (every operation is logged)
|
||
- **OAuth state** (`oauth_state.json`) and **web sessions** (`web_sessions.json`) are file-based, not in PostgreSQL
|
||
|
||
## Troubleshooting
|
||
|
||
### Login fails with "invalid credentials"
|
||
|
||
```bash
|
||
# Reset admin password
|
||
docker exec ctxd dossier user-set-password admin -p "new-password"
|
||
```
|
||
|
||
If the password contains special characters, use the quoting that matches:
|
||
- `'` in password → use double quotes (`"`)
|
||
- `$`, `` ` ``, `!` in password → use single quotes (`'`)
|
||
|
||
### Login works on LAN but not on public host
|
||
|
||
The public host (`https://ctxd.yourdomain.com`) requires Traefik to route `/auth/login` and `/auth/me`. Check your Traefik router rule includes all paths (use `!Path(`/mcp`)` to block only the internal MCP).
|
||
|
||
### MCP connection fails
|
||
|
||
1. Check OAuth discovery: `curl https://ctxd.yourdomain.com/.well-known/oauth-authorization-server`
|
||
2. Check MCP endpoint: `curl -o /dev/null -w '%{http_code}' https://ctxd.yourdomain.com/readonly/mcp` → should be `401` (not `404`)
|
||
3. If `404`: Traefik isn't routing `/readonly/mcp` — update the router rule
|
||
4. If `401`: auth is working — check OAuth token scope and expiry
|
||
|
||
### PostgreSQL connection fails
|
||
|
||
```bash
|
||
# Check PG is running
|
||
docker compose ps postgres
|
||
|
||
# Check connection
|
||
docker exec ctxd python3 -c "
|
||
import os
|
||
import psycopg
|
||
conn = psycopg.connect(os.environ['DATABASE_URL'])
|
||
print('Connected:', conn.info.server_version)
|
||
"
|
||
|
||
# If password mismatch (PG data volume initialized with different password):
|
||
docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'new-password'"
|
||
# Then update .env with the new password
|
||
```
|
||
|
||
### Container keeps restarting / public site 502
|
||
|
||
```bash
|
||
docker compose ps -a
|
||
docker logs ctxd --tail 40
|
||
```
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---------|--------|-----|
|
||
| `failed to resolve host 'postgres'` | `ctxd-postgres` not running (often after `--no-deps ctxd`) | `cd app && docker compose up -d postgres ctxd` or `./scripts/deploy.sh` |
|
||
| `Restarting (1)` on `ctxd` only | Same — app up without DB on compose network | Start postgres first; wait for **(healthy)** |
|
||
| Cloudflare **502** on public URL | Traefik/backend has no healthy upstream on `:9091` | Fix local `curl http://127.0.0.1:9091/status` first |
|
||
|
||
The container **entrypoint waits up to 120s** for PostgreSQL when `DATABASE_URL` is set (`CTXD_PG_WAIT_SECONDS` to override). If postgres never appears, logs print an explicit message instead of an immediate opaque crash.
|
||
|
||
Common other causes:
|
||
- `DATABASE_URL` password doesn't match what PG was initialized with
|
||
- `OAUTH_ENABLED=true` but `OAUTH_ISSUER` is empty
|
||
- Missing `CTXD_API_KEY` when `CTXD_AUTH_ENABLED=true`
|
||
|
||
## License
|
||
|
||
MIT |