Compare commits

..

22 Commits

Author SHA1 Message Date
overseer 3ef4f3e707 chore: add CLAUDE.md, stop tracking egg-info build artifacts
- Add CLAUDE.md (Claude Code orientation for the repo).
- Remove app/src/ctxd.egg-info/* from version control and gitignore
  *.egg-info/ — it is regenerated by `pip install -e` and only dirties
  the working tree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 21:11:54 +00:00
overseer bbb0215c98 docs: update repo domain to cubecraftlabs.com
Rename code.cubecraftcreations.com -> code.cubecraftlabs.com in the
deployment guide and article links.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 21:10:15 +00:00
overseer bdc984e5ff fix: stop duplicate-key 500 cascade on shared PG connection
The REST/Web-UI HTTPServer shares one long-lived PG connection across all
requests. Any statement that raised mid-request (e.g. a UniqueViolation from
a desynced SERIAL sequence) aborted the transaction; the global handler
returned 500 without rolling back, so every subsequent request failed with
InFailedSqlTransaction until restart — surfacing as "duplicate keys cause
500s" and "500 immediately after login".

- server.py: global handler now always rolls back the shared connection on
  error and maps constraint violations to 409 (was 500/400). This is the one
  funnel that guarantees the connection is never left aborted.
- db.py: add is_integrity_error() — dual-backend (psycopg + sqlite3)
  constraint-violation classifier; replaces the fragile `"UNIQUE" in msg`
  string match that never matched Postgres' error text.
- Remove make_write_mcp_server: a never-run duplicate of the MCP write tools
  that had bit-rotted (wrong file_update arg order + FK-violating hardcoded
  actor). Live writes go through oauth_mcp_app, which is correct.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 21:10:12 +00:00
overseer 80635ce011 fix: drop NOT NULL on user_id FK columns for ON DELETE SET NULL
audit_log.user_id, change_requests.submitted_by, reviews.reviewer_id
were NOT NULL but had ON DELETE SET NULL — PG can't set a NOT NULL
column to NULL on delete, so user deletion failed. Migration now drops
NOT NULL before altering the FK constraint.
2026-06-25 14:47:37 +00:00
overseer 9bb89ee62f feat: allow user deletion — FK constraints ON DELETE SET NULL
- schema.sql: all user_id FKs now ON DELETE SET NULL (was RESTRICT)
- migrate_user_fk_set_null.py: drop+readd constraints on existing DBs
- entrypoint.sh: runs migration automatically on startup (non-fatal)
- db.py: rollback moved before FK check (cleanup)

Deleting a user now nullifies their references in audit_log,
project_context, context_files, change_requests, reviews instead of
blocking with a 409.
2026-06-25 14:43:58 +00:00
overseer bc43e9a8d1 fix: rollback aborted PG transaction on user delete FK violation
PostgreSQL enters 'current transaction is aborted' state after a
ForeignKeyViolation. Without rollback(), every subsequent query on the
shared connection fails with 'commands ignored until end of transaction
block'. Now both db.py and server.py rollback on the error path.
2026-06-25 14:29:00 +00:00
overseer d2c6906c4f ui: show activate OR inactivate based on user state (not both) 2026-06-25 14:24:37 +00:00
overseer 570b7d1dba ui: move activate/inactivate/delete to user list table actions column
- New actions column (right of state) with inline buttons per row
- Manage form keeps save user + clear only (no duplicate action buttons)
- Themed to match admin dialog: JetBrains Mono, uppercase, square borders
- Delete uses danger styling
2026-06-25 14:19:11 +00:00
overseer 451732c867 ui: OAuth client list scopes button matches admin theme 2026-06-25 14:07:32 +00:00
overseer 9e85c1b8ec ops: prevent 502 from ctxd without postgres
- entrypoint: wait for DATABASE_URL (CTXD_PG_WAIT_SECONDS) with clear fatal message
- scripts/deploy.sh: postgres healthy then force-recreate ctxd
- compose + README: ban --no-deps ctxd as default; 502 troubleshooting
2026-06-25 13:59:51 +00:00
overseer 59609f93c4 fix: MCP OAuth discovery for ChatGPT (RFC 9728 /mcp PRM, WWW-Authenticate)
- scopes_supported on protected-resource metadata
- /.well-known/oauth-protected-resource/mcp (path-prefix match)
- 401 on /mcp points resource_metadata at PRM URL; advertise scopes
- Prefer client_secret_basic in AS metadata (ChatGPT connector quirk)
- README: does not implement OAuth / 502 / 404 troubleshooting
2026-06-25 13:53:31 +00:00
overseer 07cf223d16 docs: ChatGPT MCP connector checklist in README 2026-06-25 13:46:22 +00:00
overseer ce1c0a175f docs: admin OAuth scope caps in SKILL.md 2026-06-25 13:13:31 +00:00
overseer 1c9d8f7648 feat: OAuth client scope assignment in admin panel
- Create client: ctxd.read / ctxd.write checkboxes
- Client list: show scopes, edit via PATCH /oauth/clients/:id
- Authorize grants intersection of client allowed scopes and request
- CLI oauth-client-create --scope; DCR default ctxd.read ctxd.write
2026-06-25 13:13:25 +00:00
overseer 87f02eb4d1 test: hit /mcp in tools/list fallback branch 2026-06-25 12:46:05 +00:00
overseer 289c6b9300 refactor: public OAuth MCP at /mcp (readonly path is alias)
- OAuth read+write on /mcp; API key on /mcp still full internal tools (LAN)
- /readonly/mcp and /oauth/mcp remain OAuth aliases
- OAuth metadata and connector_url point to /mcp
- README + Traefik template: route Host without blocking /mcp
2026-06-25 12:45:59 +00:00
overseer 12b60ee8c7 feat: unified OAuth MCP connector (read+write on /readonly/mcp)
Scope-gated tools on one Streamable HTTP URL; /oauth/mcp alias.
Reconnect Claude once with ctxd.read ctxd.write.
2026-06-25 12:44:01 +00:00
overseer e3567f649f fix: OAuth write MCP via SSE; require container recreate after build
- Route public write at GET /write/sse and POST /write/messages (ctxd.write)
- Always require write token on /write/messages (was optional)
- Remove debug tracing; document SSE write surface in SKILL.md
- Add scripts/test_write_mcp.py for local OAuth write smoke test
2026-06-25 12:38:50 +00:00
overseer fe63ad350e fix: reset SERIAL sequences after SQLite migration to prevent UniqueViolation
Migration script now resets SERIAL sequences (audit_log, context_files, etc.)
to max(existing_id) after copying rows. Without this, the sequence is still at
its post-schema-creation value and every INSERT hits a duplicate key error.
2026-06-25 10:50:35 +00:00
overseer 364c7795d4 fix: entrypoint.sh set -e crash, __main__.py PostgreSQL check, cli.py seed placeholders
- entrypoint.sh: use NEEDS_INIT variable instead of exit codes (set -e was killing the script)
- __main__.py: skip db_path.exists() check when using PostgreSQL
- cli.py: use dual-backend placeholders for _seed_context (was using SQLite ? syntax)
- cli.py: use 'admin' instead of 'system' for seed updated_by (FK constraint)
2026-06-25 00:20:55 +00:00
overseer b9f911994d refactor: rename container from dossier to ctxd
- docker-compose.yml: service name dossier → ctxd, container_name dossier → ctxd
- README.md, SKILL.md, LLM.txt: all docker exec/logs references updated
- Hermes skill files: all references updated
2026-06-24 23:19:08 +00:00
overseer b91d03a6cd docs: add LLM.txt for automated installation and deployment assistance 2026-06-24 22:55:05 +00:00
25 changed files with 1680 additions and 295 deletions
+3
View File
@@ -20,3 +20,6 @@ app/.env
# Python cache # Python cache
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
# Build artifacts (regenerated by `pip install -e`)
*.egg-info/
+108
View File
@@ -0,0 +1,108 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What CTXD is
A single-process daemon that stores per-project "context dossiers" (multiple `.MD` files:
`CONTEXT.MD`, `DECISIONS.MD`, `RUNBOOKS.MD`, `PROMPTS.MD`, `GLOSSARY.MD`) and serves them to
LLM harnesses (Claude, ChatGPT, Hermes, Codex, Cursor) over **MCP via Streamable HTTP**, plus a
web UI and REST API. `CONTEXT.MD` can be synced to a repo as `AGENTS.md` with symlinks
(`CLAUDE.md`, `.cursorrules`, `CODEX.md`). The full user/operator guide is `README.md`; this file
is the orientation for editing the code.
All source lives in `app/src/ctxd/`. Run commands from `app/`.
## Architecture (the parts that need multiple files to grasp)
- **One ASGI app multiplexes three protocols.** `server.py``CombinedApp` (~line 1455) dispatches
every request to: REST + Web UI, the OAuth 2.0 authorization server, and the MCP endpoints. There
is no separate service per surface — "MCP is 502" and "web UI is 502" are the same process being
down. `serve_sync(cfg)` boots it under uvicorn.
- **Two MCP transports.** Streamable HTTP is the public one (paths in `MCP_STREAMABLE_PATHS` =
`/mcp`, `/readonly/mcp`, `/oauth/mcp`, all served by `CombinedApp`). `mcp_stdio.py` is a *separate*
stdin/stdout JSON-RPC server Hermes spawns directly — keep tool definitions in sync between the two
when adding tools.
- **MCP tools are built in two functions, gated by scope.** `make_mcp_server(cfg, readonly, oauth_scoped)`
exposes read tools (+ everything when API-key/LAN); `make_write_mcp_server(cfg)` exposes the write
set (`update_file`, `set_project_tags`, `sync_to_project`). OAuth `ctxd.read` vs `ctxd.write` scopes
decide what a token sees. To add an MCP tool, edit the `list_tools`/`call_tool` handlers inside these
builders.
- **`db.py` is plain functions over a `conn`, no ORM, dual-backend.** PostgreSQL is primary; SQLite
(`$CTXD_HOME/ctxd.db`) is the fallback when `DATABASE_URL` is empty (`cfg.use_postgres`). The same
query strings run on both via `_is_pg(conn)` and the placeholder helper `_ph(conn, n)`. **`schema.sql`
(PG) and `schema_sqlite.sql` must be kept in lockstep** — a table added to one must be added to the other.
- **OAuth state and web sessions are file-based JSON, not in the DB.** `OAuthStore` (`oauth_state.json`)
and `WebSessionStore` (`web_sessions.json`) live in `$CTXD_HOME` (`/data` in the container). They
survive DB swaps but are *not* covered by `pg_dump` — back them up separately.
- **Metadata headers are computed, never stored.** `build_metadata_header()` prepends the header on
read; `strip_metadata_header()` removes it before persisting. Don't store headers in the DB.
- **File paths are normalized** to uppercase with `.MD` (`normalize_file_path`). `CONTEXT.MD` is the
minimum file and cannot be deleted; in the `ctxd-docs` project, `CONTEXT.MD` and `LLM-CLIENT.MD` are
also locked against update/delete.
- **Writes are version-checked.** `file_update`/`context_update` take `base_version`; a mismatch is a
`409 conflict`. Every mutating op also writes an append-only `audit_log` row and may take a rotating
snapshot (`SNAPSHOT_MIN_KEEP`/`MAX_KEEP`).
### Latent / not wired up
`schema.sql` and `db.py` define a full collaborative-review flow — `user_workspaces``workspace_files`
`change_requests``reviews`, with `workspace_fork`, `workspace_submit`, `change_request_approve`,
etc. As of now this is **DB-layer only**: it is exposed through neither `server.py` (REST/MCP) nor
`cli.py`. Treat it as scaffolding, not a live feature.
## Entry points
`pyproject.toml` defines two console scripts (and `python -m ctxd` dispatches the same way via
`__main__.py`, choosing CLI vs daemon by the first arg):
- `dossier <command>` → CLI (`cli.py`, `cli_entry`). In production these run inside the container:
`docker exec ctxd dossier <command>`.
- `ctxd` → starts the daemon (`daemon_entry``serve_sync`).
## Build / run / deploy
Everything runs from `app/`. There is **no test runner or linter configured**`scripts/test_*.py`
are standalone MCP smoke scripts, not a suite.
```bash
# Local dev (SQLite, no Docker)
cd app
pip install -e ".[mcp]"
export CTXD_HOME=./dev-data
python -m ctxd init # initialize DB
python -m ctxd # serve → http://localhost:9091
# Production deploy (Docker + bundled PostgreSQL 16)
./scripts/deploy.sh # builds ctxd, starts postgres, waits for healthy, recreates ctxd, smoke-tests /status
# MCP smoke tests against a running server
python scripts/test_unified_mcp.py
python scripts/test_write_mcp.py
# Health
curl http://localhost:9091/status # → {"status":"ok", ...}
```
**Deploy gotcha (the #1 source of public 502s):** `ctxd` depends on the `postgres` container being up
first. Always use `./scripts/deploy.sh` (or `docker compose up -d`, which starts both). **Never**
`docker compose up -d --no-deps ctxd` and **never** `docker restart ctxd` after a code change — the
former crash-loops without the DB, the latter runs the old image. After editing code you must rebuild
(`docker compose build ctxd`) before recreating.
## Config
All config is env-driven through `config.py` (`CtxConfig`), precedence **env var > `data/ctxd.yaml` >
default**. Key switches: `DATABASE_URL` (empty ⇒ SQLite fallback), `CTXD_HOME` (`/data` in container),
`CTXD_AUTH_ENABLED` + `CTXD_API_KEY` (shared key for LAN/Hermes = full MCP tools), `OAUTH_ENABLED` +
`OAUTH_ISSUER` (must be set together or the app won't start cleanly). The full table is in `README.md`.
## Conventions when changing code
- Adding a context-data operation: write the `conn`-taking function in `db.py` (handle both backends
via `_is_pg`/`_ph`), then expose it in the relevant surface(s) — `server.py` for REST/MCP, `cli.py`
for CLI, and `mcp_stdio.py` if Hermes needs it.
- Adding a column/table: edit **both** `schema.sql` and `schema_sqlite.sql`; for live PG instances add
a guarded migration (see `_migrate_pg` in `db.py` and the standalone `migrate_*.py` scripts).
- Don't persist metadata headers; don't bypass `base_version` checking on writes; keep every mutation
audit-logged.
+251
View File
@@ -0,0 +1,251 @@
# CTXD — Context Dossier
# LLM Installation & Deployment Guide
# Feed this file to an LLM to assist with setup and deployment.
## What is CTXD?
CTXD (Context Dossier) is a single source of truth for multi-harness project context. It serves one canonical AGENTS.md per project to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP. It runs as two Docker containers: a PostgreSQL 16 database and a Python ASGI daemon.
## Prerequisites
- Docker 24+ and Docker Compose v2+
- A reverse proxy with TLS (Traefik recommended; Caddy or nginx also work)
- Linux host with ~1GB free RAM
- (Optional) Existing PostgreSQL 14+ instance if not using the bundled one
## Quick Deployment
1. Clone the repository:
git clone ssh://git@code.cubecraftlabs.com:2288/CubeCraft-Creations/CTXD.git
cd CTXD/app
2. Copy the environment template:
cp .env.example .env
3. Generate secrets and fill in .env:
python3 -c "import secrets; print(secrets.token_urlsafe(32))" # Use for CTXD_API_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))" # Use for OAUTH_APPROVAL_KEY
4. Edit .env with your values. Required variables:
- DATABASE_URL — PostgreSQL connection string
- POSTGRES_PASSWORD — Password for the bundled PG container
- CTXD_API_KEY — Shared key for internal MCP and HTTP auth
- OAUTH_ENABLED=true
- OAUTH_ISSUER=https://your-domain.com
- OAUTH_APPROVAL_KEY — Key for approving OAuth authorizations
5. Build and start:
docker compose up -d
6. Wait for PostgreSQL to be healthy and the daemon to start:
docker compose ps
# Both ctxd-postgres and dossier should show "healthy" / "running"
7. Set the admin password:
docker exec ctxd dossier user-set-password admin -p "your-admin-password"
8. Verify:
curl http://localhost:9091/status
# Expected: {"status": "ok", "db": "/data/ctxd.db"}
9. Access the Web UI:
http://<server-ip>:9091/
Sign in with admin / your-admin-password
## Environment Variables Reference
All config is driven by environment variables in .env. Precedence: env var > ctxd.yaml > defaults.
### Database
- DATABASE_URL — PostgreSQL connection string. If empty, falls back to SQLite.
- POSTGRES_USER — PostgreSQL user for bundled container (default: ctxd)
- POSTGRES_PASSWORD — PostgreSQL password for bundled container
- POSTGRES_DB — PostgreSQL database name (default: ctxd)
### Server
- CTXD_HOST — Bind address (default: 0.0.0.0)
- CTXD_PORT — Listen port (default: 9091)
- CTXD_HOME — Data directory inside container (default: /data)
- LOG_LEVEL — Uvicorn log level: debug, info, warning, error (default: info)
### Auth
- CTXD_AUTH_ENABLED — Enable authentication globally (default: false)
- CTXD_API_KEY — Shared API key for Hermes/internal MCP and HTTP auth
- CTXD_EXTERNAL_READONLY_KEY — Legacy ?key= on read-only MCP (migration only)
### OAuth
- OAUTH_ENABLED — Enable OAuth authorization server (default: false)
- OAUTH_ISSUER — Public URL used in OAuth discovery metadata
- OAUTH_APPROVAL_KEY — Fallback key for /oauth/authorize approval
- OAUTH_APPROVAL_USER_ID — User ID for attributing approvals (default: admin)
- OAUTH_ACCESS_TOKEN_TTL — Access token lifetime in seconds (default: 3600)
- OAUTH_REFRESH_TOKEN_TTL — Refresh token lifetime in seconds (default: 2592000)
### Web Sessions
- WEB_SESSION_TTL — Session cookie lifetime in seconds (default: 604800)
### Snapshots
- SNAPSHOT_MIN_KEEP — Minimum snapshots per project (default: 5)
- SNAPSHOT_MAX_KEEP — Maximum snapshots before rotation (default: 25)
## MCP Endpoints
CTXD exposes three MCP surfaces via Streamable HTTP:
1. /readonly/mcp — Read-only tools (OAuth ctxd.read or API key)
Tools: list_projects, get_project_context, search_context, get_project_tags, list_files, get_file, get_client_guide
2. /write/mcp — Write tools (OAuth ctxd.write only)
Tools: update_file, set_project_tags, sync_to_project, get_client_guide
3. /mcp — Internal full MCP (shared API key only, not exposed publicly)
All read + write tools plus auto_generate_tags and get_user_profile
## Connecting LLM Clients
### Claude Desktop
Connector URL: https://your-domain.com/readonly/mcp
Claude auto-discovers OAuth metadata and registers via DCR.
For write access, use: https://your-domain.com/write/mcp
### Hermes Agent
Edit ~/.hermes/config.yaml:
mcp_servers:
dossier:
url: http://<server-ip>:9091/mcp
timeout: 30
headers:
Authorization: "Bearer your-api-key"
Restart Hermes after changing config.
### Other MCP Clients
1. Register an OAuth client: POST /oauth/register with your redirect URI
2. Authorize: GET /oauth/authorize (admin must approve)
3. Exchange code: POST /oauth/token
4. Connect to /readonly/mcp or /write/mcp with Bearer token
## Traefik Configuration
Route everything except the internal MCP endpoint:
rule: Host(`ctxd.yourdomain.com`) && !Path(`/mcp`)
This exposes: landing page, dashboard, REST API, OAuth, read-only MCP, write MCP.
Only /mcp (internal, API key) is blocked.
## CLI Commands
All commands run inside the container:
docker exec ctxd dossier <command>
Commands:
init Initialize database
project-create <id> [--display-name N] Create a project
project-list List projects
read <project_id> Print context to stdout
edit <project_id> Open in $EDITOR
file-list <project_id> List context files
file-read <project_id> <file_path> Read a single file
sync <project_id> [path] Sync AGENTS.md to project root
search "query" Full-text search
audit [--limit N] Show audit log
user-list List users
user-create <id> --display-name "Name" [--password "pw"]
user-set-password <id> -p "password"
oauth-client-create [-n "Name"] [--redirect-uri URI]
oauth-client-list List OAuth clients
oauth-client-revoke <client_id> Revoke client and invalidate tokens
import-vault <path> Import from Obsidian vault
## Admin UI
Sign in as admin, click "admin" in the masthead. Three tabs:
1. oauth clients — Create and revoke OAuth clients
2. users — List, create, edit, activate, inactivate, delete users
3. projects — List and delete projects (typed-name confirmation required)
## Project Files
Each project has multiple context files:
- CONTEXT.MD — Canonical overview (synced as AGENTS.md, cannot be deleted)
- DECISIONS.MD — Architecture decisions and rationale
- RUNBOOKS.MD — Deploy, troubleshoot, operate procedures
- PROMPTS.MD — Project-specific prompts for different harnesses
- GLOSSARY.MD — Project-specific terms and acronyms
The compiled view (get_project_context) concatenates all files with metadata header.
## Locked Files
- CONTEXT.MD — Cannot be deleted from any project (minimum required file)
- CONTEXT.MD and LLM-CLIENT.MD — Fully locked in the ctxd-docs project (cannot update or delete)
## Backups
PostgreSQL backup:
docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup.sql
PostgreSQL restore:
cat backup.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd
Snapshots are automatic (before each context update, rotated min 5 / max 25 per project).
## Migrating from SQLite to PostgreSQL
If you started with SQLite:
1. Start PostgreSQL: docker compose up -d postgres
2. Run migration: docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
3. Set DATABASE_URL in .env and restart: docker compose up -d dossier
## Using an External PostgreSQL
1. Create database and user on your external PG instance
2. Set DATABASE_URL in .env to point to it
3. Start only the app: docker compose up -d dossier
(or scale postgres to 0: docker compose up -d --scale postgres=0)
## Troubleshooting
Login fails: Reset password with "docker exec ctxd dossier user-set-password admin -p newpass"
Use double quotes if password contains single quotes. Use single quotes for $ ` ! characters.
MCP 404: Traefik is not routing /readonly/mcp. Check router rule includes it.
MCP 401: Auth is working. Check OAuth token scope and expiry.
PostgreSQL connection fails: Check DATABASE_URL password matches POSTGRES_PASSWORD.
If PG data volume was initialized with a different password, reset it:
docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'newpass'"
Then update .env.
Container keeps restarting: Check "docker logs ctxd --tail 30".
Common causes: wrong PG password, OAUTH_ENABLED=true but OAUTH_ISSUER empty, missing CTXD_API_KEY.
## File Structure
CTXD/
├── .env Production environment (gitignored, contains secrets)
├── .env.example Template with all documented variables
├── .gitignore
├── README.md Full documentation
├── SKILL.md LLM client guide (canonical source for LLM-CLIENT.MD)
├── data/ Runtime data (gitignored)
│ ├── ctxd.yaml Fallback config (env vars take precedence)
│ ├── oauth_state.json OAuth clients, codes, tokens
│ ├── web_sessions.json Web UI sessions
│ ├── snapshots/ Point-in-time context backups
│ └── pg/ PostgreSQL data volume
└── app/ Application source
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
└── src/ctxd/
├── config.py Env-driven config
├── db.py Database layer (PostgreSQL + SQLite)
├── schema.sql PostgreSQL schema
├── schema_sqlite.sql SQLite schema fallback
├── server.py ASGI: HTTP + MCP + OAuth
├── cli.py CLI commands
├── ui.html Web UI dashboard
├── landing.html Public landing page
├── auth_password.py PBKDF2 password hashing
└── migrate_sqlite_to_pg.py Migration script
+103 -39
View File
@@ -22,17 +22,17 @@ CTXD provides:
## Architecture ## Architecture
``` ```
Context Dossier (container: dossier, 0.0.0.0:9091) Context Dossier (container: ctxd, 0.0.0.0:9091)
├── PostgreSQL 16 (container: ctxd-postgres) # Primary DB ├── PostgreSQL 16 (container: ctxd-postgres) # Primary DB
├── /data # Config, OAuth state, web sessions ├── /data # Config, OAuth state, web sessions
│ ├── ctxd.yaml # Fallback config (env vars take precedence) │ ├── ctxd.yaml # Fallback config (env vars take precedence)
│ ├── oauth_state.json # OAuth clients, codes, tokens │ ├── oauth_state.json # OAuth clients, codes, tokens
│ ├── web_sessions.json # Per-user web UI sessions │ ├── web_sessions.json # Per-user web UI sessions
│ └── snapshots/ # Point-in-time context backups │ └── snapshots/ # Point-in-time context backups
├── Streamable HTTP MCP endpoints: ├── Streamable HTTP MCP:
│ ├── /readonly/mcp (OAuth ctxd.read) # Read-only tools │ ├── /mcp (OAuth ctxd.read + ctxd.write; API key on LAN = full tools)
│ ├── /write/mcp (OAuth ctxd.write) # Write tools │ ├── /readonly/mcp (alias → same OAuth behavior)
│ └── /mcp (shared API key) # Internal full MCP │ └── /oauth/mcp (alias)
├── OAuth Authorization Server: ├── OAuth Authorization Server:
│ ├── /.well-known/oauth-authorization-server # Discovery │ ├── /.well-known/oauth-authorization-server # Discovery
│ ├── /.well-known/oauth-protected-resource # Resource metadata │ ├── /.well-known/oauth-protected-resource # Resource metadata
@@ -84,12 +84,19 @@ OAUTH_APPROVAL_KEY=your-approval-key # Generate: python3 -c "import secrets;
### 2. Build and start ### 2. Build and start
```bash ```bash
# Recommended: build + postgres + recreate ctxd (avoids 502 when PG was never started)
chmod +x scripts/deploy.sh
./scripts/deploy.sh
# Or manual (always include postgres when DATABASE_URL uses host "postgres"):
docker compose up -d docker compose up -d
``` ```
This starts: This starts:
- `ctxd-postgres` — PostgreSQL 16 (Alpine) - `ctxd-postgres` — PostgreSQL 16 (Alpine)
- `dossier` — CTXD daemon (web UI + MCP + OAuth + REST API) - `ctxd` — CTXD daemon (web UI + MCP + OAuth + REST API)
**After code changes**, use `./scripts/deploy.sh` or `docker compose up -d` — not `docker restart ctxd` alone (old image) and **not** `docker compose up -d --no-deps ctxd` (skips postgres → crash loop → public **502**).
### 3. Verify ### 3. Verify
@@ -105,7 +112,7 @@ curl http://localhost:9091/projects -H "Authorization: Bearer your-api-key"
### 4. Set admin password ### 4. Set admin password
```bash ```bash
docker exec dossier dossier user-set-password admin -p "your-admin-password" docker exec ctxd dossier user-set-password admin -p "your-admin-password"
``` ```
> **Shell quoting:** Use double quotes (`"`) if the password contains single quotes (`'`). Use single quotes (`'`) if it contains `$`, backticks, or `!`. The failure mode is shell expansion, not PBKDF2. > **Shell quoting:** Use double quotes (`"`) if the password contains single quotes (`'`). Use single quotes (`'`) if it contains `$`, backticks, or `!`. The failure mode is shell expansion, not PBKDF2.
@@ -148,6 +155,9 @@ All config is driven by environment variables. A `ctxd.yaml` file in `/data` can
| `OAUTH_APPROVAL_USER_ID` | `admin` | Which user ID to attribute OAuth approvals to | | `OAUTH_APPROVAL_USER_ID` | `admin` | Which user ID to attribute OAuth approvals to |
| `OAUTH_ACCESS_TOKEN_TTL` | `3600` | Access token lifetime in seconds | | `OAUTH_ACCESS_TOKEN_TTL` | `3600` | Access token lifetime in seconds |
| `OAUTH_REFRESH_TOKEN_TTL` | `2592000` | Refresh token lifetime in seconds (30 days) | | `OAUTH_REFRESH_TOKEN_TTL` | `2592000` | Refresh token lifetime in seconds (30 days) |
| **PostgreSQL (container)** | | |
| `CTXD_PG_WAIT_SECONDS` | `120` | Entrypoint: max wait for DB before exit (when `DATABASE_URL` set) |
| `CTXD_PG_WAIT_INTERVAL` | `2` | Seconds between connection attempts |
| **Web Sessions** | | | | **Web Sessions** | | |
| `WEB_SESSION_TTL` | `604800` | Session cookie lifetime in seconds (7 days) | | `WEB_SESSION_TTL` | `604800` | Session cookie lifetime in seconds (7 days) |
| **Snapshots** | | | | **Snapshots** | | |
@@ -160,13 +170,9 @@ To use an external PostgreSQL instead of the bundled container:
1. Create a database and user on your external PG instance 1. Create a database and user on your external PG instance
2. Set `DATABASE_URL` in `.env` to point to it 2. Set `DATABASE_URL` in `.env` to point to it
3. Start only the dossier service: `docker compose up -d dossier` 3. Start only the app (no bundled postgres): `docker compose up -d --scale postgres=0 ctxd`
The bundled `ctxd-postgres` container will start but is unused. To prevent it from starting, run it with zero scale: Ensure `DATABASE_URL` points at your external host (not `postgres`). The entrypoint skips the compose-network wait when the URL is reachable.
```bash
docker compose up -d --scale postgres=0
```
### Fallback to SQLite ### Fallback to SQLite
@@ -174,22 +180,73 @@ If `DATABASE_URL` is empty or not set, CTXD falls back to SQLite at `$CTXD_HOME/
## MCP Surfaces ## MCP Surfaces
CTXD exposes three MCP endpoints via Streamable HTTP: CTXD exposes MCP via Streamable HTTP on **`/mcp`** (single public connector):
| Endpoint | Auth | Scope | Tools | | Endpoint | Auth | Scope | Tools |
|----------|------|-------|-------| |----------|------|-------|-------|
| `/readonly/mcp` | OAuth bearer or API key | `ctxd.read` | `list_projects`, `get_project_context`, `search_context`, `get_project_tags`, `list_files`, `get_file`, `get_client_guide` | | `/mcp` | OAuth bearer | `ctxd.read` / `ctxd.write` | Scope-gated read + write tools |
| `/write/mcp` | OAuth bearer | `ctxd.write` | `update_file`, `set_project_tags`, `sync_to_project`, `get_client_guide` | | `/mcp` | Shared API key (LAN/Hermes) | *(full)* | All tools including `get_user_profile`, `auto_generate_tags` |
| `/mcp` | Shared API key | *(full)* | All read + write tools + `sync_to_project`, `auto_generate_tags`, `get_user_profile` | | `/readonly/mcp`, `/oauth/mcp` | OAuth (aliases) | same as `/mcp` | Backward-compatible URLs |
### Connecting an LLM Client ### Connecting an LLM Client
**Claude Desktop / Claude Web:** **Claude Desktop / Claude Web:**
``` ```
Connector URL: https://ctxd.yourdomain.com/readonly/mcp Connector URL: https://ctxd.yourdomain.com/mcp
``` ```
Claude auto-discovers OAuth metadata and registers via DCR. Request `scope=ctxd.read ctxd.write` for write access. Claude auto-discovers OAuth metadata and registers via DCR. Request `scope=ctxd.read ctxd.write` for write access.
**ChatGPT (MCP connector — recommended; no Custom GPT required):**
ChatGPT can attach a **remote MCP server** in **Developer mode** (Plus/Pro and higher tiers for custom connectors). Use the same public URL as Claude — OAuth on `/mcp`, not REST Actions or a shared API key.
1. **Server** — deploy current CTXD and expose the public host (Traefik must route `/mcp` and `/oauth/*`; do not block `/mcp`):
```bash
cd app
./scripts/deploy.sh
```
Smoke: `curl -sS https://ctxd.yourdomain.com/.well-known/oauth-authorization-server | head`
2. **ChatGPT** — Settings → **Connectors** → enable **Developer mode** → **Add connector** (MCP / custom remote) → **Server URL:**
```
https://ctxd.yourdomain.com/mcp
```
Start OAuth and copy ChatGPTs **callback / redirect URL** exactly.
3. **CTXD admin** — Web UI → **admin** → **oauth clients**:
- After DCR, the client may appear automatically; otherwise **create client** with ChatGPTs redirect URI.
- Set **allowed scopes** to **`ctxd.read`** and **`ctxd.write`** (create form or **scopes** → save on an existing row).
4. **Authorize** — complete the browser approval (sign in as CTXD admin in that browser, or use the OAuth approval key). Tokens are capped by the clients allowed scopes.
5. **Chat** — enable the CTXD connector for the conversation. Tools include `list_projects`, `get_project_context`, `search_context`, `get_file`, `update_file`, etc. Call **`get_client_guide`** first in a new session.
**Optional CLI pre-register** (if ChatGPT asks for client credentials before DCR):
```bash
docker exec ctxd dossier oauth-client-create -n "ChatGPT MCP" \
--redirect-uri 'PASTE_CHATGPT_CALLBACK_URL' \
--scope "ctxd.read ctxd.write"
```
| If connect fails | Check |
|------------------|--------|
| **“does not implement OAuth”** | Public host must return **200** JSON (not 502/404) for `/.well-known/oauth-protected-resource/mcp` and `/.well-known/oauth-authorization-server`. Traefik must proxy **all** paths to CTXD `:9091`. Set `OAUTH_ENABLED=true` and `OAUTH_ISSUER=https://ctxd.cubecraftlabs.com`. |
| 502 on well-known URLs | Backend down or wrong Traefik service — fix routing before OAuth can work |
| 404 on `/mcp` | Router not forwarding `/mcp` to CTXD (often a stale `!Path(/mcp)` rule) |
| 401 on `/mcp` | Re-authorize; access token may have expired |
| Redirect mismatch | Callback URL in admin must match ChatGPTs string exactly |
| `invalid_request` on authorize | CTXD requires PKCE **S256** (`code_challenge` + `code_challenge_method=S256`) |
| Stale routes / old code | `./scripts/deploy.sh` or `docker compose up -d --force-recreate ctxd` after postgres is healthy (not `restart` alone; never `--no-deps` unless PG is already up) |
| No write tools | Client scopes include `ctxd.write`; re-authorize after changing scopes |
**Not required for ChatGPT MCP:** Custom GPT, OpenAPI Actions, or `CTXD_API_KEY` in ChatGPT (public path is OAuth). Hermes continues to use LAN `http://<server-ip>:9091/mcp` with the API key.
**Hermes Agent:** **Hermes Agent:**
```yaml ```yaml
# ~/.hermes/config.yaml # ~/.hermes/config.yaml
@@ -203,12 +260,12 @@ mcp_servers:
**Other MCP clients (Codex, Cursor, etc.):** **Other MCP clients (Codex, Cursor, etc.):**
- Register an OAuth client via `POST /oauth/register` with your redirect URI - Register an OAuth client via `POST /oauth/register` with your redirect URI
- Connect to `/readonly/mcp` (read) or `/write/mcp` (read + write) - Connect to **`/mcp`** with `scope=ctxd.read ctxd.write`
- Use the access token as `Authorization: Bearer <token>` - Use the access token as `Authorization: Bearer <token>`
### MCP Tool Reference ### MCP Tool Reference
#### Read-only tools (`/readonly/mcp` and `/write/mcp`) #### Read tools (require `ctxd.read` on `/mcp`)
| Tool | Args | Returns | | Tool | Args | Returns |
|------|------|---------| |------|------|---------|
@@ -220,7 +277,7 @@ mcp_servers:
| `list_files` | `project_id` | All context files in a project | | `list_files` | `project_id` | All context files in a project |
| `get_file` | `project_id`, `file_path` | Single file with metadata header | | `get_file` | `project_id`, `file_path` | Single file with metadata header |
#### Write tools (`/write/mcp` only) #### Write tools (require `ctxd.write` on `/mcp`)
| Tool | Args | Returns | | Tool | Args | Returns |
|------|------|---------| |------|------|---------|
@@ -252,6 +309,7 @@ Request both: `scope=ctxd.read ctxd.write`
| Platform | Redirect URI | | Platform | Redirect URI |
|----------|-------------| |----------|-------------|
| Claude Desktop | `https://claude.ai/api/mcp/auth_callback` | | Claude Desktop | `https://claude.ai/api/mcp/auth_callback` |
| ChatGPT (MCP) | Paste callback URL from ChatGPT connector OAuth UI (per connector) |
| Claude Code | `http://localhost:5555/oauth/callback` | | Claude Code | `http://localhost:5555/oauth/callback` |
| Codex CLI | `http://localhost:7777/oauth/callback` | | Codex CLI | `http://localhost:7777/oauth/callback` |
| Custom | Your app's documented OAuth callback | | Custom | Your app's documented OAuth callback |
@@ -263,13 +321,13 @@ Request both: `scope=ctxd.read ctxd.write`
**Via CLI:** **Via CLI:**
```bash ```bash
# Create # Create
docker exec dossier dossier oauth-client-create -n "Claude Desktop" --redirect-uri https://claude.ai/api/mcp/auth_callback docker exec ctxd dossier oauth-client-create -n "Claude Desktop" --redirect-uri https://claude.ai/api/mcp/auth_callback
# List # List
docker exec dossier dossier oauth-client-list docker exec ctxd dossier oauth-client-list
# Revoke (invalidates all tokens for that client) # Revoke (invalidates all tokens for that client)
docker exec dossier dossier oauth-client-revoke ctxd_xxxxxxxx docker exec ctxd dossier oauth-client-revoke ctxd_xxxxxxxx
``` ```
**Via API (admin session or API key):** **Via API (admin session or API key):**
@@ -291,12 +349,10 @@ curl -X DELETE http://localhost:9091/oauth/clients/ctxd_xxxxxxxx -H "Authorizati
### Router Rule ### Router Rule
Route everything except the internal MCP endpoint: Route the public host to the backend (include `/mcp` — OAuth protects it):
```yaml ```yaml
rule: > rule: Host(`ctxd.yourdomain.com`)
Host(`ctxd.yourdomain.com`) &&
!Path(`/mcp`)
``` ```
This exposes: This exposes:
@@ -304,10 +360,9 @@ This exposes:
- Login (`POST /auth/login`, `GET /auth/me`) - Login (`POST /auth/login`, `GET /auth/me`)
- Full Web UI dashboard (all REST API endpoints) - Full Web UI dashboard (all REST API endpoints)
- OAuth (`/oauth/*`, `/.well-known/*`) - OAuth (`/oauth/*`, `/.well-known/*`)
- Read-only MCP (`/readonly/mcp`) - Public MCP (`/mcp` — OAuth read + write)
- Write MCP (`/write/mcp`)
Only blocked: `/mcp` (internal full MCP — shared API key only) Hermes uses the same `/mcp` path on LAN with the shared API key (not exposed via public OAuth).
### Landing Page Behavior ### Landing Page Behavior
@@ -348,7 +403,7 @@ The compiled view (`get_project_context`) concatenates all files with `## FILENA
All commands run inside the container: All commands run inside the container:
```bash ```bash
docker exec dossier dossier <command> docker exec ctxd dossier <command>
``` ```
### Commands ### Commands
@@ -461,7 +516,7 @@ If you started with SQLite and want to move to PostgreSQL:
docker compose up -d postgres docker compose up -d postgres
# 2. Run the migration script (reads from /data/ctxd.db, writes to DATABASE_URL) # 2. Run the migration script (reads from /data/ctxd.db, writes to DATABASE_URL)
docker exec dossier python3 -m ctxd.migrate_sqlite_to_pg docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
# 3. Set DATABASE_URL in .env and restart # 3. Set DATABASE_URL in .env and restart
docker compose up -d dossier docker compose up -d dossier
@@ -526,7 +581,7 @@ docker compose build
docker compose up -d --no-build docker compose up -d --no-build
# Verify: # Verify:
curl http://localhost:9091/status curl http://localhost:9091/status
docker logs dossier --tail 20 docker logs ctxd --tail 20
``` ```
### Key Conventions ### Key Conventions
@@ -544,7 +599,7 @@ docker logs dossier --tail 20
```bash ```bash
# Reset admin password # Reset admin password
docker exec dossier dossier user-set-password admin -p "new-password" docker exec ctxd dossier user-set-password admin -p "new-password"
``` ```
If the password contains special characters, use the quoting that matches: If the password contains special characters, use the quoting that matches:
@@ -569,7 +624,7 @@ The public host (`https://ctxd.yourdomain.com`) requires Traefik to route `/auth
docker compose ps postgres docker compose ps postgres
# Check connection # Check connection
docker exec dossier python3 -c " docker exec ctxd python3 -c "
import os import os
import psycopg import psycopg
conn = psycopg.connect(os.environ['DATABASE_URL']) conn = psycopg.connect(os.environ['DATABASE_URL'])
@@ -581,13 +636,22 @@ docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'new-passwor
# Then update .env with the new password # Then update .env with the new password
``` ```
### Container keeps restarting ### Container keeps restarting / public site 502
```bash ```bash
docker logs dossier --tail 30 docker compose ps -a
docker logs ctxd --tail 40
``` ```
Common causes: | Symptom | Cause | Fix |
|---------|--------|-----|
| `failed to resolve host 'postgres'` | `ctxd-postgres` not running (often after `--no-deps ctxd`) | `cd app && docker compose up -d postgres ctxd` or `./scripts/deploy.sh` |
| `Restarting (1)` on `ctxd` only | Same — app up without DB on compose network | Start postgres first; wait for **(healthy)** |
| Cloudflare **502** on public URL | Traefik/backend has no healthy upstream on `:9091` | Fix local `curl http://127.0.0.1:9091/status` first |
The container **entrypoint waits up to 120s** for PostgreSQL when `DATABASE_URL` is set (`CTXD_PG_WAIT_SECONDS` to override). If postgres never appears, logs print an explicit message instead of an immediate opaque crash.
Common other causes:
- `DATABASE_URL` password doesn't match what PG was initialized with - `DATABASE_URL` password doesn't match what PG was initialized with
- `OAUTH_ENABLED=true` but `OAUTH_ISSUER` is empty - `OAUTH_ENABLED=true` but `OAUTH_ISSUER` is empty
- Missing `CTXD_API_KEY` when `CTXD_AUTH_ENABLED=true` - Missing `CTXD_API_KEY` when `CTXD_AUTH_ENABLED=true`
+16 -16
View File
@@ -22,28 +22,26 @@ metadata:
# CTXD Client — LLM Agent Guide # CTXD Client — LLM Agent Guide
Use when an LLM agent (Claude Desktop, Codex CLI, Hermes, custom harness) needs to read or update project context stored in a Context Dossier (CTXD) daemon. Use when an LLM agent (Claude Desktop, Codex CLI, Hermes, custom harness) needs to read or update project context stored in Context Dossier (CTXD).
## Overview ## Overview
CTXD is a single source of truth for multi-harness project context. It exposes: CTXD is a single source of truth for multi-harness project context. It exposes:
- **Read-only MCP** (`/readonly/mcp`) — OAuth bearer, `ctxd.read` scope, Streamable HTTP - **Public MCP** (`/mcp`) — OAuth read + write (scope-gated) on the public host
- **Write MCP** (`/write/mcp`) — OAuth bearer, `ctxd.write` scope, Streamable HTTP - **Hermes / automation** — same `http://<lan>:9091/mcp` with `CTXD_API_KEY` (full tool surface)
- **Web UI** (LAN `:9091` or public host) — per-user password login, admin panel
- **Internal full MCP** (`/mcp`) — shared API key (Hermes/automation only), Streamable HTTP
Public host: `https://ctxd.cubecraftcreations.com` (OAuth + MCP + landing page + dashboard). Public host: `https://ctxd.cubecraftlabs.com` (OAuth + MCP + landing page + dashboard).
## Connection URLs ## Connection URLs
| Surface | URL | Auth | | Surface | URL | Auth |
|---------|-----|------| |---------|-----|------|
| Read-only MCP | `https://ctxd.cubecraftcreations.com/readonly/mcp` | OAuth `ctxd.read` | | Public OAuth MCP (read + write) | `https://ctxd.cubecraftlabs.com/mcp` | OAuth `ctxd.read` and/or `ctxd.write` |
| Write MCP | `https://ctxd.cubecraftcreations.com/write/mcp` | OAuth `ctxd.write` | | Legacy aliases | `/readonly/mcp`, `/oauth/mcp` | Same behavior as `/mcp` for OAuth |
| OAuth discovery | `https://ctxd.cubecraftcreations.com/.well-known/oauth-authorization-server` | Public | | OAuth discovery | `https://ctxd.cubecraftlabs.com/.well-known/oauth-authorization-server` | Public |
| DCR registration | `POST https://ctxd.cubecraftcreations.com/oauth/register` | Public | | DCR registration | `POST https://ctxd.cubecraftlabs.com/oauth/register` | Public |
| Landing page | `https://ctxd.cubecraftcreations.com/` | Public | | Landing page | `https://ctxd.cubecraftlabs.com/` | Public |
## OAuth Flow ## OAuth Flow
@@ -51,7 +49,7 @@ Public host: `https://ctxd.cubecraftcreations.com` (OAuth + MCP + landing page +
2. **Register** a client via `POST /oauth/register` (DCR) with your redirect URI 2. **Register** a client via `POST /oauth/register` (DCR) with your redirect URI
3. **Authorize** — open `/oauth/authorize` in a browser; an admin must approve 3. **Authorize** — open `/oauth/authorize` in a browser; an admin must approve
4. **Exchange** the authorization code for tokens at `POST /oauth/token` 4. **Exchange** the authorization code for tokens at `POST /oauth/token`
5. **Use** the access token as `Authorization: Bearer *** on MCP Streamable HTTP connections 5. **Use** the access token as `Authorization: Bearer <token>` on MCP connections (Streamable HTTP for read; SSE for write — see Connection URLs)
### Scopes ### Scopes
@@ -60,7 +58,9 @@ Public host: `https://ctxd.cubecraftcreations.com` (OAuth + MCP + landing page +
| `ctxd.read` | `list_projects`, `get_project_context`, `search_context`, `get_project_tags`, `list_files`, `get_file`, `get_client_guide` | | `ctxd.read` | `list_projects`, `get_project_context`, `search_context`, `get_project_tags`, `list_files`, `get_file`, `get_client_guide` |
| `ctxd.write` | `update_file`, `set_project_tags`, `sync_to_project`, `get_client_guide` | | `ctxd.write` | `update_file`, `set_project_tags`, `sync_to_project`, `get_client_guide` |
Request both scopes for full read+write: `scope=ctxd.read ctxd.write` Request both scopes for full read+write in **one connector**: `scope=ctxd.read ctxd.write`
**Admin:** In the Web UI → Admin → OAuth clients, set **allowed scopes** per client (create form or **scopes** on an existing row). Tokens cannot exceed what the client is allowed, even if the user requests more at authorize time.
### Redirect URIs by Platform ### Redirect URIs by Platform
@@ -85,7 +85,7 @@ Access tokens expire per server config (default ~1 hour). Refresh tokens are iss
The guide lives in the `ctxd-docs` project as `LLM-CLIENT.MD`. It is **locked** — cannot be updated or deleted by any MCP/API client. If you need the guide updated, ask an admin to edit it via the Web UI. The guide lives in the `ctxd-docs` project as `LLM-CLIENT.MD`. It is **locked** — cannot be updated or deleted by any MCP/API client. If you need the guide updated, ask an admin to edit it via the Web UI.
### Read-only (`/readonly/mcp`) ### Read-only tools (require `ctxd.read` on the same connector)
| Tool | Description | | Tool | Description |
|------|-------------| |------|-------------|
@@ -96,7 +96,7 @@ The guide lives in the `ctxd-docs` project as `LLM-CLIENT.MD`. It is **locked**
| `list_files` | All context files in a project (multi-file mode) | | `list_files` | All context files in a project (multi-file mode) |
| `get_file` | Single file with metadata header | | `get_file` | Single file with metadata header |
### Write (`/write/mcp`) ### Write tools (require `ctxd.write` on the same connector)
| Tool | Description | | Tool | Description |
|------|-------------| |------|-------------|
@@ -164,7 +164,7 @@ Both return `403 cannot_update_locked` on PUT and `400 cannot_delete_context` on
| `version_conflict` | Someone else updated the file | Re-read, merge, retry with new `base_version` | | `version_conflict` | Someone else updated the file | Re-read, merge, retry with new `base_version` |
| `not_found` | Project or file doesn't exist | Check slug spelling; create project via Web UI | | `not_found` | Project or file doesn't exist | Check slug spelling; create project via Web UI |
| `invalid_grant` | OAuth token expired or wrong scope | Refresh token or re-authorize with correct scope | | `invalid_grant` | OAuth token expired or wrong scope | Refresh token or re-authorize with correct scope |
| `forbidden` | Wrong endpoint for the tool | Write tools only on `/write/mcp`; read tools on `/readonly/mcp` | | `forbidden` | Wrong scope for the tool | Request `ctxd.write` for write tools; `ctxd.read` for read tools (same MCP URL) |
| `cannot_delete_context` | Tried to delete CONTEXT.MD or LLM-CLIENT.MD | Protected file — update content instead | | `cannot_delete_context` | Tried to delete CONTEXT.MD or LLM-CLIENT.MD | Protected file — update content instead |
| `cannot_update_locked` | Tried to update LLM-CLIENT.MD | Locked guide — ask admin to update via Web UI | | `cannot_update_locked` | Tried to update LLM-CLIENT.MD | Locked guide — ask admin to update via Web UI |
+7 -2
View File
@@ -2,6 +2,11 @@ name: ctxd
# Docker Compose reads .env automatically for variable substitution. # Docker Compose reads .env automatically for variable substitution.
# See .env.example for all available variables. # See .env.example for all available variables.
#
# IMPORTANT: Production uses PostgreSQL (DATABASE_URL host "postgres").
# Always start BOTH services: docker compose up -d
# Or use: ./scripts/deploy.sh
# Do NOT use docker compose up -d --no-deps ctxd unless postgres is already running.
services: services:
postgres: postgres:
@@ -20,11 +25,11 @@ services:
timeout: 5s timeout: 5s
retries: 5 retries: 5
dossier: ctxd:
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
container_name: dossier container_name: ctxd
restart: unless-stopped restart: unless-stopped
ports: ports:
- "${CTXD_PORT:-9091}:${CTXD_PORT:-9091}" - "${CTXD_PORT:-9091}:${CTXD_PORT:-9091}"
+71 -5
View File
@@ -1,11 +1,77 @@
#!/bin/bash #!/bin/bash
set -e set -e
# Initialize if database doesn't exist wait_for_postgres() {
if [ ! -f "$CTXD_HOME/ctxd.db" ]; then if [ -z "$DATABASE_URL" ]; then
echo "ctxd: initializing database at $CTXD_HOME" return 0
python3 -m ctxd init --home "$CTXD_HOME" fi
echo "ctxd: waiting for PostgreSQL (DATABASE_URL is set)..."
python3 <<'PY'
import os, sys, time
url = os.environ.get("DATABASE_URL", "")
max_wait = int(os.environ.get("CTXD_PG_WAIT_SECONDS", "120"))
interval = int(os.environ.get("CTXD_PG_WAIT_INTERVAL", "2"))
deadline = time.time() + max_wait
last_err = ""
attempt = 0
while time.time() < deadline:
attempt += 1
try:
import psycopg
conn = psycopg.connect(url, connect_timeout=5)
conn.close()
print("ctxd: PostgreSQL is ready")
sys.exit(0)
except Exception as e:
last_err = str(e)
if attempt == 1 or attempt % 10 == 0:
print(f"ctxd: postgres not ready (attempt {attempt}): {last_err}", file=sys.stderr)
time.sleep(interval)
print("ctxd: FATAL: PostgreSQL not reachable within wait window.", file=sys.stderr)
print(f"ctxd: last error: {last_err}", file=sys.stderr)
print("ctxd:", file=sys.stderr)
print("ctxd: Fix: start the full stack (postgres + ctxd), e.g.", file=sys.stderr)
print("ctxd: cd app && docker compose up -d postgres ctxd", file=sys.stderr)
print("ctxd:", file=sys.stderr)
print("ctxd: Avoid: docker compose up -d --no-deps ctxd (skips postgres)", file=sys.stderr)
print("ctxd: If postgres is on another host, fix DATABASE_URL / network.", file=sys.stderr)
sys.exit(1)
PY
}
wait_for_postgres
# Initialize if needed
if [ -n "$DATABASE_URL" ]; then
# PostgreSQL mode — check if schema exists
NEEDS_INIT=$(python3 -c "
import psycopg, os, sys
try:
conn = psycopg.connect(os.environ['DATABASE_URL'])
cur = conn.execute(\"SELECT EXISTS (SELECT FROM pg_tables WHERE schemaname='public' AND tablename='users')\")
if not cur.fetchone()[0]:
print('yes')
else:
print('no')
except Exception:
print('yes')
" 2>/dev/null || echo "yes")
if [ "$NEEDS_INIT" = "yes" ]; then
echo "ctxd: initializing PostgreSQL database"
python3 -m ctxd init --home "$CTXD_HOME"
else
echo "ctxd: running pending migrations"
python3 -m ctxd.migrate_user_fk_set_null 2>&1 || echo "ctxd: migration warning (non-fatal)"
fi
else
# SQLite mode — check for db file
if [ ! -f "$CTXD_HOME/ctxd.db" ]; then
echo "ctxd: initializing database at $CTXD_HOME"
python3 -m ctxd init --home "$CTXD_HOME"
fi
fi fi
echo "ctxd: starting daemon on 0.0.0.0:9091" echo "ctxd: starting daemon on ${CTXD_HOST:-0.0.0.0}:${CTXD_PORT:-9091}"
exec python3 -m ctxd exec python3 -m ctxd
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Safe CTXD deploy: always ensures postgres is up before recreating ctxd.
# Usage: ./scripts/deploy.sh [--build-only | --no-build]
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
BUILD=1
if [[ "${1:-}" == "--no-build" ]]; then
BUILD=0
elif [[ "${1:-}" == "--build-only" ]]; then
docker compose build ctxd
echo "Built ctxd image only."
exit 0
fi
if [[ "$BUILD" -eq 1 ]]; then
echo "==> docker compose build ctxd"
docker compose build ctxd
fi
echo "==> docker compose up -d postgres"
docker compose up -d postgres
echo "==> waiting for postgres healthy..."
for i in $(seq 1 60); do
if docker compose ps postgres 2>/dev/null | grep -q '(healthy)'; then
break
fi
sleep 2
done
if ! docker compose ps postgres 2>/dev/null | grep -q '(healthy)'; then
echo "ERROR: ctxd-postgres did not become healthy. Check: docker compose logs postgres" >&2
exit 1
fi
echo "==> docker compose up -d --force-recreate ctxd"
docker compose up -d --force-recreate ctxd
PORT="${CTXD_PORT:-9091}"
echo "==> smoke: http://127.0.0.1:${PORT}/status"
sleep 3
curl -sf "http://127.0.0.1:${PORT}/status" | head -c 200
echo ""
docker compose ps postgres ctxd
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""Smoke test: one OAuth connector lists read+write tools on /mcp."""
import json, urllib.request, urllib.parse, hashlib, base64, secrets, http.client
from pathlib import Path
BASE = "http://127.0.0.1:9091"
env_path = Path("/mnt/ai-storage/projects/ctxd/app/.env")
approval_key = ""
for line in env_path.read_text().splitlines():
if line.startswith("OAUTH_APPROVAL_KEY="):
approval_key = line.split("=", 1)[1]
break
if not approval_key:
raise SystemExit("no OAUTH_APPROVAL_KEY")
class NR(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = urllib.request.build_opener(NR)
r = opener.open(
urllib.request.Request(
f"{BASE}/oauth/register",
data=json.dumps(
{
"redirect_uris": ["http://localhost:9999/cb"],
"client_name": "unified-mcp-test",
"scope": "ctxd.read ctxd.write",
}
).encode(),
headers={"Content-Type": "application/json"},
method="POST",
),
timeout=15,
)
client = json.loads(r.read())
cid, secret = client["client_id"], client["client_secret"]
cv = secrets.token_urlsafe(32)
cc = base64.urlsafe_b64encode(hashlib.sha256(cv.encode()).digest()).rstrip(b"=").decode()
params = {
"response_type": "code",
"client_id": cid,
"redirect_uri": "http://localhost:9999/cb",
"code_challenge": cc,
"code_challenge_method": "S256",
"scope": "ctxd.read ctxd.write",
"state": "t",
}
try:
opener.open(
urllib.request.Request(
f"{BASE}/oauth/authorize?" + urllib.parse.urlencode(params),
data=urllib.parse.urlencode({"approval_key": approval_key}).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"},
method="POST",
),
timeout=15,
)
except urllib.error.HTTPError as e:
loc = e.headers.get("Location", "")
code = dict(urllib.parse.parse_qsl(urllib.parse.urlparse(loc).query)).get("code", "")
r3 = opener.open(
urllib.request.Request(
f"{BASE}/oauth/token",
data=urllib.parse.urlencode(
{
"grant_type": "authorization_code",
"code": code,
"redirect_uri": "http://localhost:9999/cb",
"client_id": cid,
"client_secret": secret,
"code_verifier": cv,
}
).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"},
method="POST",
),
timeout=15,
)
token = json.loads(r3.read())["access_token"]
def mcp_post(path: str, body: dict, session_id: str | None = None) -> tuple[int, str, str | None]:
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
}
if session_id:
headers["mcp-session-id"] = session_id
conn = http.client.HTTPConnection("127.0.0.1", 9091, timeout=15)
conn.request("POST", path, body=json.dumps(body), headers=headers)
resp = conn.getresponse()
sid = resp.getheader("mcp-session-id")
data = resp.read(8000).decode(errors="replace")
conn.close()
return resp.status, data, sid
init_body = {
"jsonrpc": "2.0",
"id": 1,
"method": "initialize",
"params": {
"protocolVersion": "2025-03-26",
"capabilities": {},
"clientInfo": {"name": "unified-test", "version": "1"},
},
}
st, raw, sid = mcp_post("/mcp", init_body)
print("initialize:", st, "session:", sid)
if st != 200:
raise SystemExit(raw[:500])
if sid:
st2, raw2, _ = mcp_post(
"/mcp",
{"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}},
session_id=sid.split(",")[0].strip(),
)
else:
st2, raw2, _ = mcp_post("/mcp", {"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}})
print("tools/list:", st2)
names = []
for line in raw2.splitlines():
if line.startswith("data:"):
try:
payload = json.loads(line[5:].strip())
for t in payload.get("result", {}).get("tools", []):
names.append(t.get("name"))
except json.JSONDecodeError:
pass
if '"tools"' in raw2 and not names:
try:
payload = json.loads(raw2)
names = [t["name"] for t in payload.get("result", {}).get("tools", [])]
except json.JSONDecodeError:
pass
print("tools:", sorted(set(names)))
need = {"list_projects", "update_file", "sync_to_project"}
missing = need - set(names)
if missing:
raise SystemExit(f"missing tools: {missing}")
print("OK unified connector exposes read+write tools")
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""E2E: OAuth write token -> /write/sse and tools/list via SSE."""
import json, urllib.request, urllib.parse, hashlib, base64, secrets, http.client, time
from pathlib import Path
BASE = "http://127.0.0.1:9091"
env_path = Path("/mnt/ai-storage/projects/ctxd/app/.env")
approval_key = ""
for line in env_path.read_text().splitlines():
if line.startswith("OAUTH_APPROVAL_KEY="):
approval_key = line.split("=", 1)[1]
break
if not approval_key:
raise SystemExit("no OAUTH_APPROVAL_KEY")
class NR(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *a): return None
opener = urllib.request.build_opener(NR)
r = opener.open(urllib.request.Request(
f"{BASE}/oauth/register",
data=json.dumps({"redirect_uris": ["http://localhost:9999/cb"], "client_name": "write-test", "scope": "ctxd.read ctxd.write"}).encode(),
headers={"Content-Type": "application/json"}, method="POST"), timeout=15)
client = json.loads(r.read())
cid, secret = client["client_id"], client["client_secret"]
cv = secrets.token_urlsafe(32)
cc = base64.urlsafe_b64encode(hashlib.sha256(cv.encode()).digest()).rstrip(b"=").decode()
params = {"response_type": "code", "client_id": cid, "redirect_uri": "http://localhost:9999/cb",
"code_challenge": cc, "code_challenge_method": "S256", "scope": "ctxd.read ctxd.write", "state": "t"}
try:
opener.open(urllib.request.Request(f"{BASE}/oauth/authorize?" + urllib.parse.urlencode(params),
data=urllib.parse.urlencode({"approval_key": approval_key}).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"}, method="POST"), timeout=15)
except urllib.error.HTTPError as e:
loc = e.headers.get("Location", "")
code = dict(urllib.parse.parse_qsl(urllib.parse.urlparse(loc).query)).get("code", "")
r3 = opener.open(urllib.request.Request(f"{BASE}/oauth/token",
data=urllib.parse.urlencode({"grant_type": "authorization_code", "code": code, "redirect_uri": "http://localhost:9999/cb",
"client_id": cid, "client_secret": secret, "code_verifier": cv}).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"}, method="POST"), timeout=15)
token = json.loads(r3.read())["access_token"]
conn = http.client.HTTPConnection("127.0.0.1", 9091, timeout=5)
conn.request("GET", "/write/sse", headers={"Authorization": f"Bearer {token}"})
resp = conn.getresponse()
print("write/sse status:", resp.status, "ctype:", resp.getheader("content-type"))
chunk = resp.read(500).decode(errors="replace")
print("body[:200]:", chunk[:200])
conn.close()
# POST initialize to /write/messages (needs session from SSE - simplified check)
conn2 = http.client.HTTPConnection("127.0.0.1", 9091, timeout=10)
body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
conn2.request("POST", "/write/messages", body=body,
headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"})
r2 = conn2.getresponse()
print("write/messages POST:", r2.status, r2.read(300).decode(errors="replace")[:200])
conn2.close()
-5
View File
@@ -1,5 +0,0 @@
Metadata-Version: 2.4
Name: ctxd
Version: 0.1.0
Summary: Context daemon — single source of truth for multi-harness project context
Requires-Python: >=3.11
-13
View File
@@ -1,13 +0,0 @@
pyproject.toml
src/ctxd/__init__.py
src/ctxd/__main__.py
src/ctxd/cli.py
src/ctxd/config.py
src/ctxd/db.py
src/ctxd/mcp_stdio.py
src/ctxd/server.py
src/ctxd.egg-info/PKG-INFO
src/ctxd.egg-info/SOURCES.txt
src/ctxd.egg-info/dependency_links.txt
src/ctxd.egg-info/entry_points.txt
src/ctxd.egg-info/top_level.txt
@@ -1 +0,0 @@
-3
View File
@@ -1,3 +0,0 @@
[console_scripts]
ctx = ctxd:cli_entry
ctxd = ctxd:daemon_entry
-1
View File
@@ -1 +0,0 @@
ctxd
+7 -2
View File
@@ -4,7 +4,10 @@ import sys
if len(sys.argv) > 1 and sys.argv[1] in ('init', 'serve', 'project-list', 'project-create', if len(sys.argv) > 1 and sys.argv[1] in ('init', 'serve', 'project-list', 'project-create',
'read', 'cat', 'edit', 'search', 'sync', 'audit', 'read', 'cat', 'edit', 'search', 'sync', 'audit',
'user-list', 'user-create', 'import-vault'): 'user-list', 'user-create', 'import-vault',
'user-set-password', 'oauth-client-create',
'oauth-client-list', 'oauth-client-revoke',
'file-list', 'file-read'):
from ctxd.cli import cli_entry from ctxd.cli import cli_entry
cli_entry() cli_entry()
else: else:
@@ -12,7 +15,9 @@ else:
from ctxd.server import serve_sync from ctxd.server import serve_sync
cfg = CtxConfig.from_home() cfg = CtxConfig.from_home()
if not cfg.db_path.exists(): # When using PostgreSQL, the DB is external — skip the file check.
# When using SQLite, verify the db file exists.
if not cfg.use_postgres and not cfg.db_path.exists():
print("Not initialized. Run 'ctx init' first.", file=sys.stderr) print("Not initialized. Run 'ctx init' first.", file=sys.stderr)
sys.exit(1) sys.exit(1)
serve_sync(cfg) serve_sync(cfg)
+17 -10
View File
@@ -64,14 +64,15 @@ Multi-camera remote monitoring system.
def _seed_context(conn): def _seed_context(conn):
"""Insert seed project context with real newlines.""" """Insert seed project context with real newlines."""
conn.execute( from ctxd.db import _is_pg, _ph
"INSERT OR IGNORE INTO project_context (project_id, content, version, updated_by) VALUES (?, ?, 0, 'system')", ph = _ph(conn, 2)
('welcome', _WELCOME_CONTENT), placeholders = ph.split(", ")
) if _is_pg(conn):
conn.execute( sql = f"INSERT INTO project_context (project_id, content, version, updated_by) VALUES ({placeholders[0]}, {placeholders[1]}, 0, 'admin') ON CONFLICT DO NOTHING"
"INSERT OR IGNORE INTO project_context (project_id, content, version, updated_by) VALUES (?, ?, 0, 'system')", else:
('remote-rig', _REMOTE_RIG_CONTENT), sql = f"INSERT OR IGNORE INTO project_context (project_id, content, version, updated_by) VALUES ({placeholders[0]}, {placeholders[1]}, 0, 'admin')"
) conn.execute(sql, ('welcome', _WELCOME_CONTENT))
conn.execute(sql, ('remote-rig', _REMOTE_RIG_CONTENT))
def cmd_init(args): def cmd_init(args):
@@ -350,6 +351,10 @@ def cmd_oauth_client_create(args):
client = store.register_client({ client = store.register_client({
"client_name": args.name or "Claude MCP Client", "client_name": args.name or "Claude MCP Client",
"redirect_uris": redirects, "redirect_uris": redirects,
"scopes": getattr(args, "scopes", None) or (
[s for s in (args.scope or "").split() if s]
or ["ctxd.read", "ctxd.write"]
),
}) })
issuer = (cfg.oauth_issuer or "").rstrip("/") or "https://ctxd.cubecraftcreations.com" issuer = (cfg.oauth_issuer or "").rstrip("/") or "https://ctxd.cubecraftcreations.com"
print(json.dumps({ print(json.dumps({
@@ -357,7 +362,8 @@ def cmd_oauth_client_create(args):
"client_secret": client["client_secret"], "client_secret": client["client_secret"],
"client_name": client.get("client_name"), "client_name": client.get("client_name"),
"redirect_uris": client.get("redirect_uris"), "redirect_uris": client.get("redirect_uris"),
"connector_url": f"{issuer}/readonly/sse", "scope": client.get("scope"),
"connector_url": f"{issuer}/mcp",
"authorization_server": issuer, "authorization_server": issuer,
"note": "Claude usually registers via POST /oauth/register automatically; save client_secret now — it is not shown again.", "note": "Claude usually registers via POST /oauth/register automatically; save client_secret now — it is not shown again.",
}, indent=2)) }, indent=2))
@@ -370,7 +376,7 @@ def cmd_oauth_client_list(args):
cfg = CtxConfig.from_home(args.home) cfg = CtxConfig.from_home(args.home)
store = OAuthStore(cfg) store = OAuthStore(cfg)
for c in store.list_clients_public(): for c in store.list_clients_public():
print(f"{c.get('client_id')} {c.get('client_name', '')} redirects={c.get('redirect_uris')}") print(f"{c.get('client_id')} {c.get('client_name', '')} scope={c.get('scope', '')} redirects={c.get('redirect_uris')}")
def cmd_oauth_client_revoke(args): def cmd_oauth_client_revoke(args):
@@ -558,6 +564,7 @@ def build_parser() -> argparse.ArgumentParser:
sp.set_defaults(func=cmd_oauth_client_create) sp.set_defaults(func=cmd_oauth_client_create)
sp.add_argument("--name", "-n", default="Claude MCP Client", help="Client display name") sp.add_argument("--name", "-n", default="Claude MCP Client", help="Client display name")
sp.add_argument("--redirect-uri", action="append", dest="redirect_uri", help="Redirect URI (default: Claude MCP callback; repeat for multiple)") sp.add_argument("--redirect-uri", action="append", dest="redirect_uri", help="Redirect URI (default: Claude MCP callback; repeat for multiple)")
sp.add_argument("--scope", default="ctxd.read ctxd.write", help="Allowed scopes (space-separated: ctxd.read ctxd.write)")
sp.add_argument("--home") sp.add_argument("--home")
# oauth-client-list # oauth-client-list
+4
View File
@@ -130,6 +130,10 @@ class CtxConfig:
def log_level(self) -> str: def log_level(self) -> str:
return _env_str("LOG_LEVEL", "info") return _env_str("LOG_LEVEL", "info")
@property
def demo_mode(self) -> bool:
return _env_bool("CTXD_DEMO_MODE", False)
# ── Snapshots ───────────────────────────────────────────────── # ── Snapshots ─────────────────────────────────────────────────
@property @property
def min_snapshots(self) -> int: def min_snapshots(self) -> int:
+17 -1
View File
@@ -114,6 +114,22 @@ def _is_pg(conn) -> bool:
return not isinstance(conn, sqlite3.Connection) return not isinstance(conn, sqlite3.Connection)
def is_integrity_error(exc: BaseException) -> bool:
"""True if exc is a DB constraint violation (unique/primary-key, FK, check)
on either backend. Used to map duplicate-key errors to HTTP 409.
SQLite raises sqlite3.IntegrityError; psycopg raises subclasses of
psycopg.errors.IntegrityError (UniqueViolation, ForeignKeyViolation, ...).
"""
if isinstance(exc, sqlite3.IntegrityError):
return True
try:
import psycopg
except ImportError:
return False
return isinstance(exc, psycopg.errors.IntegrityError)
# ── Helpers ─────────────────────────────────────────────────────────────────── # ── Helpers ───────────────────────────────────────────────────────────────────
def _row_to_dict(row) -> dict | None: def _row_to_dict(row) -> dict | None:
@@ -190,9 +206,9 @@ def user_delete(conn, user_id: str) -> dict:
conn.execute(f"DELETE FROM users WHERE user_id = {ph}", (user_id,)) conn.execute(f"DELETE FROM users WHERE user_id = {ph}", (user_id,))
return {"ok": True} return {"ok": True}
except (sqlite3.IntegrityError, Exception) as e: except (sqlite3.IntegrityError, Exception) as e:
# Check if it's a foreign key violation
if _is_pg(conn): if _is_pg(conn):
import psycopg import psycopg
conn.rollback()
if isinstance(e, psycopg.errors.ForeignKeyViolation): if isinstance(e, psycopg.errors.ForeignKeyViolation):
return {"ok": False, "error": "user_has_references", "hint": "Inactivate the user instead of deleting."} return {"ok": False, "error": "user_has_references", "hint": "Inactivate the user instead of deleting."}
raise raise
+17
View File
@@ -179,6 +179,23 @@ def migrate():
print(f" context_files: {fts_cf} entries") print(f" context_files: {fts_cf} entries")
print(f" user_profiles: {fts_up} entries") print(f" user_profiles: {fts_up} entries")
# Step 4: Reset SERIAL sequences to max(existing_id) + 1
print()
print("Resetting SERIAL sequences...")
for table, col in [
("audit_log", "entry_id"),
("context_files", "file_id"),
("project_permissions", "id"),
("reviews", "review_id"),
("workspace_files", "file_id"),
]:
try:
pconn.execute(f"SELECT setval(pg_get_serial_sequence('{table}', '{col}'), COALESCE((SELECT MAX({col}) FROM {table}), 1))")
print(f" {table}.{col}: seq reset")
except Exception as e:
print(f" {table}.{col}: skipped ({e})")
pconn.commit()
print() print()
print("=" * 60) print("=" * 60)
print(f"Migration complete! {total_rows} total rows migrated.") print(f"Migration complete! {total_rows} total rows migrated.")
+83
View File
@@ -0,0 +1,83 @@
"""
Migration: alter user FK constraints to ON DELETE SET NULL.
PostgreSQL doesn't support ALTER CONSTRAINT inline — you must drop and
re-add the constraint. This script does that for all user_id FKs that
were originally created as RESTRICT (no ON DELETE action).
Run once against the production database:
docker exec ctxd python3 -m ctxd.migrate_user_fk_set_null
Safe to run multiple times (skips if constraint already has ON DELETE SET NULL).
"""
import os
import psycopg
# Tables + constraints to fix. constraint_name is the auto-generated PG name.
FKS_TO_FIX = [
("project_context", "project_context_updated_by_fkey", "updated_by"),
("context_files", "context_files_updated_by_fkey", "updated_by"),
("change_requests", "change_requests_submitted_by_fkey", "submitted_by"),
("reviews", "reviews_reviewer_id_fkey", "reviewer_id"),
("audit_log", "audit_log_user_id_fkey", "user_id"),
]
# Columns that were NOT NULL but must be nullable for ON DELETE SET NULL to work
DROP_NOT_NULL = [
("audit_log", "user_id"),
("change_requests", "submitted_by"),
("reviews", "reviewer_id"),
]
def main():
url = os.environ.get("DATABASE_URL")
if not url:
print("DATABASE_URL not set — nothing to do (SQLite mode)")
return
conn = psycopg.connect(url)
conn.autocommit = True
# Step 1: drop NOT NULL where needed (ON DELETE SET NULL requires nullable)
for table, col in DROP_NOT_NULL:
row = conn.execute(
f"SELECT is_nullable FROM information_schema.columns WHERE table_name = %s AND column_name = %s",
(table, col),
).fetchone()
if row and row[0] == "NO":
print(f" DROP NOT NULL {table}.{col}")
conn.execute(f"ALTER TABLE {table} ALTER COLUMN {col} DROP NOT NULL")
print(f" DONE {table}.{col}")
else:
print(f" SKIP {table}.{col}: already nullable")
# Step 2: fix FK constraints
for table, constraint, col in FKS_TO_FIX:
# Check if the constraint exists and what its current ON DELETE action is
row = conn.execute(
"""SELECT confdeltype FROM pg_constraint
WHERE conname = %s AND contype = 'f'""",
(constraint,),
).fetchone()
if not row:
print(f" SKIP {constraint}: not found (already migrated or table missing)")
continue
if row[0] == "n": # 'n' = SET NULL
print(f" SKIP {constraint}: already ON DELETE SET NULL")
continue
col = constraint.replace(f"{table}_", "").replace("_fkey", "")
print(f" ALTER {table}.{col} ({constraint}): {row[0]} -> SET NULL")
conn.execute(f'ALTER TABLE {table} DROP CONSTRAINT {constraint}')
conn.execute(
f'ALTER TABLE {table} ADD CONSTRAINT {constraint} '
f'FOREIGN KEY ({col}) REFERENCES users(user_id) ON DELETE SET NULL'
)
print(f" DONE {constraint}")
conn.close()
print("Migration complete.")
if __name__ == "__main__":
main()
+4 -4
View File
@@ -65,7 +65,7 @@ CREATE TABLE project_context (
project_id TEXT PRIMARY KEY REFERENCES projects(project_id) ON DELETE CASCADE, project_id TEXT PRIMARY KEY REFERENCES projects(project_id) ON DELETE CASCADE,
content TEXT NOT NULL DEFAULT '', content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 0, version INTEGER NOT NULL DEFAULT 0,
updated_by TEXT REFERENCES users(user_id), updated_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"') updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
@@ -124,7 +124,7 @@ CREATE TABLE change_requests (
request_id TEXT PRIMARY KEY, request_id TEXT PRIMARY KEY,
workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE, workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE,
project_id TEXT NOT NULL REFERENCES projects(project_id), project_id TEXT NOT NULL REFERENCES projects(project_id),
submitted_by TEXT NOT NULL REFERENCES users(user_id), submitted_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
status TEXT NOT NULL DEFAULT 'pending' status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'approved', 'rejected', 'merged')), CHECK (status IN ('pending', 'approved', 'rejected', 'merged')),
diff_summary TEXT, diff_summary TEXT,
@@ -140,7 +140,7 @@ CREATE TABLE change_requests (
CREATE TABLE reviews ( CREATE TABLE reviews (
review_id SERIAL PRIMARY KEY, review_id SERIAL PRIMARY KEY,
request_id TEXT NOT NULL REFERENCES change_requests(request_id) ON DELETE CASCADE, request_id TEXT NOT NULL REFERENCES change_requests(request_id) ON DELETE CASCADE,
reviewer_id TEXT NOT NULL REFERENCES users(user_id), reviewer_id TEXT REFERENCES users(user_id) ON DELETE SET NULL,
decision TEXT NOT NULL CHECK (decision IN ('approved', 'rejected')), decision TEXT NOT NULL CHECK (decision IN ('approved', 'rejected')),
comments TEXT, comments TEXT,
created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'), created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
@@ -171,7 +171,7 @@ CREATE INDEX idx_snapshots_cleanup ON snapshots (project_id, user_id, created_at
-- ============================================================================ -- ============================================================================
CREATE TABLE audit_log ( CREATE TABLE audit_log (
entry_id SERIAL PRIMARY KEY, entry_id SERIAL PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(user_id), user_id TEXT REFERENCES users(user_id) ON DELETE SET NULL,
agent_id TEXT NOT NULL DEFAULT 'cli', agent_id TEXT NOT NULL DEFAULT 'cli',
session_id TEXT, session_id TEXT,
project_id TEXT REFERENCES projects(project_id) ON DELETE SET NULL, project_id TEXT REFERENCES projects(project_id) ON DELETE SET NULL,
+368 -182
View File
@@ -3,6 +3,7 @@ ctxd server — dual-protocol daemon serving context over MCP + HTTP.
""" """
import asyncio import asyncio
import base64 import base64
import contextvars
import hashlib import hashlib
import html import html
import json import json
@@ -44,6 +45,15 @@ WRITE_MCP_TOOLS = {
"sync_to_project", "sync_to_project",
} }
# Per-request OAuth scopes for the public MCP connector (/mcp).
_oauth_mcp_ctx: contextvars.ContextVar[dict] = contextvars.ContextVar(
"oauth_mcp_ctx",
default={"has_read": False, "has_write": False, "user_id": "oauth"},
)
# Streamable HTTP MCP paths. Canonical public URL is /mcp; aliases for older configs.
MCP_STREAMABLE_PATHS = frozenset({"/mcp", "/readonly/mcp", "/oauth/mcp"})
def _b64url_sha256(value: str) -> str: def _b64url_sha256(value: str) -> str:
digest = hashlib.sha256(value.encode()).digest() digest = hashlib.sha256(value.encode()).digest()
@@ -57,6 +67,61 @@ def _now() -> int:
# Claude remote MCP OAuth redirect (Dynamic Client Registration default) # Claude remote MCP OAuth redirect (Dynamic Client Registration default)
CLAUDE_MCP_REDIRECT_URI = "https://claude.ai/api/mcp/auth_callback" CLAUDE_MCP_REDIRECT_URI = "https://claude.ai/api/mcp/auth_callback"
OAUTH_SCOPES_ALLOWED = frozenset({"ctxd.read", "ctxd.write"})
def normalize_client_scope(payload: dict, *, default: str = "ctxd.read") -> str:
"""Return space-separated allowed scopes from payload scope or scopes[]."""
parts: list[str] = []
if isinstance(payload.get("scopes"), list):
parts = [str(s).strip() for s in payload["scopes"] if str(s).strip() in OAUTH_SCOPES_ALLOWED]
else:
raw = payload.get("scope", default)
if isinstance(raw, list):
parts = [str(s).strip() for s in raw if str(s).strip() in OAUTH_SCOPES_ALLOWED]
else:
parts = [s for s in str(raw).split() if s in OAUTH_SCOPES_ALLOWED]
if not parts:
raise ValueError("at least one scope required: ctxd.read, ctxd.write")
order = {"ctxd.read": 0, "ctxd.write": 1}
return " ".join(sorted(set(parts), key=lambda x: order.get(x, 9)))
def intersect_oauth_scopes(client: dict, requested: str) -> str:
"""Grant only scopes both requested and allowed on the client registration."""
allowed = {s for s in (client.get("scope") or "ctxd.read").split() if s in OAUTH_SCOPES_ALLOWED}
if not allowed:
allowed = {"ctxd.read"}
req = {s for s in (requested or "").split() if s in OAUTH_SCOPES_ALLOWED}
if not req:
req = allowed
granted = req & allowed
if not granted:
return ""
order = {"ctxd.read": 0, "ctxd.write": 1}
return " ".join(sorted(granted, key=lambda x: order.get(x, 9)))
def oauth_protected_resource_document(base: str, *, resource_path: str = "/mcp") -> dict:
"""RFC 9728 / MCP authorization discovery document."""
resource = f"{base.rstrip('/')}{resource_path}"
return {
"resource": resource,
"authorization_servers": [base.rstrip("/")],
"bearer_methods_supported": ["header"],
"scopes_supported": ["ctxd.read", "ctxd.write"],
"resource_documentation": f"{base.rstrip('/')}/",
}
def oauth_protected_resource_metadata_url(base: str, *, mcp_path: str = "/mcp") -> str:
"""Well-known URL for the MCP resource (RFC 9728 path insertion)."""
base = base.rstrip("/")
if mcp_path in ("", "/"):
return f"{base}/.well-known/oauth-protected-resource"
suffix = mcp_path.lstrip("/")
return f"{base}/.well-known/oauth-protected-resource/{suffix}"
def _is_public_oauth_as_route(path: str, method: str) -> bool: def _is_public_oauth_as_route(path: str, method: str) -> bool:
"""OAuth authorization-server endpoints (public). Admin /oauth/clients stays on HTTPServer.""" """OAuth authorization-server endpoints (public). Admin /oauth/clients stays on HTTPServer."""
@@ -110,8 +175,12 @@ class OAuthStore:
redirect_uris = payload.get("redirect_uris") or [] redirect_uris = payload.get("redirect_uris") or []
if not isinstance(redirect_uris, list) or not redirect_uris: if not isinstance(redirect_uris, list) or not redirect_uris:
raise ValueError("redirect_uris required") raise ValueError("redirect_uris required")
reg = dict(payload)
if "scope" not in reg and "scopes" not in reg:
reg["scope"] = "ctxd.read ctxd.write"
client_id = "ctxd_" + secrets.token_urlsafe(24) client_id = "ctxd_" + secrets.token_urlsafe(24)
client_secret = secrets.token_urlsafe(32) client_secret = secrets.token_urlsafe(32)
scope_str = normalize_client_scope(reg)
client = { client = {
"client_id": client_id, "client_id": client_id,
"client_secret": client_secret, "client_secret": client_secret,
@@ -122,7 +191,7 @@ class OAuthStore:
"grant_types": payload.get("grant_types", ["authorization_code", "refresh_token"]), "grant_types": payload.get("grant_types", ["authorization_code", "refresh_token"]),
"response_types": payload.get("response_types", ["code"]), "response_types": payload.get("response_types", ["code"]),
"token_endpoint_auth_method": payload.get("token_endpoint_auth_method", "client_secret_post"), "token_endpoint_auth_method": payload.get("token_endpoint_auth_method", "client_secret_post"),
"scope": payload.get("scope", "ctxd.read"), "scope": scope_str,
} }
self.state.setdefault("clients", {})[client_id] = client self.state.setdefault("clients", {})[client_id] = client
self._save() self._save()
@@ -159,6 +228,19 @@ class OAuthStore:
self._save() self._save()
return True return True
def update_client(self, client_id: str, payload: dict) -> dict | None:
"""Update admin-editable client fields (scope, client_name)."""
clients = self.state.get("clients", {})
client = clients.get(client_id)
if not client:
return None
if "client_name" in payload and payload["client_name"]:
client["client_name"] = str(payload["client_name"]).strip()
if "scope" in payload or "scopes" in payload:
client["scope"] = normalize_client_scope(payload)
self._save()
return {k: v for k, v in client.items() if k != "client_secret"}
def create_code(self, *, client_id: str, redirect_uri: str, code_challenge: str, scope: str, resource: str | None, approved_by_user_id: str | None = None) -> str: def create_code(self, *, client_id: str, redirect_uri: str, code_challenge: str, scope: str, resource: str | None, approved_by_user_id: str | None = None) -> str:
code = secrets.token_urlsafe(32) code = secrets.token_urlsafe(32)
self.state.setdefault("codes", {})[code] = { self.state.setdefault("codes", {})[code] = {
@@ -243,6 +325,20 @@ class OAuthStore:
return False, None return False, None
return True, record.get("approved_by_user_id") return True, record.get("approved_by_user_id")
def resolve_public_oauth(self, token: str) -> tuple[bool, bool, bool, str | None]:
"""Valid OAuth access token for public MCP. Returns (ok, has_read, has_write, user_id)."""
if not token:
return False, False, False, None
record = self.state.get("access_tokens", {}).get(token)
if not record or record.get("expires_at", 0) <= _now():
return False, False, False, None
scope = record.get("scope", "")
has_read = "ctxd.read" in scope
has_write = "ctxd.write" in scope
if not has_read and not has_write:
return False, False, False, None
return True, has_read, has_write, record.get("approved_by_user_id")
class WebSessionStore: class WebSessionStore:
"""Opaque bearer sessions for CTXD Web UI per-user login.""" """Opaque bearer sessions for CTXD Web UI per-user login."""
@@ -331,9 +427,12 @@ def _public_user_row(user: dict) -> dict:
# ── MCP Server ───────────────────────────────────────────────────────────────── # ── MCP Server ─────────────────────────────────────────────────────────────────
def make_mcp_server(cfg: CtxConfig, readonly: bool = False): def make_mcp_server(cfg: CtxConfig, readonly: bool = False, oauth_scoped: bool = False):
"""Create an MCP server instance wired to our database.""" """Create an MCP server instance wired to our database."""
app = Server("ctxd-readonly" if readonly else "context-dossier") if oauth_scoped:
app = Server("ctxd-oauth")
else:
app = Server("ctxd-readonly" if readonly else "context-dossier")
def _conn(): def _conn():
"""Short-lived connection per request (WAL allows concurrent reads).""" """Short-lived connection per request (WAL allows concurrent reads)."""
@@ -503,17 +602,50 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
}, },
), ),
] ]
if oauth_scoped:
ctx = _oauth_mcp_ctx.get()
allowed: set[str] = set()
if ctx.get("has_read"):
allowed |= READONLY_MCP_TOOLS
if ctx.get("has_write"):
allowed |= WRITE_MCP_TOOLS
if not allowed:
allowed = {"get_client_guide"}
return [tool for tool in tools if tool.name in allowed]
if readonly: if readonly:
return [tool for tool in tools if tool.name in READONLY_MCP_TOOLS] return [tool for tool in tools if tool.name in READONLY_MCP_TOOLS]
return tools return tools
@app.call_tool() @app.call_tool()
async def call_tool(name: str, arguments: dict): async def call_tool(name: str, arguments: dict):
if readonly and name not in READONLY_MCP_TOOLS: if oauth_scoped:
ctx = _oauth_mcp_ctx.get()
if name in WRITE_MCP_TOOLS and not ctx.get("has_write"):
return [types.TextContent(
type="text",
text=json.dumps({
"error": "forbidden",
"tool": name,
"message": "ctxd.write scope required",
}, indent=2),
)]
if name in READONLY_MCP_TOOLS and not ctx.get("has_read"):
return [types.TextContent(
type="text",
text=json.dumps({
"error": "forbidden",
"tool": name,
"message": "ctxd.read scope required",
}, indent=2),
)]
actor = ctx.get("user_id") or "oauth"
elif readonly and name not in READONLY_MCP_TOOLS:
return [types.TextContent( return [types.TextContent(
type="text", type="text",
text=json.dumps({"error": "forbidden", "tool": name, "message": "read-only endpoint"}, indent=2), text=json.dumps({"error": "forbidden", "tool": name, "message": "read-only endpoint"}, indent=2),
)] )]
else:
actor = "hermes-gateway"
conn = _conn() conn = _conn()
try: try:
@@ -536,7 +668,7 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
"updated_at": ctx.get("updated_at"), "updated_at": ctx.get("updated_at"),
"content": ctx["content"], "content": ctx["content"],
} }
_db.audit_log(conn, "hermes-gateway", "read", _db.audit_log(conn, actor, "read",
f"MCP read context for {pid}", f"MCP read context for {pid}",
agent_id="hermes", project_id=pid, agent_id="hermes", project_id=pid,
entity_type="project", entity_id=pid) entity_type="project", entity_id=pid)
@@ -554,7 +686,7 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
query = arguments["query"] query = arguments["query"]
limit = arguments.get("limit", 10) limit = arguments.get("limit", 10)
results = _db.search(conn, query, limit=limit) results = _db.search(conn, query, limit=limit)
_db.audit_log(conn, "hermes-gateway", "search", _db.audit_log(conn, actor, "search",
f"MCP search: {query[:80]}", f"MCP search: {query[:80]}",
agent_id="hermes") agent_id="hermes")
conn.commit() conn.commit()
@@ -584,7 +716,7 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
pid = arguments["project_id"] pid = arguments["project_id"]
tags = [str(t).upper().replace(" ", "-") for t in arguments.get("tags", [])] tags = [str(t).upper().replace(" ", "-") for t in arguments.get("tags", [])]
_db.project_set_tags(conn, pid, tags) _db.project_set_tags(conn, pid, tags)
_db.audit_log(conn, "hermes-gateway", "set_tags", _db.audit_log(conn, actor, "set_tags",
f"Set tags for {pid}: {', '.join(tags)}", f"Set tags for {pid}: {', '.join(tags)}",
agent_id="hermes", project_id=pid, agent_id="hermes", project_id=pid,
entity_type="project", entity_id=pid) entity_type="project", entity_id=pid)
@@ -625,7 +757,7 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
result = _db.file_read(conn, pid, file_path) result = _db.file_read(conn, pid, file_path)
if result is None: if result is None:
return [types.TextContent(type="text", text=f"File '{file_path}' not found in project '{pid}'")] return [types.TextContent(type="text", text=f"File '{file_path}' not found in project '{pid}'")]
_db.audit_log(conn, "hermes-gateway", "read", _db.audit_log(conn, actor, "read",
f"MCP read file {file_path} for {pid}", f"MCP read file {file_path} for {pid}",
agent_id="hermes", project_id=pid, agent_id="hermes", project_id=pid,
entity_type="file", entity_id=file_path) entity_type="file", entity_id=file_path)
@@ -649,7 +781,7 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
file_path = arguments["file_path"] file_path = arguments["file_path"]
content = arguments["content"] content = arguments["content"]
base_version = arguments["base_version"] base_version = arguments["base_version"]
result = _db.file_update(conn, pid, file_path, content, "hermes-gateway", base_version) result = _db.file_update(conn, pid, file_path, content, actor, base_version)
if result.get("ok"): if result.get("ok"):
conn.commit() conn.commit()
return [types.TextContent( return [types.TextContent(
@@ -664,111 +796,6 @@ def make_mcp_server(cfg: CtxConfig, readonly: bool = False):
return app return app
def make_write_mcp_server(cfg: CtxConfig):
"""Create an MCP server that exposes only write tools (OAuth ctxd.write scope)."""
app = Server("ctxd-write")
def _conn():
return _db.init_db(cfg)
@app.list_tools()
async def list_tools():
tools = [
types.Tool(
name="update_file",
description="Update a single context file in a project with optimistic version checking",
inputSchema={
"type": "object",
"properties": {
"project_id": {"type": "string", "description": "Project slug"},
"file_path": {"type": "string", "description": "File name e.g. CONTEXT.md"},
"content": {"type": "string", "description": "New file content (markdown)"},
"base_version": {"type": "integer", "description": "Current version of the file (for conflict detection)"},
},
"required": ["project_id", "file_path", "content", "base_version"],
},
),
types.Tool(
name="set_project_tags",
description="Set metadata tags for a project (replaces all tags)",
inputSchema={
"type": "object",
"properties": {
"project_id": {"type": "string", "description": "Project slug"},
"tags": {"type": "array", "items": {"type": "string"}, "description": "Uppercase metadata tags"},
},
"required": ["project_id", "tags"],
},
),
types.Tool(
name="sync_to_project",
description="Write current shared context as AGENTS.md + symlinks to the project root",
inputSchema={
"type": "object",
"properties": {
"project_id": {"type": "string", "description": "Project slug"},
},
"required": ["project_id"],
},
),
types.Tool(
name="get_client_guide",
description="Return the CTXD client guide (LLM-CLIENT.MD) — always read this first. Covers OAuth, MCP tools, read/write endpoints, version-checked updates, and error handling.",
inputSchema={
"type": "object",
"properties": {},
},
),
]
return tools
@app.call_tool()
async def call_tool(name: str, arguments: dict):
conn = _conn()
try:
if name == "get_client_guide":
result = _db.file_read(conn, "ctxd-docs", "LLM-CLIENT.MD")
if result is None:
return [types.TextContent(type="text", text="Client guide not found — ctxd-docs/LLM-CLIENT.MD is missing")]
return [types.TextContent(
type="text",
text=json.dumps(result, indent=2),
)]
elif name == "update_file":
pid = arguments["project_id"]
file_path = arguments["file_path"]
content = arguments["content"]
base_version = arguments.get("base_version", 0)
result = _db.file_update(conn, pid, file_path, content, base_version, updated_by="oauth-write")
conn.commit()
return [types.TextContent(type="text", text=json.dumps(result, indent=2))]
elif name == "set_project_tags":
pid = arguments["project_id"]
tags = [str(t).upper().replace(" ", "-") for t in arguments.get("tags", [])]
_db.project_set_tags(conn, pid, tags)
_db.audit_log(conn, "oauth-write", "set_tags",
f"Set tags for {pid}: {', '.join(tags)}",
agent_id="oauth", project_id=pid,
entity_type="project", entity_id=pid)
conn.commit()
return [types.TextContent(type="text", text=json.dumps({"ok": True, "project_id": pid, "tags": tags}, indent=2))]
elif name == "sync_to_project":
pid = arguments["project_id"]
result = _db.sync_to_project(conn, pid)
conn.commit()
return [types.TextContent(type="text", text=json.dumps(result, indent=2))]
else:
return [types.TextContent(type="text", text=json.dumps({"error": "unknown tool", "tool": name}, indent=2))]
finally:
conn.close()
return app
# ── HTTP Server (stdlib-only, no dependencies) ──────────────────────────────── # ── HTTP Server (stdlib-only, no dependencies) ────────────────────────────────
class HTTPServer: class HTTPServer:
@@ -785,6 +812,17 @@ class HTTPServer:
try: try:
return self._route(method, path, body, auth or {}) return self._route(method, path, body, auth or {})
except Exception as e: except Exception as e:
# The shared PG connection is left in an aborted-transaction state by
# any failed statement; without this rollback every subsequent request
# 500s ("current transaction is aborted"). This is the single funnel
# that guarantees the connection is clean no matter which path failed.
try:
self._conn.rollback()
except Exception:
logger.exception("Rollback failed after request error")
if _db.is_integrity_error(e):
return (409, {"Content-Type": "application/json"},
json.dumps({"error": "conflict", "detail": str(e)}))
logger.exception("HTTP error") logger.exception("HTTP error")
return (500, {"Content-Type": "text/plain"}, f"Internal error: {e}") return (500, {"Content-Type": "text/plain"}, f"Internal error: {e}")
@@ -911,10 +949,9 @@ class HTTPServer:
return (200, {"Content-Type": "application/json"}, json.dumps({"ok": True, "user_id": user_id})) return (200, {"Content-Type": "application/json"}, json.dumps({"ok": True, "user_id": user_id}))
except Exception as e: except Exception as e:
self._conn.rollback() self._conn.rollback()
msg = str(e) if _db.is_integrity_error(e):
if "UNIQUE" in msg or "idx_users_lower" in msg:
return (409, {"Content-Type": "application/json"}, json.dumps({"error": "user already exists"})) return (409, {"Content-Type": "application/json"}, json.dumps({"error": "user already exists"}))
return (400, {"Content-Type": "application/json"}, json.dumps({"error": msg})) return (400, {"Content-Type": "application/json"}, json.dumps({"error": str(e)}))
# POST /users/<id>/password — set or reset password (admin only) # POST /users/<id>/password — set or reset password (admin only)
if method == "POST" and path.startswith("/users/") and path.endswith("/password"): if method == "POST" and path.startswith("/users/") and path.endswith("/password"):
@@ -973,6 +1010,7 @@ class HTTPServer:
return (400, {"Content-Type": "application/json"}, json.dumps({"error": "cannot delete your own account"})) return (400, {"Content-Type": "application/json"}, json.dumps({"error": "cannot delete your own account"}))
result = _db.user_delete(self._conn, uid) result = _db.user_delete(self._conn, uid)
if not result.get("ok"): if not result.get("ok"):
self._conn.rollback()
status = 404 if result.get("error") == "not_found" else 409 status = 404 if result.get("error") == "not_found" else 409
return (status, {"Content-Type": "application/json"}, json.dumps(result)) return (status, {"Content-Type": "application/json"}, json.dumps(result))
actor = self._auth_user_id(auth) actor = self._auth_user_id(auth)
@@ -1003,6 +1041,8 @@ class HTTPServer:
return (200, {"Content-Type": "application/json"}, json.dumps({"ok": True, "project_id": pid})) return (200, {"Content-Type": "application/json"}, json.dumps({"ok": True, "project_id": pid}))
except Exception as e: except Exception as e:
self._conn.rollback() self._conn.rollback()
if _db.is_integrity_error(e):
return (409, {"Content-Type": "application/json"}, json.dumps({"error": "project already exists"}))
return (400, {"Content-Type": "application/json"}, json.dumps({"error": str(e)})) return (400, {"Content-Type": "application/json"}, json.dumps({"error": str(e)}))
# DELETE /projects/<id> — delete a project (admin only) # DELETE /projects/<id> — delete a project (admin only)
@@ -1245,10 +1285,14 @@ class HTTPServer:
if isinstance(redirect_uris, str): if isinstance(redirect_uris, str):
redirect_uris = [redirect_uris] redirect_uris = [redirect_uris]
try: try:
reg_payload = dict(payload)
if "scope" not in reg_payload and "scopes" not in reg_payload:
reg_payload.setdefault("scope", "ctxd.read ctxd.write")
client = self.oauth_store.register_client({ client = self.oauth_store.register_client({
"client_name": payload.get("client_name") or "MCP connector", "client_name": payload.get("client_name") or "MCP connector",
"redirect_uris": redirect_uris, "redirect_uris": redirect_uris,
"scope": payload.get("scope") or "ctxd.read", "scope": reg_payload.get("scope"),
"scopes": reg_payload.get("scopes"),
}) })
except ValueError as e: except ValueError as e:
return (400, {"Content-Type": "application/json"}, json.dumps({"error": str(e)})) return (400, {"Content-Type": "application/json"}, json.dumps({"error": str(e)}))
@@ -1261,7 +1305,7 @@ class HTTPServer:
self._conn.commit() self._conn.commit()
issuer = (self.cfg.oauth_issuer or "").rstrip("/") or "https://ctxd.cubecraftcreations.com" issuer = (self.cfg.oauth_issuer or "").rstrip("/") or "https://ctxd.cubecraftcreations.com"
out = {k: v for k, v in client.items()} out = {k: v for k, v in client.items()}
out["connector_url"] = f"{issuer}/readonly/mcp" out["connector_url"] = f"{issuer}/mcp"
out["authorization_server"] = issuer out["authorization_server"] = issuer
return (201, {"Content-Type": "application/json"}, json.dumps(out)) return (201, {"Content-Type": "application/json"}, json.dumps(out))
@@ -1285,6 +1329,31 @@ class HTTPServer:
self._conn.commit() self._conn.commit()
return (200, {"Content-Type": "application/json"}, json.dumps({"ok": True, "client_id": client_id})) return (200, {"Content-Type": "application/json"}, json.dumps({"ok": True, "client_id": client_id}))
# PATCH /oauth/clients/<client_id> — update scopes / name (admin only)
if method == "PATCH" and path.startswith("/oauth/clients/"):
if auth.get("type") == "session" and not self._is_admin(auth):
return (403, {"Content-Type": "application/json"}, json.dumps({"error": "admin required"}))
if not self.oauth_store:
return (503, {"Content-Type": "application/json"}, json.dumps({"error": "oauth unavailable"}))
client_id = path[len("/oauth/clients/"):].strip()
if not client_id:
return (400, {"Content-Type": "application/json"}, json.dumps({"error": "client_id required"}))
payload = json.loads(body or b"{}")
try:
updated = self.oauth_store.update_client(client_id, payload)
except ValueError as e:
return (400, {"Content-Type": "application/json"}, json.dumps({"error": str(e)}))
if not updated:
return (404, {"Content-Type": "application/json"}, json.dumps({"error": "client not found"}))
actor = self._auth_user_id(auth)
_db.audit_log(
self._conn, actor, "update",
f"Updated OAuth client {client_id} scopes={updated.get('scope', '')}",
agent_id="oauth", entity_type="oauth_client", entity_id=client_id,
)
self._conn.commit()
return (200, {"Content-Type": "application/json"}, json.dumps(updated))
return (404, {"Content-Type": "text/plain"}, f"Not found: {method} {path}") return (404, {"Content-Type": "text/plain"}, f"Not found: {method} {path}")
@@ -1300,11 +1369,11 @@ class CombinedApp:
self.oauth_store = OAuthStore(cfg) self.oauth_store = OAuthStore(cfg)
self.http_handler = HTTPServer(cfg, self.session_store, self.oauth_store) self.http_handler = HTTPServer(cfg, self.session_store, self.oauth_store)
self.mcp_app = make_mcp_server(cfg) self.mcp_app = make_mcp_server(cfg)
self.oauth_mcp_app = make_mcp_server(cfg, oauth_scoped=True)
self.readonly_mcp_app = make_mcp_server(cfg, readonly=True) self.readonly_mcp_app = make_mcp_server(cfg, readonly=True)
self.write_mcp_app = make_write_mcp_server(cfg)
self._mcp_init_opts = self.mcp_app.create_initialization_options() self._mcp_init_opts = self.mcp_app.create_initialization_options()
self._oauth_mcp_init_opts = self.oauth_mcp_app.create_initialization_options()
self._readonly_mcp_init_opts = self.readonly_mcp_app.create_initialization_options() self._readonly_mcp_init_opts = self.readonly_mcp_app.create_initialization_options()
self._write_mcp_init_opts = self.write_mcp_app.create_initialization_options()
async def __call__(self, scope, receive, send): async def __call__(self, scope, receive, send):
if scope["type"] == "http": if scope["type"] == "http":
@@ -1371,12 +1440,7 @@ class CombinedApp:
qs = urllib.parse.parse_qs(scope.get("query_string", b"").decode()) qs = urllib.parse.parse_qs(scope.get("query_string", b"").decode())
if method == "GET" and path.startswith("/.well-known/oauth-protected-resource"): if method == "GET" and path.startswith("/.well-known/oauth-protected-resource"):
payload = { payload = oauth_protected_resource_document(base)
"resource": f"{base}/readonly/mcp",
"authorization_servers": [issuer],
"bearer_methods_supported": ["header"],
"resource_documentation": f"{base}/",
}
await self._send_json(send, 200, payload) await self._send_json(send, 200, payload)
return return
@@ -1389,7 +1453,11 @@ class CombinedApp:
"response_types_supported": ["code"], "response_types_supported": ["code"],
"grant_types_supported": ["authorization_code", "refresh_token"], "grant_types_supported": ["authorization_code", "refresh_token"],
"code_challenge_methods_supported": ["S256"], "code_challenge_methods_supported": ["S256"],
"token_endpoint_auth_methods_supported": ["client_secret_post", "client_secret_basic", "none"], "token_endpoint_auth_methods_supported": [
"client_secret_basic",
"client_secret_post",
"none",
],
"scopes_supported": ["ctxd.read", "ctxd.write"], "scopes_supported": ["ctxd.read", "ctxd.write"],
} }
await self._send_json(send, 200, payload) await self._send_json(send, 200, payload)
@@ -1457,11 +1525,20 @@ class CombinedApp:
if not ok_approve or not approver: if not ok_approve or not approver:
await self._send_html(send, 403, "<h1>Forbidden</h1><p>Sign in to CTXD as an admin in this browser, or provide a valid OAuth approval key.</p>") await self._send_html(send, 403, "<h1>Forbidden</h1><p>Sign in to CTXD as an admin in this browser, or provide a valid OAuth approval key.</p>")
return return
granted_scope = intersect_oauth_scopes(client, scope_value)
if not granted_scope:
target = error_redirect("invalid_scope")
if target:
await send({"type": "http.response.start", "status": 302, "headers": [(b"location", target.encode())]})
await send({"type": "http.response.body", "body": b""})
else:
await self._send_json(send, 400, {"error": "invalid_scope"})
return
code = self.oauth_store.create_code( code = self.oauth_store.create_code(
client_id=client_id, client_id=client_id,
redirect_uri=redirect_uri, redirect_uri=redirect_uri,
code_challenge=code_challenge, code_challenge=code_challenge,
scope=scope_value, scope=granted_scope,
resource=resource, resource=resource,
approved_by_user_id=approver, approved_by_user_id=approver,
) )
@@ -1469,11 +1546,11 @@ class CombinedApp:
self.http_handler._conn, self.http_handler._conn,
approver, approver,
"update", "update",
f"Approved OAuth read-only access for {client.get('client_name', client_id)}", f"Approved OAuth access for {client.get('client_name', client_id)}",
agent_id="oauth", agent_id="oauth",
entity_type="oauth_client", entity_type="oauth_client",
entity_id=client_id, entity_id=client_id,
details={"redirect_uri": redirect_uri, "scope": scope_value}, details={"redirect_uri": redirect_uri, "scope": granted_scope},
) )
self.http_handler._conn.commit() self.http_handler._conn.commit()
q = {"code": code} q = {"code": code}
@@ -1544,14 +1621,19 @@ class CombinedApp:
break break
return token return token
async def _auth_error(status: int = 401, message: str = "unauthorized"): async def _auth_error(status: int = 401, message: str = "unauthorized", *, mcp_resource: bool = False):
headers = [(b"content-type", b"application/json")] headers = [(b"content-type", b"application/json")]
if self.cfg.oauth_enabled: if self.cfg.oauth_enabled:
base = self._base_url(scope) base = self._base_url(scope)
headers.append(( if mcp_resource:
b"www-authenticate", meta_url = oauth_protected_resource_metadata_url(base, mcp_path="/mcp")
f'Bearer resource_metadata="{base}/.well-known/oauth-protected-resource"'.encode(), else:
)) meta_url = oauth_protected_resource_metadata_url(base)
www = (
f'Bearer resource_metadata="{meta_url}", '
f'scope="ctxd.read ctxd.write"'
)
headers.append((b"www-authenticate", www.encode()))
await send({ await send({
"type": "http.response.start", "type": "http.response.start",
"status": status, "status": status,
@@ -1569,54 +1651,45 @@ class CombinedApp:
await self._serve_oauth(scope, receive, send, oauth_body) await self._serve_oauth(scope, receive, send, oauth_body)
return return
# Public read-only MCP endpoint for Claude Desktop/web connectors. # Streamable HTTP MCP — /mcp (OAuth read+write or API key full access on /mcp only).
# GET accepts OAuth bearer tokens. The legacy external_readonly_key query if path in MCP_STREAMABLE_PATHS:
# fallback remains temporarily for migration away from ?key= URLs.
def _readonly_token_valid(token: str) -> bool:
if self.cfg.api_key and token == self.cfg.api_key:
return True
if self.cfg.external_readonly_key and token == self.cfg.external_readonly_key:
return True
if self.cfg.oauth_enabled and self.oauth_store.validate_access_token(token):
return True
return False
# Public read-only MCP (Streamable HTTP) — OAuth bearer or legacy readonly key.
if path == "/readonly/mcp":
token = _request_token() token = _request_token()
if not _readonly_token_valid(token): if path == "/mcp" and self.cfg.api_key and token == self.cfg.api_key:
await _auth_error() await self._serve_streamable_mcp(
scope, receive, send,
self.mcp_app, self._mcp_init_opts,
surface_name="internal",
)
return return
await self._serve_streamable_mcp( ok, ctx = self._public_mcp_auth_context(token)
scope, receive, send, if ok:
self.readonly_mcp_app, self._readonly_mcp_init_opts, await self._serve_oauth_mcp_streamable(scope, receive, send, ctx)
) return
await _auth_error(mcp_resource=True)
return return
# Public write MCP (Streamable HTTP) — OAuth bearer with ctxd.write scope. # Legacy SSE write path (prefer /mcp Streamable HTTP).
if path == "/write/mcp": # Note: Using SSE instead of Streamable HTTP because MCP SDK 1.28's
# Streamable HTTP transport has a race condition where EventSourceResponse
# is killed by the task group before sending headers.
if method == "GET" and path == "/write/sse":
token = _request_token() token = _request_token()
valid, _ = self.oauth_store.validate_write_token(token) if self.cfg.oauth_enabled else (False, None) ok, ctx = self._public_mcp_auth_context(token)
if not valid: if not ok:
await _auth_error() await _auth_error()
return return
await self._serve_streamable_mcp( await self._serve_write_mcp_sse(scope, receive, send, ctx)
scope, receive, send,
self.write_mcp_app, self._write_mcp_init_opts,
)
return return
# Internal full MCP (Streamable HTTP) — shared API key only. if method == "POST" and path in ("/write/messages", "/write/messages/"):
if path == "/mcp":
token = _request_token() token = _request_token()
if self.cfg.auth_enabled and token != self.cfg.api_key: ok, ctx = self._public_mcp_auth_context(token)
if not ok:
await _auth_error() await _auth_error()
return return
await self._serve_streamable_mcp( await self._serve_write_mcp_sse(scope, receive, send, ctx)
scope, receive, send,
self.mcp_app, self._mcp_init_opts,
)
return return
token = _request_token() token = _request_token()
def _resolve_http_auth(tok: str) -> tuple[bool, dict]: def _resolve_http_auth(tok: str) -> tuple[bool, dict]:
@@ -1701,20 +1774,133 @@ class CombinedApp:
"body": body_str.encode(), "body": body_str.encode(),
}) })
async def _serve_streamable_mcp(self, scope, receive, send, mcp_app, init_opts): def _public_mcp_auth_context(self, token: str) -> tuple[bool, dict]:
"""Streamable HTTP transport for MCP (single endpoint handles POST/GET/DELETE).""" """Resolve auth for the unified public MCP connector."""
import anyio if self.cfg.api_key and token == self.cfg.api_key:
return True, {"has_read": True, "has_write": False, "user_id": "api-key"}
if self.cfg.external_readonly_key and token == self.cfg.external_readonly_key:
return True, {"has_read": True, "has_write": False, "user_id": "readonly-key"}
if self.cfg.oauth_enabled:
ok, has_read, has_write, uid = self.oauth_store.resolve_public_oauth(token)
if ok:
return True, {
"has_read": has_read,
"has_write": has_write,
"user_id": uid or "oauth",
}
return False, {}
async def _serve_oauth_mcp_streamable(self, scope, receive, send, ctx: dict):
reset = _oauth_mcp_ctx.set(ctx)
try:
await self._serve_streamable_mcp(
scope, receive, send,
self.oauth_mcp_app, self._oauth_mcp_init_opts,
surface_name="oauth",
)
finally:
_oauth_mcp_ctx.reset(reset)
async def _serve_write_mcp_sse(self, scope, receive, send, ctx: dict | None = None):
"""Write MCP over SSE transport (legacy; same app as /readonly/mcp)."""
from mcp.server.sse import SseServerTransport
if ctx is not None:
reset = _oauth_mcp_ctx.set(ctx)
else:
reset = None
try:
if not hasattr(self, "_write_sse_transport") or self._write_sse_transport is None:
self._write_sse_transport = SseServerTransport("/write/messages")
sse = self._write_sse_transport
method = scope.get("method", "GET")
path = scope.get("path", "/")
if method == "GET" and path == "/write/sse":
async with sse.connect_sse(scope, receive, send) as streams:
await self.oauth_mcp_app.run(
streams[0], streams[1], self._oauth_mcp_init_opts,
)
return
if method == "POST" and path in ("/write/messages", "/write/messages/"):
await sse.handle_post_message(scope, receive, send)
return
await send({
"type": "http.response.start",
"status": 404,
"headers": [(b"content-type", b"text/plain")],
})
await send({"type": "http.response.body", "body": b"Not found"})
finally:
if reset is not None:
_oauth_mcp_ctx.reset(reset)
async def _serve_streamable_mcp(self, scope, receive, send, mcp_app, init_opts, surface_name="mcp"):
"""Streamable HTTP transport for MCP with persistent session handling.
Transports are cached by session ID. The first request creates a
transport, starts the MCP server, and handles the request. Subsequent
requests with the same session ID reuse the transport.
"""
from mcp.server.streamable_http import StreamableHTTPServerTransport from mcp.server.streamable_http import StreamableHTTPServerTransport
transport = StreamableHTTPServerTransport(mcp_session_id=None) # Check for existing session ID from request headers
async with transport.connect() as (read_stream, write_stream): req_headers = {k.decode().lower(): v.decode() for k, v in scope.get("headers", [])}
async def run_server(): session_id = req_headers.get("mcp-session-id", "")
await mcp_app.run(read_stream, write_stream, init_opts)
async with anyio.create_task_group() as tg: # Get or create session cache for this surface
tg.start_soon(run_server) cache_attr = f"_streamable_{surface_name}_sessions"
await transport.handle_request(scope, receive, send) if not hasattr(self, cache_attr):
tg.cancel_scope.cancel() setattr(self, cache_attr, {})
sessions = getattr(self, cache_attr)
# Reuse existing transport for this session
if session_id and session_id in sessions:
transport, server_task = sessions[session_id]
await transport.handle_request(scope, receive, send)
return
# New session — generate session ID, create transport, start server
import secrets as _secrets
import asyncio
new_session_id = _secrets.token_hex(16)
transport = StreamableHTTPServerTransport(
mcp_session_id=new_session_id,
is_json_response_enabled=True,
)
# Connect the transport (creates read/write streams) — keep open for session lifetime
cm = transport.connect()
read_stream, write_stream = await cm.__aenter__()
async def run_server():
await mcp_app.run(read_stream, write_stream, init_opts)
server_task = asyncio.create_task(run_server())
# Wrap send to inject Mcp-Session-Id header
original_send = send
header_injected = False
async def send_with_session_id(message):
nonlocal header_injected
if message.get("type") == "http.response.start" and not header_injected:
existing = list(message.get("headers", []))
if not any(k.lower() == b"mcp-session-id" for k, _ in existing):
existing.append((b"mcp-session-id", new_session_id.encode()))
message["headers"] = existing
header_injected = True
await original_send(message)
# Handle the request (this blocks until the HTTP response is complete)
await transport.handle_request(scope, receive, send_with_session_id)
# Store transport and server task for reuse on subsequent requests
sessions[new_session_id] = (transport, server_task)
# ── Entry point ──────────────────────────────────────────────────────────────── # ── Entry point ────────────────────────────────────────────────────────────────
+201 -11
View File
@@ -831,6 +831,38 @@ body {
letter-spacing: 0.06em; letter-spacing: 0.06em;
color: var(--ink-dim); color: var(--ink-dim);
} }
.user-list-table .user-actions-cell {
white-space: nowrap;
display: flex;
gap: 0.3rem;
justify-content: flex-end;
}
.user-list-table .user-actions-cell button {
font-family: var(--font);
font-size: 0.55rem;
text-transform: uppercase;
letter-spacing: 0.06em;
padding: 0.2rem 0.4rem;
min-height: 1.7rem;
cursor: pointer;
border: 1px solid var(--border-light);
background: var(--input-bg);
color: var(--ink);
border-radius: 0;
-webkit-appearance: none;
}
.user-list-table .user-actions-cell button:hover {
background: var(--hover);
border-color: var(--accent);
color: var(--accent);
}
.user-list-table .user-actions-cell button.danger {
color: var(--danger);
}
.user-list-table .user-actions-cell button.danger:hover {
border-color: var(--danger);
color: var(--danger);
}
.user-state-active { color: var(--accent3); } .user-state-active { color: var(--accent3); }
.user-state-inactive { color: var(--danger); } .user-state-inactive { color: var(--danger); }
/* OAuth client list — compact, matches user table density */ /* OAuth client list — compact, matches user table density */
@@ -869,8 +901,42 @@ body {
word-break: break-all; word-break: break-all;
line-height: 1.35; line-height: 1.35;
} }
.admin-oauth-row .oauth-client-scopes {
color: var(--accent3);
font-size: 0.58rem;
margin-top: 0.2rem;
letter-spacing: 0.04em;
}
.oauth-scope-row {
display: flex;
flex-wrap: wrap;
gap: 0.75rem 1.25rem;
margin: 0.35rem 0 0.5rem;
align-items: center;
}
.oauth-scope-row label {
display: flex;
align-items: center;
gap: 0.35rem;
font-size: 0.62rem;
text-transform: uppercase;
letter-spacing: 0.06em;
color: var(--ink);
margin: 0;
}
.admin-oauth-row .oauth-row-actions {
display: flex;
flex-direction: column;
gap: 0.35rem;
align-items: stretch;
min-width: 5.5rem;
}
.admin-oauth-row .oauth-row-actions button {
width: 100%;
}
.dialog.dialog-admin button.oauth-revoke, .dialog.dialog-admin button.oauth-revoke,
.dialog.dialog-admin .manage-user-actions button { .dialog.dialog-admin .manage-user-actions button,
.dialog.dialog-admin .oauth-row-actions button {
font-family: var(--font); font-family: var(--font);
font-size: 0.62rem; font-size: 0.62rem;
text-transform: uppercase; text-transform: uppercase;
@@ -884,6 +950,7 @@ body {
border-radius: 0; border-radius: 0;
-webkit-appearance: none; -webkit-appearance: none;
} }
.dialog.dialog-admin .oauth-row-actions button.oauth-scopes:hover,
.dialog.dialog-admin .manage-user-actions button:hover { .dialog.dialog-admin .manage-user-actions button:hover {
background: var(--hover); background: var(--hover);
border-color: var(--accent); border-color: var(--accent);
@@ -899,6 +966,39 @@ body {
border-color: #d4ae5c; border-color: #d4ae5c;
color: var(--bg); color: var(--bg);
} }
.dialog.dialog-admin .oauth-row-actions button.oauth-scopes {
color: var(--ink);
background: var(--input-bg);
}
.dialog.dialog-admin .oauth-row-actions button.oauth-scopes.active {
border-color: var(--accent);
color: var(--accent);
background: var(--hover);
}
.oauth-scope-edit {
margin-top: 0.45rem;
padding: 0.45rem 0.5rem;
border: 1px solid var(--border);
background: var(--input-bg);
}
.dialog.dialog-admin .oauth-scope-edit button.primary {
font-family: var(--font);
font-size: 0.58rem;
text-transform: uppercase;
letter-spacing: 0.08em;
padding: 0.28rem 0.5rem;
min-height: 1.85rem;
cursor: pointer;
border: 1px solid var(--accent);
border-radius: 0;
background: var(--accent);
color: var(--bg);
-webkit-appearance: none;
}
.dialog.dialog-admin .oauth-scope-edit button.primary:hover {
background: #d4ae5c;
border-color: #d4ae5c;
}
.dialog.dialog-admin button.oauth-revoke { .dialog.dialog-admin button.oauth-revoke {
border-color: var(--border-light); border-color: var(--border-light);
color: var(--danger); color: var(--danger);
@@ -1266,12 +1366,17 @@ body {
</div> </div>
</div> </div>
<div id="admin-oauth-sub-create" class="admin-oauth-subpanel"> <div id="admin-oauth-sub-create" class="admin-oauth-subpanel">
<p class="dialog-hint">Register OAuth clients for any MCP or OAuth consumer. Public connector URL: <code>/readonly/sse</code>. Many clients auto-register via <code>POST /oauth/register</code>; use this form when you need explicit <code>client_id</code> / <code>client_secret</code> (secret shown once).</p> <p class="dialog-hint">Register OAuth clients for MCP connectors. Public URL: <code>/mcp</code>. Scopes cap what tokens may grant at authorize time.</p>
<label>client name</label> <label>client name</label>
<input type="text" id="admin-oauth-name" value="MCP connector" autocomplete="off"> <input type="text" id="admin-oauth-name" value="MCP connector" autocomplete="off">
<label>redirect uri</label> <label>redirect uri</label>
<input type="url" id="admin-oauth-redirect" placeholder="https://your-app.example/oauth/callback" autocomplete="off"> <input type="url" id="admin-oauth-redirect" placeholder="https://your-app.example/oauth/callback" autocomplete="off">
<p class="dialog-hint" style="margin-top:0.25rem">Use the callback URL your OAuth client documents.</p> <p class="dialog-hint" style="margin-top:0.25rem">Use the callback URL your OAuth client documents.</p>
<label>allowed scopes</label>
<div class="oauth-scope-row">
<label><input type="checkbox" id="admin-oauth-scope-read" checked> ctxd.read</label>
<label><input type="checkbox" id="admin-oauth-scope-write" checked> ctxd.write</label>
</div>
<div class="actions" style="justify-content: flex-start; margin: 0.5rem 0;"> <div class="actions" style="justify-content: flex-start; margin: 0.5rem 0;">
<button class="primary" type="button" onclick="createOAuthClient()">generate client id / secret</button> <button class="primary" type="button" onclick="createOAuthClient()">generate client id / secret</button>
</div> </div>
@@ -1311,9 +1416,6 @@ body {
<input type="password" id="admin-password" placeholder="required for new users; leave blank to keep unchanged" autocomplete="new-password"> <input type="password" id="admin-password" placeholder="required for new users; leave blank to keep unchanged" autocomplete="new-password">
<div class="manage-user-actions"> <div class="manage-user-actions">
<button class="primary" type="button" onclick="saveManagedUser()">save user</button> <button class="primary" type="button" onclick="saveManagedUser()">save user</button>
<button type="button" onclick="setManagedUserActive(false)">inactivate</button>
<button type="button" onclick="setManagedUserActive(true)">activate</button>
<button type="button" onclick="deleteManagedUser()">delete</button>
<button type="button" onclick="clearManageUserForm()">clear</button> <button type="button" onclick="clearManageUserForm()">clear</button>
</div> </div>
</div> </div>
@@ -1641,14 +1743,21 @@ function renderUserList() {
el.innerHTML = '<div class="empty-state">no users</div>'; el.innerHTML = '<div class="empty-state">no users</div>';
return; return;
} }
el.innerHTML = `<table class="user-list-table"><thead><tr><th>user id</th><th>name</th><th>role</th><th>state</th></tr></thead><tbody>${ el.innerHTML = `<table class="user-list-table"><thead><tr><th>user id</th><th>name</th><th>role</th><th>state</th><th style="text-align:right">actions</th></tr></thead><tbody>${
state.users.map(u => { state.users.map(u => {
const active = u.active !== false; const active = u.active !== false;
const uid = escapeHtml(u.user_id || '');
return `<tr> return `<tr>
<td class="user-id">${escapeHtml(u.user_id || '')}</td> <td class="user-id">${uid}</td>
<td>${escapeHtml(u.display_name || '')}</td> <td>${escapeHtml(u.display_name || '')}</td>
<td>${escapeHtml(u.role || '')}</td> <td>${escapeHtml(u.role || '')}</td>
<td class="${active ? 'user-state-active' : 'user-state-inactive'}">${active ? 'active' : 'inactive'}</td> <td class="${active ? 'user-state-active' : 'user-state-inactive'}">${active ? 'active' : 'inactive'}</td>
<td class="user-actions-cell">
${active
? `<button type="button" onclick="userListAction('${uid}', 'inactivate')">inactivate</button>`
: `<button type="button" onclick="userListAction('${uid}', 'activate')">activate</button>`}
<button type="button" class="danger" onclick="userListAction('${uid}', 'delete')">delete</button>
</td>
</tr>`; </tr>`;
}).join('') }).join('')
}</tbody></table>`; }</tbody></table>`;
@@ -1712,6 +1821,29 @@ async function setManagedUserActive(active) {
} }
} }
async function userListAction(userId, action) {
if (!userId) return;
if (action === 'delete') {
if (!confirm(`Delete user "${userId}"? If delete fails, inactivate instead.`)) return;
try {
await api('DELETE', `/users/${encodeURIComponent(userId)}`);
showToast(`user deleted · ${userId}`, 'success');
await loadUsers();
} catch (e) {
if (e.message !== 'unauthorized') showToast('delete failed: ' + e.message, 'error');
}
return;
}
const active = action === 'activate';
try {
await api('PATCH', `/users/${encodeURIComponent(userId)}`, { active });
showToast(active ? `activated · ${userId}` : `inactivated · ${userId}`, 'success');
await loadUsers();
} catch (e) {
if (e.message !== 'unauthorized') showToast('update failed: ' + e.message, 'error');
}
}
async function deleteManagedUser() { async function deleteManagedUser() {
const userId = document.getElementById('admin-manage-select').value || document.getElementById('admin-user-id').value.trim(); const userId = document.getElementById('admin-manage-select').value || document.getElementById('admin-user-id').value.trim();
if (!userId) { if (!userId) {
@@ -1729,6 +1861,17 @@ async function deleteManagedUser() {
} }
} }
function oauthScopesFromCheckboxes(readId, writeId) {
const scopes = [];
if (document.getElementById(readId).checked) scopes.push('ctxd.read');
if (document.getElementById(writeId).checked) scopes.push('ctxd.write');
return scopes;
}
function parseClientScope(scopeStr) {
return new Set((scopeStr || '').split(/\s+/).filter(Boolean));
}
async function loadOAuthClients() { async function loadOAuthClients() {
const el = document.getElementById('admin-oauth-list'); const el = document.getElementById('admin-oauth-list');
el.innerHTML = '<div class="loading">loading oauth clients…</div>'; el.innerHTML = '<div class="loading">loading oauth clients…</div>';
@@ -1740,14 +1883,25 @@ async function loadOAuthClients() {
} }
el.innerHTML = clients.map(c => { el.innerHTML = clients.map(c => {
const cid = c.client_id || ''; const cid = c.client_id || '';
const scopeSet = parseClientScope(c.scope);
const esc = escapeHtml(cid);
return ` return `
<div class="admin-oauth-row"> <div class="admin-oauth-row" data-client-id="${esc}">
<div class="oauth-client-meta"> <div class="oauth-client-meta">
<div class="oauth-client-id">${escapeHtml(cid)}</div> <div class="oauth-client-id">${esc}</div>
<div class="oauth-client-name">${escapeHtml(c.client_name || '')}</div> <div class="oauth-client-name">${escapeHtml(c.client_name || '')}</div>
<div class="oauth-client-uri">${escapeHtml((c.redirect_uris || []).join(', '))}</div> <div class="oauth-client-uri">${escapeHtml((c.redirect_uris || []).join(', '))}</div>
<div class="oauth-client-scopes">scopes: ${escapeHtml(c.scope || 'ctxd.read')}</div>
<div class="oauth-scope-row oauth-scope-edit" id="oauth-scope-edit-${esc}" style="display:none">
<label><input type="checkbox" class="oauth-edit-read" data-cid="${esc}" ${scopeSet.has('ctxd.read') ? 'checked' : ''}> ctxd.read</label>
<label><input type="checkbox" class="oauth-edit-write" data-cid="${esc}" ${scopeSet.has('ctxd.write') ? 'checked' : ''}> ctxd.write</label>
<button type="button" class="primary" data-save-scopes="${esc}" onclick="saveOAuthClientScopes(this.dataset.saveScopes)">save scopes</button>
</div>
</div>
<div class="oauth-row-actions">
<button type="button" class="oauth-scopes" data-toggle-scopes="${esc}" onclick="toggleOAuthScopeEdit(this.dataset.toggleScopes, this)">scopes</button>
<button type="button" class="oauth-revoke" data-client-id="${esc}" onclick="revokeOAuthClient(this.dataset.clientId)">revoke</button>
</div> </div>
<button type="button" class="oauth-revoke" data-client-id="${escapeHtml(cid)}" onclick="revokeOAuthClient(this.dataset.clientId)">revoke</button>
</div>`; </div>`;
}).join(''); }).join('');
} catch (e) { } catch (e) {
@@ -1757,6 +1911,36 @@ async function loadOAuthClients() {
} }
} }
function toggleOAuthScopeEdit(clientId, btn) {
const panel = document.getElementById(`oauth-scope-edit-${clientId}`);
if (!panel) return;
const open = panel.style.display === 'none';
panel.style.display = open ? 'flex' : 'none';
const row = document.querySelector(`.admin-oauth-row[data-client-id="${CSS.escape(clientId)}"]`);
const toggleBtn = btn || row?.querySelector('button.oauth-scopes');
if (toggleBtn) toggleBtn.classList.toggle('active', open);
}
async function saveOAuthClientScopes(clientId) {
if (!clientId) return;
const row = document.querySelector(`.admin-oauth-row[data-client-id="${CSS.escape(clientId)}"]`);
if (!row) return;
const scopes = [];
if (row.querySelector('.oauth-edit-read')?.checked) scopes.push('ctxd.read');
if (row.querySelector('.oauth-edit-write')?.checked) scopes.push('ctxd.write');
if (!scopes.length) {
showToast('select at least one scope', 'error');
return;
}
try {
await api('PATCH', `/oauth/clients/${encodeURIComponent(clientId)}`, { scopes });
showToast(`scopes updated · ${clientId}`, 'success');
await loadOAuthClients();
} catch (e) {
if (e.message !== 'unauthorized') showToast('scope update failed: ' + e.message, 'error');
}
}
async function revokeOAuthClient(clientId) { async function revokeOAuthClient(clientId) {
if (!clientId) return; if (!clientId) return;
if (!confirm(`Revoke OAuth client "${clientId}"?\n\nPending auth codes and access/refresh tokens for this client are invalidated.`)) return; if (!confirm(`Revoke OAuth client "${clientId}"?\n\nPending auth codes and access/refresh tokens for this client are invalidated.`)) return;
@@ -1777,7 +1961,12 @@ async function createOAuthClient() {
return; return;
} }
try { try {
const client = await api('POST', '/oauth/clients', { client_name: name, redirect_uris: [redirect] }); const scopes = oauthScopesFromCheckboxes('admin-oauth-scope-read', 'admin-oauth-scope-write');
if (!scopes.length) {
showToast('select at least one scope', 'error');
return;
}
const client = await api('POST', '/oauth/clients', { client_name: name, redirect_uris: [redirect], scopes });
const pre = document.getElementById('admin-oauth-result'); const pre = document.getElementById('admin-oauth-result');
pre.style.display = 'block'; pre.style.display = 'block';
pre.textContent = [ pre.textContent = [
@@ -1787,6 +1976,7 @@ async function createOAuthClient() {
`authorization_server: ${client.authorization_server || ''}`, `authorization_server: ${client.authorization_server || ''}`,
`client_id: ${client.client_id || ''}`, `client_id: ${client.client_id || ''}`,
`client_secret: ${client.client_secret || ''}`, `client_secret: ${client.client_secret || ''}`,
`scope: ${client.scope || scopes.join(' ')}`,
`redirect_uris: ${JSON.stringify(client.redirect_uris || [])}`, `redirect_uris: ${JSON.stringify(client.redirect_uris || [])}`,
].join('\n'); ].join('\n');
showToast('oauth client created — copy secret from admin panel', 'success'); showToast('oauth client created — copy secret from admin panel', 'success');
+147
View File
@@ -0,0 +1,147 @@
# I Built a Context Server to Fight My ADHD-Induced Context Sprawl
**How replacing 7 scattered config files with a single source of truth saved my sanity (and my tokens).**
---
If you work with AI tools the way I do — Claude Desktop for research, Claude Code for implementation, Codex CLI for reviews, Cursor for quick edits, and Hermes Agent for orchestration — you have a problem you might not have named yet.
I call it **context sprawl**. And if you have ADHD, it's worse than you think.
## The Problem
Every AI harness has its own context file convention:
| Harness | File |
|---------|------|
| Hermes | `AGENTS.md` |
| Claude Code | `CLAUDE.md` |
| Cursor | `.cursorrules` |
| Copilot | `.github/copilot-instructions.md` |
| Codex CLI | `CODEX.md` |
| Windsurf | `.windsurfrules` |
| Continue.dev | `.continuerc` |
When you work on the same project across multiple harnesses, you end up with N copies of the same project context. They drift. One says the build system is Go; another still says Python. One knows about the new OAuth flow; another doesn't. The LLM reads stale instructions and makes confident, wrong decisions.
For someone with ADHD, this is a specific kind of hell:
- You **forget** which file you last updated
- You **avoid** updating because it means updating 4-6 files
- You **lose track** of which harness has the "right" version
- You **context-switch** between tools and can't remember what the project actually needs
- The friction of maintaining context across tools becomes so high that you **stop maintaining context at all**
And then every LLM session starts from scratch — re-asking questions you already answered, re-discovering decisions you already made, re-making mistakes you already fixed.
## The Tipping Point
I was working on a project called RemoteRig — a multi-camera remote monitoring system built in Go with ESP32-C6 camera nodes. I had context in:
- `AGENTS.md` (Hermes)
- `CLAUDE.md` (Claude Code)
- `.cursorrules` (Cursor)
- `CODEX.md` (Codex CLI)
- A Notion page (for humans, but also where I drafted things)
- My own memory of what I'd told each tool (which is the worst place to store anything when you have ADHD)
I'd update `AGENTS.md` with a new architecture decision, forget to update `CLAUDE.md`, and then Claude Code would suggest something I'd already ruled out. I'd spend the first 20 minutes of every session re-explaining the project. Sometimes I'd update the wrong file and wonder why Codex was still using the old Go module structure.
The context was sprawled across the filesystem like a hoarder's garage, and I was the only one who knew where anything was — except I didn't, because ADHD.
## The Solution: CTXD (Context Dossier)
I built **CTXD** — a single source of truth for project context that every AI harness can read from and write to.
### Architecture
```
Context Dossier (Docker container, port 9091)
├── PostgreSQL 16 backend
├── Multi-file context per project:
│ ├── CONTEXT.MD ← canonical overview, synced as AGENTS.md to repos
│ ├── DECISIONS.MD ← architecture decisions, rationale
│ ├── RUNBOOKS.MD ← deploy, troubleshoot, operate procedures
│ ├── PROMPTS.MD ← project-specific prompts for different harnesses
│ └── GLOSSARY.MD ← project-specific terms, acronyms
├── OAuth 2.0 authorization server
│ ├── Dynamic Client Registration (Claude auto-registers)
│ ├── Authorization Code + PKCE
│ └── Scopes: ctxd.read, ctxd.write
├── Streamable HTTP MCP endpoints:
│ ├── /readonly/mcp (read-only, OAuth ctxd.read)
│ ├── /write/mcp (read-write, OAuth ctxd.write)
│ └── /mcp (internal, shared API key)
├── Web UI (per-user login, admin panel)
├── Version-checked writes (optimistic concurrency)
├── Append-only audit log
├── Point-in-time snapshots (auto-rotated)
└── Full-text search (PostgreSQL tsvector + GIN)
```
### How it works
1. **One canonical file per project**`CONTEXT.MD` is the source of truth. CTXD syncs it to `AGENTS.md` in the project root and creates symlinks (`CLAUDE.md`, `.cursorrules`, `CODEX.md``AGENTS.md`). Every harness reads the same file.
2. **LLMs connect via MCP** — Claude Desktop connects to `https://ctxd.yourdomain.com/readonly/mcp` and auto-discovers OAuth. No manual config, no API keys in URLs. It can read project context, search across all projects, and list individual files.
3. **LLMs can write context** — With a `ctxd.write`-scoped OAuth token, an LLM can update files with version checking. If two agents try to edit simultaneously, the second one gets a `version_conflict` and knows to re-read, merge, and retry.
4. **The compiled view** — When an LLM calls `get_project_context("remote-rig")`, it gets all files concatenated into a single document with a metadata header. One call, complete context, no guessing.
5. **A locked client guide** — There's a `get_client_guide()` MCP tool that returns a locked `LLM-CLIENT.MD` file. Every LLM session starts by reading this guide, which explains how to connect, what tools are available, and what the discipline rules are. The file is immutable — no LLM can modify its own instruction manual.
### What it solves for ADHD
- **One place to update** — Change `CONTEXT.MD` once. Every harness gets it via sync or MCP. No more updating 6 files.
- **No remembering required** — The LLM reads context at the start of every session. You don't have to remember what you told it last time.
- **Audit trail** — Every read, write, and search is logged. When you can't remember what changed (and you can't, because ADHD), the audit log tells you.
- **Version checking** — Optimistic concurrency means no silent overwrites. If you and an LLM edit at the same time, the conflict is caught, not lost.
- **Search** — Full-text search across all projects. "What did I decide about the camera node?" → search → answer.
- **Snapshots** — Automatic point-in-time backups before every edit. Roll back if something goes wrong.
## The Build
I built CTXD over a series of focused sessions with Hermes Agent — itself powered by the same MCP infrastructure. Some of the tools that made this possible:
- **Hermes Agent** for orchestration and the MCP framework
- **PostgreSQL 16** as the primary database (migrated from SQLite)
- **MCP SDK 1.28** with Streamable HTTP transport (replaced the older SSE pattern)
- **OAuth 2.0** with DCR, PKCE, and per-scope access control
- **Docker Compose** for deployment — env-driven, no config files needed
- **Traefik** for public TLS termination and routing
The development was iterative and honest. I hit real bugs: the `set -e` in the entrypoint script killed the container because a PostgreSQL schema check returned exit code 1. The `__main__.py` checked for a SQLite file that doesn't exist when using PostgreSQL, causing an infinite re-initialization loop. The FTS5 search query used `?` placeholders instead of `%s` for psycopg. Each one was a "oh, of course" moment — the kind you only get by actually building and deploying.
## What I Learned
**Context sprawl is an accessibility problem.** For neurotypical developers, maintaining 6 config files is annoying. For someone with ADHD, it's a barrier that leads to giving up on context entirely — and then every AI interaction starts from zero.
**The single source of truth pattern isn't just about consistency.** It's about reducing the cognitive load of context management to one action: update `CONTEXT.MD`. That's one decision, one file, one habit — something even an ADHD brain can maintain.
**OAuth + MCP is the right protocol stack for this.** LLMs don't need API keys in environment variables. They need to discover a resource, authenticate via OAuth, and use standardized tools. The fact that Claude Desktop can connect to CTXD with zero manual configuration — just a URL — is the difference between "I should set this up" and "I'll just re-explain the project again."
**PostgreSQL matters more than you think.** I started with SQLite, which was fine for one user. But the moment you want concurrent agents writing context, proper backups, and full-text search that actually works — PostgreSQL is the right tool. The migration was a real engineering task (dual-backend support, schema rewrite, FTS5 → tsvector), and it was worth every hour.
## Where It's Going
CTXD is now running in production at `ctxd.cubecraftcreations.com`, serving context to Claude Desktop, Hermes, and any OAuth-capable MCP client. It backs:
- RemoteRig (Go, ESP32-C6)
- ExtrudeX (3D printing control system)
- Tracehound (network diagnostic tool)
- Control Center (homelab dashboard)
The code is open and documented. There's a static demo at `ctxd-demo` that shows the UI without needing a database. There's a full README, an LLM.txt for automated deployment, and a SKILL.md that LLMs can read to understand how to use the server.
## The Takeaway
If you work across multiple AI tools and find yourself re-explaining your project every session — you have context sprawl. If you have ADHD, context sprawl isn't just inefficient; it's a wall between you and productive AI-assisted work.
Building CTXD was my answer. It turned 6 files into 1, turned "remember what I told Claude" into "Claude reads the context server," and turned "I'll update the context later" into "I update one file and every tool gets it."
One source of truth. One habit. Zero context sprawl.
---
*CTXD is open source. The production repo is at [CubeCraft-Creations/CTXD](https://code.cubecraftlabs.com/CubeCraft-Creations/CTXD), with a static demo at [CTXD-Demo](https://code.cubecraftlabs.com/CubeCraft-Creations/CTXD-Demo). Built by Joshua at CubeCraft Creations.*