Compare commits

..

27 Commits

Author SHA1 Message Date
overseer 295693f641 Merge pull request 'fix: complete user-delete FK lockstep across PG and SQLite schemas' (#1) from fix/user-delete-fk-schema-lockstep into main
Reviewed-on: #1
2026-06-30 15:31:16 -04:00
overseer 8395df0f80 fix: complete user-delete FK lockstep across PG and SQLite schemas
The prior user-deletion work updated the PG schema and a live-PG migration
but left the canonical schema definitions inconsistent, breaking user
deletion on fresh PG installs and on all SQLite dev installs.

- schema.sql: add ON DELETE SET NULL to context_files.updated_by (was the
  only user FK missing it; fresh PG installs could not delete an authoring
  user).
- schema_sqlite.sql: bring five user_id FK columns into lockstep with PG
  (drop NOT NULL, add ON DELETE SET NULL): project_context.updated_by,
  context_files.updated_by, change_requests.submitted_by,
  reviews.reviewer_id, audit_log.user_id.
- schema_sqlite.sql: remove the audit_log append-only UPDATE/DELETE triggers.
  ON DELETE SET NULL on audit_log.user_id is an UPDATE the trigger aborted,
  so deleting any user who had ever logged in failed. This mirrors schema.sql,
  which dropped the equivalent PG triggers in fc1a2f5; append-only is enforced
  at the application layer (db.py only INSERTs into audit_log).
- db.py: user_delete no longer swallows non-FK exceptions on the SQLite path
  (Exception masked sqlite3.IntegrityError); only FK violations map to the
  soft "user_has_references" response, everything else propagates. PG
  rollback-on-any-error (shared-connection cascade fix) is preserved.
- db.py: document that SQLite cannot ALTER FK constraints in place; existing
  dev DBs must be recreated to pick up these changes.
- server.py: the global 409 handler no longer leaks raw psycopg text (index
  names, column expressions) to API callers; it is logged instead.
- migrate_user_fk_set_null.py: use the column from FKS_TO_FIX directly instead
  of re-deriving it from the constraint name.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 17:35:34 +00:00
overseer 3ef4f3e707 chore: add CLAUDE.md, stop tracking egg-info build artifacts
- Add CLAUDE.md (Claude Code orientation for the repo).
- Remove app/src/ctxd.egg-info/* from version control and gitignore
  *.egg-info/ — it is regenerated by `pip install -e` and only dirties
  the working tree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 21:11:54 +00:00
overseer bbb0215c98 docs: update repo domain to cubecraftlabs.com
Rename code.cubecraftcreations.com -> code.cubecraftlabs.com in the
deployment guide and article links.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 21:10:15 +00:00
overseer bdc984e5ff fix: stop duplicate-key 500 cascade on shared PG connection
The REST/Web-UI HTTPServer shares one long-lived PG connection across all
requests. Any statement that raised mid-request (e.g. a UniqueViolation from
a desynced SERIAL sequence) aborted the transaction; the global handler
returned 500 without rolling back, so every subsequent request failed with
InFailedSqlTransaction until restart — surfacing as "duplicate keys cause
500s" and "500 immediately after login".

- server.py: global handler now always rolls back the shared connection on
  error and maps constraint violations to 409 (was 500/400). This is the one
  funnel that guarantees the connection is never left aborted.
- db.py: add is_integrity_error() — dual-backend (psycopg + sqlite3)
  constraint-violation classifier; replaces the fragile `"UNIQUE" in msg`
  string match that never matched Postgres' error text.
- Remove make_write_mcp_server: a never-run duplicate of the MCP write tools
  that had bit-rotted (wrong file_update arg order + FK-violating hardcoded
  actor). Live writes go through oauth_mcp_app, which is correct.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 21:10:12 +00:00
overseer 80635ce011 fix: drop NOT NULL on user_id FK columns for ON DELETE SET NULL
audit_log.user_id, change_requests.submitted_by, reviews.reviewer_id
were NOT NULL but had ON DELETE SET NULL — PG can't set a NOT NULL
column to NULL on delete, so user deletion failed. Migration now drops
NOT NULL before altering the FK constraint.
2026-06-25 14:47:37 +00:00
overseer 9bb89ee62f feat: allow user deletion — FK constraints ON DELETE SET NULL
- schema.sql: all user_id FKs now ON DELETE SET NULL (was RESTRICT)
- migrate_user_fk_set_null.py: drop+readd constraints on existing DBs
- entrypoint.sh: runs migration automatically on startup (non-fatal)
- db.py: rollback moved before FK check (cleanup)

Deleting a user now nullifies their references in audit_log,
project_context, context_files, change_requests, reviews instead of
blocking with a 409.
2026-06-25 14:43:58 +00:00
overseer bc43e9a8d1 fix: rollback aborted PG transaction on user delete FK violation
PostgreSQL enters 'current transaction is aborted' state after a
ForeignKeyViolation. Without rollback(), every subsequent query on the
shared connection fails with 'commands ignored until end of transaction
block'. Now both db.py and server.py rollback on the error path.
2026-06-25 14:29:00 +00:00
overseer d2c6906c4f ui: show activate OR inactivate based on user state (not both) 2026-06-25 14:24:37 +00:00
overseer 570b7d1dba ui: move activate/inactivate/delete to user list table actions column
- New actions column (right of state) with inline buttons per row
- Manage form keeps save user + clear only (no duplicate action buttons)
- Themed to match admin dialog: JetBrains Mono, uppercase, square borders
- Delete uses danger styling
2026-06-25 14:19:11 +00:00
overseer 451732c867 ui: OAuth client list scopes button matches admin theme 2026-06-25 14:07:32 +00:00
overseer 9e85c1b8ec ops: prevent 502 from ctxd without postgres
- entrypoint: wait for DATABASE_URL (CTXD_PG_WAIT_SECONDS) with clear fatal message
- scripts/deploy.sh: postgres healthy then force-recreate ctxd
- compose + README: ban --no-deps ctxd as default; 502 troubleshooting
2026-06-25 13:59:51 +00:00
overseer 59609f93c4 fix: MCP OAuth discovery for ChatGPT (RFC 9728 /mcp PRM, WWW-Authenticate)
- scopes_supported on protected-resource metadata
- /.well-known/oauth-protected-resource/mcp (path-prefix match)
- 401 on /mcp points resource_metadata at PRM URL; advertise scopes
- Prefer client_secret_basic in AS metadata (ChatGPT connector quirk)
- README: does not implement OAuth / 502 / 404 troubleshooting
2026-06-25 13:53:31 +00:00
overseer 07cf223d16 docs: ChatGPT MCP connector checklist in README 2026-06-25 13:46:22 +00:00
overseer ce1c0a175f docs: admin OAuth scope caps in SKILL.md 2026-06-25 13:13:31 +00:00
overseer 1c9d8f7648 feat: OAuth client scope assignment in admin panel
- Create client: ctxd.read / ctxd.write checkboxes
- Client list: show scopes, edit via PATCH /oauth/clients/:id
- Authorize grants intersection of client allowed scopes and request
- CLI oauth-client-create --scope; DCR default ctxd.read ctxd.write
2026-06-25 13:13:25 +00:00
overseer 87f02eb4d1 test: hit /mcp in tools/list fallback branch 2026-06-25 12:46:05 +00:00
overseer 289c6b9300 refactor: public OAuth MCP at /mcp (readonly path is alias)
- OAuth read+write on /mcp; API key on /mcp still full internal tools (LAN)
- /readonly/mcp and /oauth/mcp remain OAuth aliases
- OAuth metadata and connector_url point to /mcp
- README + Traefik template: route Host without blocking /mcp
2026-06-25 12:45:59 +00:00
overseer 12b60ee8c7 feat: unified OAuth MCP connector (read+write on /readonly/mcp)
Scope-gated tools on one Streamable HTTP URL; /oauth/mcp alias.
Reconnect Claude once with ctxd.read ctxd.write.
2026-06-25 12:44:01 +00:00
overseer e3567f649f fix: OAuth write MCP via SSE; require container recreate after build
- Route public write at GET /write/sse and POST /write/messages (ctxd.write)
- Always require write token on /write/messages (was optional)
- Remove debug tracing; document SSE write surface in SKILL.md
- Add scripts/test_write_mcp.py for local OAuth write smoke test
2026-06-25 12:38:50 +00:00
overseer fe63ad350e fix: reset SERIAL sequences after SQLite migration to prevent UniqueViolation
Migration script now resets SERIAL sequences (audit_log, context_files, etc.)
to max(existing_id) after copying rows. Without this, the sequence is still at
its post-schema-creation value and every INSERT hits a duplicate key error.
2026-06-25 10:50:35 +00:00
overseer 364c7795d4 fix: entrypoint.sh set -e crash, __main__.py PostgreSQL check, cli.py seed placeholders
- entrypoint.sh: use NEEDS_INIT variable instead of exit codes (set -e was killing the script)
- __main__.py: skip db_path.exists() check when using PostgreSQL
- cli.py: use dual-backend placeholders for _seed_context (was using SQLite ? syntax)
- cli.py: use 'admin' instead of 'system' for seed updated_by (FK constraint)
2026-06-25 00:20:55 +00:00
overseer b9f911994d refactor: rename container from dossier to ctxd
- docker-compose.yml: service name dossier → ctxd, container_name dossier → ctxd
- README.md, SKILL.md, LLM.txt: all docker exec/logs references updated
- Hermes skill files: all references updated
2026-06-24 23:19:08 +00:00
overseer b91d03a6cd docs: add LLM.txt for automated installation and deployment assistance 2026-06-24 22:55:05 +00:00
overseer fc1a2f5103 feat: PostgreSQL migration, OAuth write MCP, Streamable HTTP, env-driven config, admin UI, landing page
- Migrate database from SQLite to PostgreSQL 16 (dual-backend with SQLite fallback)
- Add Streamable HTTP MCP transport (replaces SSE): /readonly/mcp, /write/mcp, /mcp
- Add OAuth ctxd.write scope and public write MCP surface
- Add ctxd.write token validation (write-scoped tokens only on /write/mcp)
- Add env-driven configuration (.env file with env var precedence over ctxd.yaml)
- Add PostgreSQL to docker-compose.yml with healthcheck
- Add psycopg dependency, migration script (SQLite → PostgreSQL)
- Add admin UI: projects tab with typed-confirm delete, user management (list/manage subtabs)
- Add OAuth client management: create, list, revoke (UI, CLI, API)
- Add user active/inactive lifecycle (PATCH/DELETE APIs)
- Add public landing page with themed login form (cookie-based session)
- Add get_client_guide MCP tool (locked LLM-CLIENT.MD in ctxd-docs project)
- Add DELETE /projects/<id> endpoint with cascading deletes
- Add project_delete to db.py with FK ON DELETE SET NULL for audit_log
- Add cookie-based session auth (ctxd_session cookie on login)
- Add landing.html (public host) vs ui.html (internal dashboard)
- Add schema_sqlite.sql for SQLite fallback
- Add auth_password.py (PBKDF2-SHA256 password hashing)
- Add .env.example template with all documented env vars
- Add README.md with full setup, config, API, CLI, and troubleshooting docs
- Add SKILL.md (canonical LLM client guide, lives in project root)
- Update Traefik template: route everything except /mcp
- Update OAuth discovery: advertise ctxd.write scope, /readonly/mcp resource
- Update Hermes MCP config: /mcp endpoint with Bearer header
- Remove DB-level audit_log triggers (conflict with FK ON DELETE SET NULL)
- Remove SSE transport code (replaced by Streamable HTTP)
- Untrack __pycache__ and data/ctxd.db from git
2026-06-24 22:50:54 +00:00
overseer a9ccfa2694 Stop tracking ctxd.db-shm and ctxd.db-wal and add to .gitignore 2026-06-24 11:12:33 +00:00
overseer 5a0aa2d4fe Stop tracking ctxd.yaml and add to .gitignore 2026-06-24 10:57:53 +00:00
46 changed files with 5847 additions and 405 deletions
+25
View File
@@ -0,0 +1,25 @@
# Runtime CTXD data/state
/data/oauth_state.json
/data/web_sessions.json
/data/.ctxd.yaml.swp
/data/ctxd.db*
/data/ctxd.db-shm
/data/ctxd.db-wal
/data/ctxd.yaml
/data/snapshots/
/data/projects/
/data/users/
# Environment files (contains secrets)
.env
app/.env
# PostgreSQL data volume
/data/pg/
# Python cache
__pycache__/
*.py[cod]
# Build artifacts (regenerated by `pip install -e`)
*.egg-info/
+108
View File
@@ -0,0 +1,108 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What CTXD is
A single-process daemon that stores per-project "context dossiers" (multiple `.MD` files:
`CONTEXT.MD`, `DECISIONS.MD`, `RUNBOOKS.MD`, `PROMPTS.MD`, `GLOSSARY.MD`) and serves them to
LLM harnesses (Claude, ChatGPT, Hermes, Codex, Cursor) over **MCP via Streamable HTTP**, plus a
web UI and REST API. `CONTEXT.MD` can be synced to a repo as `AGENTS.md` with symlinks
(`CLAUDE.md`, `.cursorrules`, `CODEX.md`). The full user/operator guide is `README.md`; this file
is the orientation for editing the code.
All source lives in `app/src/ctxd/`. Run commands from `app/`.
## Architecture (the parts that need multiple files to grasp)
- **One ASGI app multiplexes three protocols.** `server.py``CombinedApp` (~line 1455) dispatches
every request to: REST + Web UI, the OAuth 2.0 authorization server, and the MCP endpoints. There
is no separate service per surface — "MCP is 502" and "web UI is 502" are the same process being
down. `serve_sync(cfg)` boots it under uvicorn.
- **Two MCP transports.** Streamable HTTP is the public one (paths in `MCP_STREAMABLE_PATHS` =
`/mcp`, `/readonly/mcp`, `/oauth/mcp`, all served by `CombinedApp`). `mcp_stdio.py` is a *separate*
stdin/stdout JSON-RPC server Hermes spawns directly — keep tool definitions in sync between the two
when adding tools.
- **MCP tools are built in two functions, gated by scope.** `make_mcp_server(cfg, readonly, oauth_scoped)`
exposes read tools (+ everything when API-key/LAN); `make_write_mcp_server(cfg)` exposes the write
set (`update_file`, `set_project_tags`, `sync_to_project`). OAuth `ctxd.read` vs `ctxd.write` scopes
decide what a token sees. To add an MCP tool, edit the `list_tools`/`call_tool` handlers inside these
builders.
- **`db.py` is plain functions over a `conn`, no ORM, dual-backend.** PostgreSQL is primary; SQLite
(`$CTXD_HOME/ctxd.db`) is the fallback when `DATABASE_URL` is empty (`cfg.use_postgres`). The same
query strings run on both via `_is_pg(conn)` and the placeholder helper `_ph(conn, n)`. **`schema.sql`
(PG) and `schema_sqlite.sql` must be kept in lockstep** — a table added to one must be added to the other.
- **OAuth state and web sessions are file-based JSON, not in the DB.** `OAuthStore` (`oauth_state.json`)
and `WebSessionStore` (`web_sessions.json`) live in `$CTXD_HOME` (`/data` in the container). They
survive DB swaps but are *not* covered by `pg_dump` — back them up separately.
- **Metadata headers are computed, never stored.** `build_metadata_header()` prepends the header on
read; `strip_metadata_header()` removes it before persisting. Don't store headers in the DB.
- **File paths are normalized** to uppercase with `.MD` (`normalize_file_path`). `CONTEXT.MD` is the
minimum file and cannot be deleted; in the `ctxd-docs` project, `CONTEXT.MD` and `LLM-CLIENT.MD` are
also locked against update/delete.
- **Writes are version-checked.** `file_update`/`context_update` take `base_version`; a mismatch is a
`409 conflict`. Every mutating op also writes an append-only `audit_log` row and may take a rotating
snapshot (`SNAPSHOT_MIN_KEEP`/`MAX_KEEP`).
### Latent / not wired up
`schema.sql` and `db.py` define a full collaborative-review flow — `user_workspaces``workspace_files`
`change_requests``reviews`, with `workspace_fork`, `workspace_submit`, `change_request_approve`,
etc. As of now this is **DB-layer only**: it is exposed through neither `server.py` (REST/MCP) nor
`cli.py`. Treat it as scaffolding, not a live feature.
## Entry points
`pyproject.toml` defines two console scripts (and `python -m ctxd` dispatches the same way via
`__main__.py`, choosing CLI vs daemon by the first arg):
- `dossier <command>` → CLI (`cli.py`, `cli_entry`). In production these run inside the container:
`docker exec ctxd dossier <command>`.
- `ctxd` → starts the daemon (`daemon_entry``serve_sync`).
## Build / run / deploy
Everything runs from `app/`. There is **no test runner or linter configured**`scripts/test_*.py`
are standalone MCP smoke scripts, not a suite.
```bash
# Local dev (SQLite, no Docker)
cd app
pip install -e ".[mcp]"
export CTXD_HOME=./dev-data
python -m ctxd init # initialize DB
python -m ctxd # serve → http://localhost:9091
# Production deploy (Docker + bundled PostgreSQL 16)
./scripts/deploy.sh # builds ctxd, starts postgres, waits for healthy, recreates ctxd, smoke-tests /status
# MCP smoke tests against a running server
python scripts/test_unified_mcp.py
python scripts/test_write_mcp.py
# Health
curl http://localhost:9091/status # → {"status":"ok", ...}
```
**Deploy gotcha (the #1 source of public 502s):** `ctxd` depends on the `postgres` container being up
first. Always use `./scripts/deploy.sh` (or `docker compose up -d`, which starts both). **Never**
`docker compose up -d --no-deps ctxd` and **never** `docker restart ctxd` after a code change — the
former crash-loops without the DB, the latter runs the old image. After editing code you must rebuild
(`docker compose build ctxd`) before recreating.
## Config
All config is env-driven through `config.py` (`CtxConfig`), precedence **env var > `data/ctxd.yaml` >
default**. Key switches: `DATABASE_URL` (empty ⇒ SQLite fallback), `CTXD_HOME` (`/data` in container),
`CTXD_AUTH_ENABLED` + `CTXD_API_KEY` (shared key for LAN/Hermes = full MCP tools), `OAUTH_ENABLED` +
`OAUTH_ISSUER` (must be set together or the app won't start cleanly). The full table is in `README.md`.
## Conventions when changing code
- Adding a context-data operation: write the `conn`-taking function in `db.py` (handle both backends
via `_is_pg`/`_ph`), then expose it in the relevant surface(s) — `server.py` for REST/MCP, `cli.py`
for CLI, and `mcp_stdio.py` if Hermes needs it.
- Adding a column/table: edit **both** `schema.sql` and `schema_sqlite.sql`; for live PG instances add
a guarded migration (see `_migrate_pg` in `db.py` and the standalone `migrate_*.py` scripts).
- Don't persist metadata headers; don't bypass `base_version` checking on writes; keep every mutation
audit-logged.
+251
View File
@@ -0,0 +1,251 @@
# CTXD — Context Dossier
# LLM Installation & Deployment Guide
# Feed this file to an LLM to assist with setup and deployment.
## What is CTXD?
CTXD (Context Dossier) is a single source of truth for multi-harness project context. It serves one canonical AGENTS.md per project to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP. It runs as two Docker containers: a PostgreSQL 16 database and a Python ASGI daemon.
## Prerequisites
- Docker 24+ and Docker Compose v2+
- A reverse proxy with TLS (Traefik recommended; Caddy or nginx also work)
- Linux host with ~1GB free RAM
- (Optional) Existing PostgreSQL 14+ instance if not using the bundled one
## Quick Deployment
1. Clone the repository:
git clone ssh://git@code.cubecraftlabs.com:2288/CubeCraft-Creations/CTXD.git
cd CTXD/app
2. Copy the environment template:
cp .env.example .env
3. Generate secrets and fill in .env:
python3 -c "import secrets; print(secrets.token_urlsafe(32))" # Use for CTXD_API_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))" # Use for OAUTH_APPROVAL_KEY
4. Edit .env with your values. Required variables:
- DATABASE_URL — PostgreSQL connection string
- POSTGRES_PASSWORD — Password for the bundled PG container
- CTXD_API_KEY — Shared key for internal MCP and HTTP auth
- OAUTH_ENABLED=true
- OAUTH_ISSUER=https://your-domain.com
- OAUTH_APPROVAL_KEY — Key for approving OAuth authorizations
5. Build and start:
docker compose up -d
6. Wait for PostgreSQL to be healthy and the daemon to start:
docker compose ps
# Both ctxd-postgres and dossier should show "healthy" / "running"
7. Set the admin password:
docker exec ctxd dossier user-set-password admin -p "your-admin-password"
8. Verify:
curl http://localhost:9091/status
# Expected: {"status": "ok", "db": "/data/ctxd.db"}
9. Access the Web UI:
http://<server-ip>:9091/
Sign in with admin / your-admin-password
## Environment Variables Reference
All config is driven by environment variables in .env. Precedence: env var > ctxd.yaml > defaults.
### Database
- DATABASE_URL — PostgreSQL connection string. If empty, falls back to SQLite.
- POSTGRES_USER — PostgreSQL user for bundled container (default: ctxd)
- POSTGRES_PASSWORD — PostgreSQL password for bundled container
- POSTGRES_DB — PostgreSQL database name (default: ctxd)
### Server
- CTXD_HOST — Bind address (default: 0.0.0.0)
- CTXD_PORT — Listen port (default: 9091)
- CTXD_HOME — Data directory inside container (default: /data)
- LOG_LEVEL — Uvicorn log level: debug, info, warning, error (default: info)
### Auth
- CTXD_AUTH_ENABLED — Enable authentication globally (default: false)
- CTXD_API_KEY — Shared API key for Hermes/internal MCP and HTTP auth
- CTXD_EXTERNAL_READONLY_KEY — Legacy ?key= on read-only MCP (migration only)
### OAuth
- OAUTH_ENABLED — Enable OAuth authorization server (default: false)
- OAUTH_ISSUER — Public URL used in OAuth discovery metadata
- OAUTH_APPROVAL_KEY — Fallback key for /oauth/authorize approval
- OAUTH_APPROVAL_USER_ID — User ID for attributing approvals (default: admin)
- OAUTH_ACCESS_TOKEN_TTL — Access token lifetime in seconds (default: 3600)
- OAUTH_REFRESH_TOKEN_TTL — Refresh token lifetime in seconds (default: 2592000)
### Web Sessions
- WEB_SESSION_TTL — Session cookie lifetime in seconds (default: 604800)
### Snapshots
- SNAPSHOT_MIN_KEEP — Minimum snapshots per project (default: 5)
- SNAPSHOT_MAX_KEEP — Maximum snapshots before rotation (default: 25)
## MCP Endpoints
CTXD exposes three MCP surfaces via Streamable HTTP:
1. /readonly/mcp — Read-only tools (OAuth ctxd.read or API key)
Tools: list_projects, get_project_context, search_context, get_project_tags, list_files, get_file, get_client_guide
2. /write/mcp — Write tools (OAuth ctxd.write only)
Tools: update_file, set_project_tags, sync_to_project, get_client_guide
3. /mcp — Internal full MCP (shared API key only, not exposed publicly)
All read + write tools plus auto_generate_tags and get_user_profile
## Connecting LLM Clients
### Claude Desktop
Connector URL: https://your-domain.com/readonly/mcp
Claude auto-discovers OAuth metadata and registers via DCR.
For write access, use: https://your-domain.com/write/mcp
### Hermes Agent
Edit ~/.hermes/config.yaml:
mcp_servers:
dossier:
url: http://<server-ip>:9091/mcp
timeout: 30
headers:
Authorization: "Bearer your-api-key"
Restart Hermes after changing config.
### Other MCP Clients
1. Register an OAuth client: POST /oauth/register with your redirect URI
2. Authorize: GET /oauth/authorize (admin must approve)
3. Exchange code: POST /oauth/token
4. Connect to /readonly/mcp or /write/mcp with Bearer token
## Traefik Configuration
Route everything except the internal MCP endpoint:
rule: Host(`ctxd.yourdomain.com`) && !Path(`/mcp`)
This exposes: landing page, dashboard, REST API, OAuth, read-only MCP, write MCP.
Only /mcp (internal, API key) is blocked.
## CLI Commands
All commands run inside the container:
docker exec ctxd dossier <command>
Commands:
init Initialize database
project-create <id> [--display-name N] Create a project
project-list List projects
read <project_id> Print context to stdout
edit <project_id> Open in $EDITOR
file-list <project_id> List context files
file-read <project_id> <file_path> Read a single file
sync <project_id> [path] Sync AGENTS.md to project root
search "query" Full-text search
audit [--limit N] Show audit log
user-list List users
user-create <id> --display-name "Name" [--password "pw"]
user-set-password <id> -p "password"
oauth-client-create [-n "Name"] [--redirect-uri URI]
oauth-client-list List OAuth clients
oauth-client-revoke <client_id> Revoke client and invalidate tokens
import-vault <path> Import from Obsidian vault
## Admin UI
Sign in as admin, click "admin" in the masthead. Three tabs:
1. oauth clients — Create and revoke OAuth clients
2. users — List, create, edit, activate, inactivate, delete users
3. projects — List and delete projects (typed-name confirmation required)
## Project Files
Each project has multiple context files:
- CONTEXT.MD — Canonical overview (synced as AGENTS.md, cannot be deleted)
- DECISIONS.MD — Architecture decisions and rationale
- RUNBOOKS.MD — Deploy, troubleshoot, operate procedures
- PROMPTS.MD — Project-specific prompts for different harnesses
- GLOSSARY.MD — Project-specific terms and acronyms
The compiled view (get_project_context) concatenates all files with metadata header.
## Locked Files
- CONTEXT.MD — Cannot be deleted from any project (minimum required file)
- CONTEXT.MD and LLM-CLIENT.MD — Fully locked in the ctxd-docs project (cannot update or delete)
## Backups
PostgreSQL backup:
docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup.sql
PostgreSQL restore:
cat backup.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd
Snapshots are automatic (before each context update, rotated min 5 / max 25 per project).
## Migrating from SQLite to PostgreSQL
If you started with SQLite:
1. Start PostgreSQL: docker compose up -d postgres
2. Run migration: docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
3. Set DATABASE_URL in .env and restart: docker compose up -d dossier
## Using an External PostgreSQL
1. Create database and user on your external PG instance
2. Set DATABASE_URL in .env to point to it
3. Start only the app: docker compose up -d dossier
(or scale postgres to 0: docker compose up -d --scale postgres=0)
## Troubleshooting
Login fails: Reset password with "docker exec ctxd dossier user-set-password admin -p newpass"
Use double quotes if password contains single quotes. Use single quotes for $ ` ! characters.
MCP 404: Traefik is not routing /readonly/mcp. Check router rule includes it.
MCP 401: Auth is working. Check OAuth token scope and expiry.
PostgreSQL connection fails: Check DATABASE_URL password matches POSTGRES_PASSWORD.
If PG data volume was initialized with a different password, reset it:
docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'newpass'"
Then update .env.
Container keeps restarting: Check "docker logs ctxd --tail 30".
Common causes: wrong PG password, OAUTH_ENABLED=true but OAUTH_ISSUER empty, missing CTXD_API_KEY.
## File Structure
CTXD/
├── .env Production environment (gitignored, contains secrets)
├── .env.example Template with all documented variables
├── .gitignore
├── README.md Full documentation
├── SKILL.md LLM client guide (canonical source for LLM-CLIENT.MD)
├── data/ Runtime data (gitignored)
│ ├── ctxd.yaml Fallback config (env vars take precedence)
│ ├── oauth_state.json OAuth clients, codes, tokens
│ ├── web_sessions.json Web UI sessions
│ ├── snapshots/ Point-in-time context backups
│ └── pg/ PostgreSQL data volume
└── app/ Application source
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
└── src/ctxd/
├── config.py Env-driven config
├── db.py Database layer (PostgreSQL + SQLite)
├── schema.sql PostgreSQL schema
├── schema_sqlite.sql SQLite schema fallback
├── server.py ASGI: HTTP + MCP + OAuth
├── cli.py CLI commands
├── ui.html Web UI dashboard
├── landing.html Public landing page
├── auth_password.py PBKDF2 password hashing
└── migrate_sqlite_to_pg.py Migration script
+661
View File
@@ -0,0 +1,661 @@
# CTXD — Context Dossier
A single source of truth for multi-harness project context. One canonical `AGENTS.md` per project, served to Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client via Streamable HTTP.
## Overview
CTXD solves context sprawl: when you work across multiple LLM harnesses (Claude Desktop, Claude Code, Codex CLI, Cursor, Copilot, Hermes), each has its own context file convention (`CLAUDE.md`, `.cursorrules`, `CODEX.md`, etc.). Without a canonical source, these drift independently.
CTXD provides:
- **Multi-file context per project** — `CONTEXT.MD`, `DECISIONS.MD`, `RUNBOOKS.MD`, `PROMPTS.MD`, `GLOSSARY.MD`
- **Compiled view** — all files concatenated with metadata header, served as a single document to agents
- **Sync to repos** — writes `CONTEXT.MD` as `AGENTS.md` + symlinks (`CLAUDE.md`, `.cursorrules`, `CODEX.md``AGENTS.md`)
- **Version-checked writes** — optimistic concurrency with `base_version` to prevent silent overwrites
- **OAuth 2.0 authorization server** — DCR, Authorization Code + PKCE, `ctxd.read` and `ctxd.write` scopes
- **Streamable HTTP MCP** — single-endpoint transport for read-only and write surfaces
- **Web UI** — per-user password login, admin panel, project/file management
- **PostgreSQL backend** — with SQLite fallback for local dev
- **Append-only audit log** — every read, write, create, delete, sync, and search is logged
- **Point-in-time snapshots** — automatic version snapshots with rotation (min 5, max 25 per project)
- **Full-text search** — PostgreSQL `tsvector` with GIN index (FTS5 on SQLite fallback)
## Architecture
```
Context Dossier (container: ctxd, 0.0.0.0:9091)
├── PostgreSQL 16 (container: ctxd-postgres) # Primary DB
├── /data # Config, OAuth state, web sessions
│ ├── ctxd.yaml # Fallback config (env vars take precedence)
│ ├── oauth_state.json # OAuth clients, codes, tokens
│ ├── web_sessions.json # Per-user web UI sessions
│ └── snapshots/ # Point-in-time context backups
├── Streamable HTTP MCP:
│ ├── /mcp (OAuth ctxd.read + ctxd.write; API key on LAN = full tools)
│ ├── /readonly/mcp (alias → same OAuth behavior)
│ └── /oauth/mcp (alias)
├── OAuth Authorization Server:
│ ├── /.well-known/oauth-authorization-server # Discovery
│ ├── /.well-known/oauth-protected-resource # Resource metadata
│ ├── /oauth/register # Dynamic Client Registration
│ ├── /oauth/authorize # Authorization + PKCE
│ └── /oauth/token # Token + refresh
├── Web UI + REST API (/) # Dashboard, admin, projects, files
└── Landing page (public host only) # Themed login form
```
## Quick Start
### Prerequisites
- Docker and Docker Compose
- A reverse proxy with TLS (Traefik, Caddy, nginx) for public exposure
- (Optional) An existing PostgreSQL 14+ instance if not using the bundled one
### 1. Clone and configure
```bash
cd /mnt/ai-storage/projects/ctxd/app
cp .env.example .env
```
Edit `.env` with your values:
```bash
# Database
DATABASE_URL=postgresql://ctxd:your-password@postgres:5432/ctxd
POSTGRES_USER=ctxd
POSTGRES_PASSWORD=your-strong-password
# Server
CTXD_HOST=0.0.0.0
CTXD_PORT=9091
CTXD_HOME=/data
# Auth
CTXD_AUTH_ENABLED=true
CTXD_API_KEY=your-api-key-here # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
# OAuth
OAUTH_ENABLED=true
OAUTH_ISSUER=https://ctxd.yourdomain.com
OAUTH_APPROVAL_KEY=your-approval-key # Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
```
### 2. Build and start
```bash
# Recommended: build + postgres + recreate ctxd (avoids 502 when PG was never started)
chmod +x scripts/deploy.sh
./scripts/deploy.sh
# Or manual (always include postgres when DATABASE_URL uses host "postgres"):
docker compose up -d
```
This starts:
- `ctxd-postgres` — PostgreSQL 16 (Alpine)
- `ctxd` — CTXD daemon (web UI + MCP + OAuth + REST API)
**After code changes**, use `./scripts/deploy.sh` or `docker compose up -d` — not `docker restart ctxd` alone (old image) and **not** `docker compose up -d --no-deps ctxd` (skips postgres → crash loop → public **502**).
### 3. Verify
```bash
# Health check
curl http://localhost:9091/status
# → {"status": "ok", "db": "/data/ctxd.db"}
# List projects (requires API key)
curl http://localhost:9091/projects -H "Authorization: Bearer your-api-key"
```
### 4. Set admin password
```bash
docker exec ctxd dossier user-set-password admin -p "your-admin-password"
```
> **Shell quoting:** Use double quotes (`"`) if the password contains single quotes (`'`). Use single quotes (`'`) if it contains `$`, backticks, or `!`. The failure mode is shell expansion, not PBKDF2.
### 5. Access the Web UI
- **LAN:** `http://<server-ip>:9091/`
- **Public (via Traefik):** `https://ctxd.yourdomain.com/`
Sign in with `admin` / your password.
## Configuration
### Environment Variables
All config is driven by environment variables. A `ctxd.yaml` file in `/data` can override built-in defaults, but env vars always take precedence.
**Precedence:** env var > `ctxd.yaml` > built-in default
| Variable | Default | Description |
|----------|---------|-------------|
| **Database** | | |
| `DATABASE_URL` | *(empty)* | PostgreSQL connection string. If empty, falls back to SQLite at `$CTXD_HOME/ctxd.db` |
| `POSTGRES_USER` | `ctxd` | PostgreSQL user (for bundled PG container) |
| `POSTGRES_PASSWORD` | `ctxd_local_dev` | PostgreSQL password (for bundled PG container) |
| `POSTGRES_DB` | `ctxd` | PostgreSQL database name (for bundled PG container) |
| **Server** | | |
| `CTXD_HOST` | `0.0.0.0` | Bind address |
| `CTXD_PORT` | `9091` | Listen port |
| `CTXD_HOME` | `~/.ctx` | Data directory (inside container: `/data`) |
| `LOG_LEVEL` | `info` | Uvicorn log level (`debug`, `info`, `warning`, `error`) |
| **Auth** | | |
| `CTXD_AUTH_ENABLED` | `false` | Enable authentication globally |
| `CTXD_API_KEY` | *(empty)* | Shared API key for Hermes/internal MCP + HTTP auth |
| `CTXD_EXTERNAL_READONLY_KEY` | *(empty)* | Legacy `?key=` on read-only MCP (migration only) |
| **OAuth** | | |
| `OAUTH_ENABLED` | `false` | Enable OAuth authorization server |
| `OAUTH_ISSUER` | *(empty)* | Public URL (used in OAuth discovery metadata) |
| `OAUTH_APPROVAL_KEY` | *(empty)* | Fallback approval key for `/oauth/authorize` |
| `OAUTH_APPROVAL_USER_ID` | `admin` | Which user ID to attribute OAuth approvals to |
| `OAUTH_ACCESS_TOKEN_TTL` | `3600` | Access token lifetime in seconds |
| `OAUTH_REFRESH_TOKEN_TTL` | `2592000` | Refresh token lifetime in seconds (30 days) |
| **PostgreSQL (container)** | | |
| `CTXD_PG_WAIT_SECONDS` | `120` | Entrypoint: max wait for DB before exit (when `DATABASE_URL` set) |
| `CTXD_PG_WAIT_INTERVAL` | `2` | Seconds between connection attempts |
| **Web Sessions** | | |
| `WEB_SESSION_TTL` | `604800` | Session cookie lifetime in seconds (7 days) |
| **Snapshots** | | |
| `SNAPSHOT_MIN_KEEP` | `5` | Minimum snapshots retained per project |
| `SNAPSHOT_MAX_KEEP` | `25` | Maximum snapshots before rotation |
### Using an External PostgreSQL
To use an external PostgreSQL instead of the bundled container:
1. Create a database and user on your external PG instance
2. Set `DATABASE_URL` in `.env` to point to it
3. Start only the app (no bundled postgres): `docker compose up -d --scale postgres=0 ctxd`
Ensure `DATABASE_URL` points at your external host (not `postgres`). The entrypoint skips the compose-network wait when the URL is reachable.
### Fallback to SQLite
If `DATABASE_URL` is empty or not set, CTXD falls back to SQLite at `$CTXD_HOME/ctxd.db`. This is useful for local development or single-user deployments that don't need PostgreSQL features.
## MCP Surfaces
CTXD exposes MCP via Streamable HTTP on **`/mcp`** (single public connector):
| Endpoint | Auth | Scope | Tools |
|----------|------|-------|-------|
| `/mcp` | OAuth bearer | `ctxd.read` / `ctxd.write` | Scope-gated read + write tools |
| `/mcp` | Shared API key (LAN/Hermes) | *(full)* | All tools including `get_user_profile`, `auto_generate_tags` |
| `/readonly/mcp`, `/oauth/mcp` | OAuth (aliases) | same as `/mcp` | Backward-compatible URLs |
### Connecting an LLM Client
**Claude Desktop / Claude Web:**
```
Connector URL: https://ctxd.yourdomain.com/mcp
```
Claude auto-discovers OAuth metadata and registers via DCR. Request `scope=ctxd.read ctxd.write` for write access.
**ChatGPT (MCP connector — recommended; no Custom GPT required):**
ChatGPT can attach a **remote MCP server** in **Developer mode** (Plus/Pro and higher tiers for custom connectors). Use the same public URL as Claude — OAuth on `/mcp`, not REST Actions or a shared API key.
1. **Server** — deploy current CTXD and expose the public host (Traefik must route `/mcp` and `/oauth/*`; do not block `/mcp`):
```bash
cd app
./scripts/deploy.sh
```
Smoke: `curl -sS https://ctxd.yourdomain.com/.well-known/oauth-authorization-server | head`
2. **ChatGPT** — Settings → **Connectors** → enable **Developer mode** → **Add connector** (MCP / custom remote) → **Server URL:**
```
https://ctxd.yourdomain.com/mcp
```
Start OAuth and copy ChatGPTs **callback / redirect URL** exactly.
3. **CTXD admin** — Web UI → **admin** → **oauth clients**:
- After DCR, the client may appear automatically; otherwise **create client** with ChatGPTs redirect URI.
- Set **allowed scopes** to **`ctxd.read`** and **`ctxd.write`** (create form or **scopes** → save on an existing row).
4. **Authorize** — complete the browser approval (sign in as CTXD admin in that browser, or use the OAuth approval key). Tokens are capped by the clients allowed scopes.
5. **Chat** — enable the CTXD connector for the conversation. Tools include `list_projects`, `get_project_context`, `search_context`, `get_file`, `update_file`, etc. Call **`get_client_guide`** first in a new session.
**Optional CLI pre-register** (if ChatGPT asks for client credentials before DCR):
```bash
docker exec ctxd dossier oauth-client-create -n "ChatGPT MCP" \
--redirect-uri 'PASTE_CHATGPT_CALLBACK_URL' \
--scope "ctxd.read ctxd.write"
```
| If connect fails | Check |
|------------------|--------|
| **“does not implement OAuth”** | Public host must return **200** JSON (not 502/404) for `/.well-known/oauth-protected-resource/mcp` and `/.well-known/oauth-authorization-server`. Traefik must proxy **all** paths to CTXD `:9091`. Set `OAUTH_ENABLED=true` and `OAUTH_ISSUER=https://ctxd.cubecraftlabs.com`. |
| 502 on well-known URLs | Backend down or wrong Traefik service — fix routing before OAuth can work |
| 404 on `/mcp` | Router not forwarding `/mcp` to CTXD (often a stale `!Path(/mcp)` rule) |
| 401 on `/mcp` | Re-authorize; access token may have expired |
| Redirect mismatch | Callback URL in admin must match ChatGPTs string exactly |
| `invalid_request` on authorize | CTXD requires PKCE **S256** (`code_challenge` + `code_challenge_method=S256`) |
| Stale routes / old code | `./scripts/deploy.sh` or `docker compose up -d --force-recreate ctxd` after postgres is healthy (not `restart` alone; never `--no-deps` unless PG is already up) |
| No write tools | Client scopes include `ctxd.write`; re-authorize after changing scopes |
**Not required for ChatGPT MCP:** Custom GPT, OpenAPI Actions, or `CTXD_API_KEY` in ChatGPT (public path is OAuth). Hermes continues to use LAN `http://<server-ip>:9091/mcp` with the API key.
**Hermes Agent:**
```yaml
# ~/.hermes/config.yaml
mcp_servers:
dossier:
url: http://<server-ip>:9091/mcp
timeout: 30
headers:
Authorization: "Bearer your-api-key"
```
**Other MCP clients (Codex, Cursor, etc.):**
- Register an OAuth client via `POST /oauth/register` with your redirect URI
- Connect to **`/mcp`** with `scope=ctxd.read ctxd.write`
- Use the access token as `Authorization: Bearer <token>`
### MCP Tool Reference
#### Read tools (require `ctxd.read` on `/mcp`)
| Tool | Args | Returns |
|------|------|---------|
| `get_client_guide` | *(none)* | Locked `LLM-CLIENT.MD` guide — **call this first** |
| `list_projects` | *(none)* | All projects with version numbers |
| `get_project_context` | `project_id` | Compiled markdown of all context files |
| `search_context` | `query`, `limit?` | FTS results across all projects |
| `get_project_tags` | `project_id` | Metadata tags for a project |
| `list_files` | `project_id` | All context files in a project |
| `get_file` | `project_id`, `file_path` | Single file with metadata header |
#### Write tools (require `ctxd.write` on `/mcp`)
| Tool | Args | Returns |
|------|------|---------|
| `update_file` | `project_id`, `file_path`, `content`, `base_version` | `{"ok": true, "new_version": N}` or conflict error |
| `set_project_tags` | `project_id`, `tags[]` | `{"ok": true, "tags": [...]}` |
| `sync_to_project` | `project_id` | Writes `CONTEXT.MD` as `AGENTS.md` + symlinks to project root |
### Locked Files
| File | Scope | Protection |
|------|-------|------------|
| `CONTEXT.MD` | All projects | Cannot delete — minimum required file |
| `CONTEXT.MD` | `ctxd-docs` project only | Cannot update or delete |
| `LLM-CLIENT.MD` | `ctxd-docs` project only | Cannot update or delete |
## OAuth
### Scopes
| Scope | Grants |
|-------|--------|
| `ctxd.read` | Read-only MCP tools |
| `ctxd.write` | Write MCP tools (includes read) |
Request both: `scope=ctxd.read ctxd.write`
### Redirect URIs
| Platform | Redirect URI |
|----------|-------------|
| Claude Desktop | `https://claude.ai/api/mcp/auth_callback` |
| ChatGPT (MCP) | Paste callback URL from ChatGPT connector OAuth UI (per connector) |
| Claude Code | `http://localhost:5555/oauth/callback` |
| Codex CLI | `http://localhost:7777/oauth/callback` |
| Custom | Your app's documented OAuth callback |
### Managing OAuth Clients
**Via Admin UI:** `http://<server-ip>:9091/` → sign in as admin → **admin** → **oauth clients** tab
**Via CLI:**
```bash
# Create
docker exec ctxd dossier oauth-client-create -n "Claude Desktop" --redirect-uri https://claude.ai/api/mcp/auth_callback
# List
docker exec ctxd dossier oauth-client-list
# Revoke (invalidates all tokens for that client)
docker exec ctxd dossier oauth-client-revoke ctxd_xxxxxxxx
```
**Via API (admin session or API key):**
```bash
# Create
curl -X POST http://localhost:9091/oauth/clients \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d '{"client_name": "Claude Desktop", "redirect_uris": ["https://claude.ai/api/mcp/auth_callback"]}'
# List
curl http://localhost:9091/oauth/clients -H "Authorization: Bearer your-api-key"
# Revoke
curl -X DELETE http://localhost:9091/oauth/clients/ctxd_xxxxxxxx -H "Authorization: Bearer your-api-key"
```
## Public Exposure (Traefik)
### Router Rule
Route the public host to the backend (include `/mcp` — OAuth protects it):
```yaml
rule: Host(`ctxd.yourdomain.com`)
```
This exposes:
- Landing page (`GET /`)
- Login (`POST /auth/login`, `GET /auth/me`)
- Full Web UI dashboard (all REST API endpoints)
- OAuth (`/oauth/*`, `/.well-known/*`)
- Public MCP (`/mcp` — OAuth read + write)
Hermes uses the same `/mcp` path on LAN with the shared API key (not exposed via public OAuth).
### Landing Page Behavior
- **Not signed in** → themed landing page with login form
- **Signed in** (valid session cookie) → full dashboard
- Cookie: `ctxd_session` (SameSite=Lax, 7-day expiry)
- After login: cookie set + redirect to dashboard
A full Traefik template is at `app/templates/traefik-ctxd-readonly-mcp.yaml` in the `project-context-management` Hermes skill.
## Web UI
### Admin Panel
Sign in as admin → click **admin** in the masthead.
**Tabs:**
1. **oauth clients** — client list (revoke per row) + create client form
2. **users** — user list (id, name, role, active/inactive) + manage users (create, edit, activate, inactivate, delete)
3. **projects** — manage projects (list with remove button, typed-name confirmation for delete)
### Project Files
Each project has multiple context files:
| File | Purpose |
|------|---------|
| `CONTEXT.MD` | Canonical project overview (synced as `AGENTS.md` to repos) |
| `DECISIONS.MD` | Architecture decisions, rationale |
| `RUNBOOKS.MD` | Deploy, troubleshoot, operate procedures |
| `PROMPTS.MD` | Project-specific prompts for different harnesses |
| `GLOSSARY.MD` | Project-specific terms, acronyms |
The compiled view (`get_project_context`) concatenates all files with `## FILENAME` headers and a single metadata block at the top.
## CLI
All commands run inside the container:
```bash
docker exec ctxd dossier <command>
```
### Commands
```bash
# Initialize (auto-runs on first container start)
dossier init
# Projects
dossier project-create <project_id> [--display-name "Name"] [--description "Desc"]
dossier project-list
dossier read <project_id> # Print context to stdout
dossier edit <project_id> # Open in $EDITOR
# Context files
dossier file-list <project_id>
dossier file-read <project_id> <file_path>
# Sync
dossier sync <project_id> [path] # Set sync path and/or sync AGENTS.md
# Search
dossier search "query" # FTS across all projects
# Audit
dossier audit [--limit N]
# Users
dossier user-list
dossier user-create <user_id> --display-name "Name" [--password "pw"]
dossier user-set-password <user_id> -p "password"
# OAuth
dossier oauth-client-create [-n "Name"] [--redirect-uri URI]
dossier oauth-client-list
dossier oauth-client-revoke <client_id>
# Import
dossier import-vault <path> # Import from Obsidian vault
```
## REST API
All endpoints require `Authorization: Bearer <api_key>` or `Authorization: Bearer <session_token>` unless noted.
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/` | Web UI (LAN) or landing page (public host) |
| `GET` | `/status` | Health check (no auth) |
| `POST` | `/auth/login` | Web UI login → session token + cookie |
| `POST` | `/auth/logout` | Revoke session + clear cookie |
| `GET` | `/auth/me` | Current session identity |
| `GET` | `/users` | List users |
| `POST` | `/users` | Create user |
| `PATCH` | `/users/<id>` | Update user (admin) |
| `DELETE` | `/users/<id>` | Delete user (admin) |
| `POST` | `/users/<id>/password` | Set password (admin) |
| `GET` | `/oauth/clients` | List OAuth clients (admin) |
| `POST` | `/oauth/clients` | Register OAuth client (admin) |
| `DELETE` | `/oauth/clients/<id>` | Revoke OAuth client (admin) |
| `GET` | `/projects` | List all projects |
| `POST` | `/projects` | Create a project |
| `DELETE` | `/projects/<id>` | Delete a project (admin) |
| `GET` | `/projects/<id>/context` | Compiled context (all files) |
| `POST` | `/projects/<id>/context` | Update context (legacy single-file) |
| `GET` | `/projects/<id>/files` | List context files |
| `GET` | `/projects/<id>/files/<name>` | Read a single file |
| `POST` | `/projects/<id>/files` | Create a new file |
| `PUT` | `/projects/<id>/files/<name>` | Update a file (version-checked) |
| `DELETE` | `/projects/<id>/files/<name>` | Delete a file |
| `POST` | `/projects/<id>/migrate-files` | Migrate single-context to multi-file |
| `GET` | `/projects/<id>/snapshots` | List snapshots |
| `GET` | `/projects/<id>/tags` | Get metadata tags |
| `POST` | `/projects/<id>/tags` | Set metadata tags |
| `POST` | `/projects/<id>/sync` | Sync CONTEXT.MD as AGENTS.md |
| `POST` | `/projects/<id>/import` | Import raw text as context |
| `GET` | `/search?q=...` | Full-text search |
| `GET` | `/audit?limit=N` | Audit log |
| `GET` | `/.well-known/oauth-authorization-server` | OAuth discovery |
| `GET` | `/.well-known/oauth-protected-resource` | Resource metadata |
| `POST` | `/oauth/register` | Dynamic Client Registration |
| `GET/POST` | `/oauth/authorize` | Authorization endpoint |
| `POST` | `/oauth/token` | Token endpoint |
| `POST/GET/DELETE` | `/readonly/mcp` | Read-only MCP (Streamable HTTP) |
| `POST/GET/DELETE` | `/write/mcp` | Write MCP (Streamable HTTP) |
| `POST/GET/DELETE` | `/mcp` | Internal full MCP (API key only) |
## Backups
### PostgreSQL
```bash
# Backup
docker exec ctxd-postgres pg_dump -U ctxd ctxd > backup_$(date +%Y%m%d).sql
# Restore
cat backup_YYYYMMDD.sql | docker exec -i ctxd-postgres psql -U ctxd ctxd
```
### Snapshots
CTXD automatically takes point-in-time snapshots before each context update. Snapshots are stored as files in `/data/snapshots/<project_id>/` and rotated (min 5, max 25 per project).
## Migrating from SQLite to PostgreSQL
If you started with SQLite and want to move to PostgreSQL:
```bash
# 1. Start PostgreSQL
docker compose up -d postgres
# 2. Run the migration script (reads from /data/ctxd.db, writes to DATABASE_URL)
docker exec ctxd python3 -m ctxd.migrate_sqlite_to_pg
# 3. Set DATABASE_URL in .env and restart
docker compose up -d dossier
```
The migration handles all tables, rebuilds the FTS index, and skips orphaned rows with FK violations.
## Project Structure
```
/mnt/ai-storage/projects/ctxd/
├── .env # Production environment (gitignored)
├── .env.example # Template (committed)
├── .gitignore
├── SKILL.md # LLM client guide (canonical source)
├── README.md # This file
├── data/ # Runtime data (gitignored)
│ ├── ctxd.yaml # Fallback config (env vars take precedence)
│ ├── ctxd.db # SQLite DB (if no DATABASE_URL)
│ ├── pg/ # PostgreSQL data volume
│ ├── oauth_state.json # OAuth clients, codes, tokens
│ ├── web_sessions.json # Web UI sessions
│ └── snapshots/ # Point-in-time backups
└── app/ # Application source
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
├── .env # Same as root .env (symlinked or copied)
└── src/ctxd/
├── __init__.py
├── __main__.py # CLI/daemon entry point
├── config.py # Env-driven config with yaml fallback
├── db.py # Database layer (PostgreSQL + SQLite)
├── schema.sql # PostgreSQL schema
├── schema_sqlite.sql # SQLite schema (fallback)
├── server.py # ASGI app: HTTP + MCP + OAuth
├── cli.py # CLI commands
├── ui.html # Web UI dashboard
├── landing.html # Public landing page
├── auth_password.py # PBKDF2 password hashing
└── migrate_sqlite_to_pg.py # One-time migration script
```
## Development
### Local Development (SQLite, no Docker)
```bash
cd app
pip install -e ".[mcp]"
export CTXD_HOME=./dev-data
python -m ctxd init
python -m ctxd
# → http://localhost:9091
```
### Rebuilding After Code Changes
```bash
cd app
docker compose build
docker compose up -d --no-build
# Verify:
curl http://localhost:9091/status
docker logs ctxd --tail 20
```
### Key Conventions
- **Metadata headers** are dynamically generated on read, never stored in the DB
- **File paths** are normalized to uppercase with `.MD` extension
- **`CONTEXT.MD`** is the minimum required file — cannot be deleted from any project
- **Version checking** uses `base_version` parameter — mismatches return `409 conflict`
- **Audit log** is append-only at the application layer (every operation is logged)
- **OAuth state** (`oauth_state.json`) and **web sessions** (`web_sessions.json`) are file-based, not in PostgreSQL
## Troubleshooting
### Login fails with "invalid credentials"
```bash
# Reset admin password
docker exec ctxd dossier user-set-password admin -p "new-password"
```
If the password contains special characters, use the quoting that matches:
- `'` in password → use double quotes (`"`)
- `$`, `` ` ``, `!` in password → use single quotes (`'`)
### Login works on LAN but not on public host
The public host (`https://ctxd.yourdomain.com`) requires Traefik to route `/auth/login` and `/auth/me`. Check your Traefik router rule includes all paths (use `!Path(`/mcp`)` to block only the internal MCP).
### MCP connection fails
1. Check OAuth discovery: `curl https://ctxd.yourdomain.com/.well-known/oauth-authorization-server`
2. Check MCP endpoint: `curl -o /dev/null -w '%{http_code}' https://ctxd.yourdomain.com/readonly/mcp` → should be `401` (not `404`)
3. If `404`: Traefik isn't routing `/readonly/mcp` — update the router rule
4. If `401`: auth is working — check OAuth token scope and expiry
### PostgreSQL connection fails
```bash
# Check PG is running
docker compose ps postgres
# Check connection
docker exec ctxd python3 -c "
import os
import psycopg
conn = psycopg.connect(os.environ['DATABASE_URL'])
print('Connected:', conn.info.server_version)
"
# If password mismatch (PG data volume initialized with different password):
docker exec ctxd-postgres psql -U ctxd -c "ALTER USER ctxd PASSWORD 'new-password'"
# Then update .env with the new password
```
### Container keeps restarting / public site 502
```bash
docker compose ps -a
docker logs ctxd --tail 40
```
| Symptom | Cause | Fix |
|---------|--------|-----|
| `failed to resolve host 'postgres'` | `ctxd-postgres` not running (often after `--no-deps ctxd`) | `cd app && docker compose up -d postgres ctxd` or `./scripts/deploy.sh` |
| `Restarting (1)` on `ctxd` only | Same — app up without DB on compose network | Start postgres first; wait for **(healthy)** |
| Cloudflare **502** on public URL | Traefik/backend has no healthy upstream on `:9091` | Fix local `curl http://127.0.0.1:9091/status` first |
The container **entrypoint waits up to 120s** for PostgreSQL when `DATABASE_URL` is set (`CTXD_PG_WAIT_SECONDS` to override). If postgres never appears, logs print an explicit message instead of an immediate opaque crash.
Common other causes:
- `DATABASE_URL` password doesn't match what PG was initialized with
- `OAUTH_ENABLED=true` but `OAUTH_ISSUER` is empty
- Missing `CTXD_API_KEY` when `CTXD_AUTH_ENABLED=true`
## License
MIT
+180
View File
@@ -0,0 +1,180 @@
---
name: ctxd-client
description: Use when an LLM agent needs to read or update project context via the Context Dossier (CTXD) daemon. Covers OAuth authentication, MCP tool discovery, read-only and write endpoints, and the workflow for keeping project context current.
version: 1.0.0
author: Hermes Agent
license: MIT
platforms:
- linux
- macos
metadata:
hermes:
tags:
- context-management
- ctxd
- mcp-client
- oauth
- project-context
related_skills:
- project-context-management
- native-mcp
---
# CTXD Client — LLM Agent Guide
Use when an LLM agent (Claude Desktop, Codex CLI, Hermes, custom harness) needs to read or update project context stored in Context Dossier (CTXD).
## Overview
CTXD is a single source of truth for multi-harness project context. It exposes:
- **Public MCP** (`/mcp`) — OAuth read + write (scope-gated) on the public host
- **Hermes / automation** — same `http://<lan>:9091/mcp` with `CTXD_API_KEY` (full tool surface)
Public host: `https://ctxd.cubecraftlabs.com` (OAuth + MCP + landing page + dashboard).
## Connection URLs
| Surface | URL | Auth |
|---------|-----|------|
| Public OAuth MCP (read + write) | `https://ctxd.cubecraftlabs.com/mcp` | OAuth `ctxd.read` and/or `ctxd.write` |
| Legacy aliases | `/readonly/mcp`, `/oauth/mcp` | Same behavior as `/mcp` for OAuth |
| OAuth discovery | `https://ctxd.cubecraftlabs.com/.well-known/oauth-authorization-server` | Public |
| DCR registration | `POST https://ctxd.cubecraftlabs.com/oauth/register` | Public |
| Landing page | `https://ctxd.cubecraftlabs.com/` | Public |
## OAuth Flow
1. **Discover** the authorization server metadata at `/.well-known/oauth-authorization-server`
2. **Register** a client via `POST /oauth/register` (DCR) with your redirect URI
3. **Authorize** — open `/oauth/authorize` in a browser; an admin must approve
4. **Exchange** the authorization code for tokens at `POST /oauth/token`
5. **Use** the access token as `Authorization: Bearer <token>` on MCP connections (Streamable HTTP for read; SSE for write — see Connection URLs)
### Scopes
| Scope | Grants |
|-------|--------|
| `ctxd.read` | `list_projects`, `get_project_context`, `search_context`, `get_project_tags`, `list_files`, `get_file`, `get_client_guide` |
| `ctxd.write` | `update_file`, `set_project_tags`, `sync_to_project`, `get_client_guide` |
Request both scopes for full read+write in **one connector**: `scope=ctxd.read ctxd.write`
**Admin:** In the Web UI → Admin → OAuth clients, set **allowed scopes** per client (create form or **scopes** on an existing row). Tokens cannot exceed what the client is allowed, even if the user requests more at authorize time.
### Redirect URIs by Platform
| Platform | Redirect URI |
|----------|-------------|
| Claude Desktop | `https://claude.ai/api/mcp/auth_callback` |
| Claude Code | `http://localhost:5555/oauth/callback` |
| Codex CLI | `http://localhost:7777/oauth/callback` |
| Custom | Your app's documented OAuth callback |
### Token Lifetime
Access tokens expire per server config (default ~1 hour). Refresh tokens are issued alongside access tokens. Use `POST /oauth/token` with `grant_type=refresh_token` to rotate.
## MCP Tools
### First Call — Always
| Tool | Description |
|------|-------------|
| `get_client_guide` | Return the locked LLM-CLIENT.MD guide. **Call this first in every session.** No arguments. Covers OAuth, MCP tools, read/write endpoints, version-checked updates, and error handling. |
The guide lives in the `ctxd-docs` project as `LLM-CLIENT.MD`. It is **locked** — cannot be updated or deleted by any MCP/API client. If you need the guide updated, ask an admin to edit it via the Web UI.
### Read-only tools (require `ctxd.read` on the same connector)
| Tool | Description |
|------|-------------|
| `list_projects` | All projects with version numbers |
| `get_project_context` | Compiled markdown of all context files for a project |
| `search_context` | FTS5 full-text search across all projects |
| `get_project_tags` | Metadata tags for a project |
| `list_files` | All context files in a project (multi-file mode) |
| `get_file` | Single file with metadata header |
### Write tools (require `ctxd.write` on the same connector)
| Tool | Description |
|------|-------------|
| `update_file` | Update a single context file with optimistic version checking |
| `set_project_tags` | Replace all metadata tags for a project |
| `sync_to_project` | Write CONTEXT.MD as AGENTS.md + symlinks to project root |
## Workflow: Reading Context
```
1. get_client_guide() → read the locked client guide first
2. list_projects() → find the project slug
3. get_project_context(project_id) → read compiled context
or list_files() + get_file() → read individual files
```
The compiled view returns all files concatenated with `## FILENAME` headers and a metadata block at the top.
## Workflow: Updating Context
```
1. get_file(project_id, file_path) → get current content + version
2. update_file(project_id, file_path, content, base_version)
→ base_version must match current version (optimistic locking)
→ returns {"ok": true, "version": N+1} or {"ok": false, "error": "version_conflict"}
3. On conflict: re-read, merge, retry
```
### What to Update
- **CONTEXT.MD** — canonical project overview (synced as AGENTS.md to repos)
- **DECISIONS.MD** — architecture decisions, rationale
- **RUNBOOKS.MD** — deploy, troubleshoot, operate procedures
- **PROMPTS.MD** — project-specific prompts for different harnesses
- **GLOSSARY.MD** — project-specific terms, acronyms
### What NOT to Put in Context
- Session progress logs (use session_search or audit trail)
- Temporary TODO state
- Single-session debugging notes
- Secrets, keys, tokens
## Locked Files
| File | Protection |
|------|-----------|
| `CONTEXT.MD` | Cannot delete — canonical synced file |
| `LLM-CLIENT.MD` | Cannot update or delete — locked client guide |
Both return `403 cannot_update_locked` on PUT and `400 cannot_delete_context` on DELETE.
## Discipline
1. **Call `get_client_guide()` first** in every session
2. **Read context at the start of every session** for the project you're working on
3. **Update context immediately** when a durable fact changes — not "next session"
4. **Use `base_version`** to avoid overwriting concurrent edits
5. **Don't put session progress in project context** — that's what session history is for
## Error Handling
| Error | Meaning | Action |
|-------|---------|--------|
| `version_conflict` | Someone else updated the file | Re-read, merge, retry with new `base_version` |
| `not_found` | Project or file doesn't exist | Check slug spelling; create project via Web UI |
| `invalid_grant` | OAuth token expired or wrong scope | Refresh token or re-authorize with correct scope |
| `forbidden` | Wrong scope for the tool | Request `ctxd.write` for write tools; `ctxd.read` for read tools (same MCP URL) |
| `cannot_delete_context` | Tried to delete CONTEXT.MD or LLM-CLIENT.MD | Protected file — update content instead |
| `cannot_update_locked` | Tried to update LLM-CLIENT.MD | Locked guide — ask admin to update via Web UI |
## Verification Checklist
- [ ] OAuth client registered with correct redirect URI
- [ ] Admin approved the authorization (browser session or approval key)
- [ ] Access token obtained with `ctxd.read` (and `ctxd.write` for updates)
- [ ] `get_client_guide` returns the LLM client guide
- [ ] `list_projects` returns expected projects
- [ ] `get_project_context` returns compiled markdown with metadata header
- [ ] `update_file` succeeds with correct `base_version`
- [ ] Audit log shows the write operation
+64
View File
@@ -0,0 +1,64 @@
# ════════════════════════════════════════════════════════════════════
# CTXD — Context Dossier Environment Configuration
# Copy to .env and fill in your values. All variables are optional
# unless running in Docker (DATABASE_URL, CTXD_API_KEY, OAUTH_* required).
#
# Precedence: env var > ctxd.yaml (in /data) > built-in defaults
# ════════════════════════════════════════════════════════════════════
# ── Database ────────────────────────────────────────────────────────
# Full PostgreSQL connection string. If empty, CTXD falls back to SQLite
# at /data/ctxd.db (or CTXD_HOME/ctxd.db).
DATABASE_URL=postgresql://ctxd:ctxd_local_dev@postgres:5432/ctxd
# Postgres container settings (only used when running the bundled PG)
POSTGRES_USER=ctxd
POSTGRES_PASSWORD=ctxd_local_dev
POSTGRES_DB=ctxd
# ── Server ──────────────────────────────────────────────────────────
# Bind address and port inside the container
CTXD_HOST=0.0.0.0
CTXD_PORT=9091
# Data directory (inside container)
CTXD_HOME=/data
# Uvicorn log level: debug, info, warning, error
LOG_LEVEL=info
# ── Auth ────────────────────────────────────────────────────────────
# Enable authentication globally
CTXD_AUTH_ENABLED=true
# Shared API key for Hermes/internal MCP + HTTP auth
CTXD_API_KEY=
# Legacy read-only key for /readonly/sse ?key= migration
CTXD_EXTERNAL_READONLY_KEY=
# ── OAuth ───────────────────────────────────────────────────────────
# Enable the OAuth authorization server
OAUTH_ENABLED=true
# Public URL of the server (used in OAuth discovery metadata)
OAUTH_ISSUER=https://ctxd.example.com
# Fallback approval key for /oauth/authorize
OAUTH_APPROVAL_KEY=
# Which user ID to attribute OAuth approvals to
OAUTH_APPROVAL_USER_ID=admin
# Token lifetimes in seconds
OAUTH_ACCESS_TOKEN_TTL=3600
OAUTH_REFRESH_TOKEN_TTL=2592000
# ── Web Sessions ────────────────────────────────────────────────────
# Session cookie lifetime in seconds (default: 7 days)
WEB_SESSION_TTL=604800
# ── Snapshots ───────────────────────────────────────────────────────
# Min/max snapshots retained per project before rotation
SNAPSHOT_MIN_KEEP=5
SNAPSHOT_MAX_KEEP=25
+32 -3
View File
@@ -1,17 +1,46 @@
name: ctxd name: ctxd
# Docker Compose reads .env automatically for variable substitution.
# See .env.example for all available variables.
#
# IMPORTANT: Production uses PostgreSQL (DATABASE_URL host "postgres").
# Always start BOTH services: docker compose up -d
# Or use: ./scripts/deploy.sh
# Do NOT use docker compose up -d --no-deps ctxd unless postgres is already running.
services: services:
dossier: postgres:
image: postgres:16-alpine
container_name: ctxd-postgres
restart: unless-stopped
environment:
POSTGRES_DB: ${POSTGRES_DB:-ctxd}
POSTGRES_USER: ${POSTGRES_USER:-ctxd}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-ctxd_local_dev}
volumes:
- ${CTXD_PG_DATA:-/mnt/ai-storage/projects/ctxd/data/pg}:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-ctxd}"]
interval: 5s
timeout: 5s
retries: 5
ctxd:
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
container_name: dossier container_name: ctxd
restart: unless-stopped restart: unless-stopped
ports: ports:
- "9091:9091" - "${CTXD_PORT:-9091}:${CTXD_PORT:-9091}"
volumes: volumes:
- /mnt/ai-storage/projects/ctxd/data:/data:rw - /mnt/ai-storage/projects/ctxd/data:/data:rw
- /mnt/ai-storage/projects:/projects:ro - /mnt/ai-storage/projects:/projects:ro
- /home/overseer:/host:ro - /home/overseer:/host:ro
env_file:
- .env
environment: environment:
- CTXD_HOME=/data - CTXD_HOME=/data
depends_on:
postgres:
condition: service_healthy
+71 -5
View File
@@ -1,11 +1,77 @@
#!/bin/bash #!/bin/bash
set -e set -e
# Initialize if database doesn't exist wait_for_postgres() {
if [ ! -f "$CTXD_HOME/ctxd.db" ]; then if [ -z "$DATABASE_URL" ]; then
echo "ctxd: initializing database at $CTXD_HOME" return 0
python3 -m ctxd init --home "$CTXD_HOME" fi
echo "ctxd: waiting for PostgreSQL (DATABASE_URL is set)..."
python3 <<'PY'
import os, sys, time
url = os.environ.get("DATABASE_URL", "")
max_wait = int(os.environ.get("CTXD_PG_WAIT_SECONDS", "120"))
interval = int(os.environ.get("CTXD_PG_WAIT_INTERVAL", "2"))
deadline = time.time() + max_wait
last_err = ""
attempt = 0
while time.time() < deadline:
attempt += 1
try:
import psycopg
conn = psycopg.connect(url, connect_timeout=5)
conn.close()
print("ctxd: PostgreSQL is ready")
sys.exit(0)
except Exception as e:
last_err = str(e)
if attempt == 1 or attempt % 10 == 0:
print(f"ctxd: postgres not ready (attempt {attempt}): {last_err}", file=sys.stderr)
time.sleep(interval)
print("ctxd: FATAL: PostgreSQL not reachable within wait window.", file=sys.stderr)
print(f"ctxd: last error: {last_err}", file=sys.stderr)
print("ctxd:", file=sys.stderr)
print("ctxd: Fix: start the full stack (postgres + ctxd), e.g.", file=sys.stderr)
print("ctxd: cd app && docker compose up -d postgres ctxd", file=sys.stderr)
print("ctxd:", file=sys.stderr)
print("ctxd: Avoid: docker compose up -d --no-deps ctxd (skips postgres)", file=sys.stderr)
print("ctxd: If postgres is on another host, fix DATABASE_URL / network.", file=sys.stderr)
sys.exit(1)
PY
}
wait_for_postgres
# Initialize if needed
if [ -n "$DATABASE_URL" ]; then
# PostgreSQL mode — check if schema exists
NEEDS_INIT=$(python3 -c "
import psycopg, os, sys
try:
conn = psycopg.connect(os.environ['DATABASE_URL'])
cur = conn.execute(\"SELECT EXISTS (SELECT FROM pg_tables WHERE schemaname='public' AND tablename='users')\")
if not cur.fetchone()[0]:
print('yes')
else:
print('no')
except Exception:
print('yes')
" 2>/dev/null || echo "yes")
if [ "$NEEDS_INIT" = "yes" ]; then
echo "ctxd: initializing PostgreSQL database"
python3 -m ctxd init --home "$CTXD_HOME"
else
echo "ctxd: running pending migrations"
python3 -m ctxd.migrate_user_fk_set_null 2>&1 || echo "ctxd: migration warning (non-fatal)"
fi
else
# SQLite mode — check for db file
if [ ! -f "$CTXD_HOME/ctxd.db" ]; then
echo "ctxd: initializing database at $CTXD_HOME"
python3 -m ctxd init --home "$CTXD_HOME"
fi
fi fi
echo "ctxd: starting daemon on 0.0.0.0:9091" echo "ctxd: starting daemon on ${CTXD_HOST:-0.0.0.0}:${CTXD_PORT:-9091}"
exec python3 -m ctxd exec python3 -m ctxd
+6
View File
@@ -3,6 +3,12 @@ name = "context-dossier"
version = "0.2.0" version = "0.2.0"
description = "Context Dossier — single source of truth for multi-harness project context" description = "Context Dossier — single source of truth for multi-harness project context"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [
"psycopg[binary]>=3.1",
"mcp>=1.28",
"uvicorn>=0.30",
"pyyaml>=6.0",
]
[project.scripts] [project.scripts]
dossier = "ctxd:cli_entry" dossier = "ctxd:cli_entry"
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Safe CTXD deploy: always ensures postgres is up before recreating ctxd.
# Usage: ./scripts/deploy.sh [--build-only | --no-build]
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
BUILD=1
if [[ "${1:-}" == "--no-build" ]]; then
BUILD=0
elif [[ "${1:-}" == "--build-only" ]]; then
docker compose build ctxd
echo "Built ctxd image only."
exit 0
fi
if [[ "$BUILD" -eq 1 ]]; then
echo "==> docker compose build ctxd"
docker compose build ctxd
fi
echo "==> docker compose up -d postgres"
docker compose up -d postgres
echo "==> waiting for postgres healthy..."
for i in $(seq 1 60); do
if docker compose ps postgres 2>/dev/null | grep -q '(healthy)'; then
break
fi
sleep 2
done
if ! docker compose ps postgres 2>/dev/null | grep -q '(healthy)'; then
echo "ERROR: ctxd-postgres did not become healthy. Check: docker compose logs postgres" >&2
exit 1
fi
echo "==> docker compose up -d --force-recreate ctxd"
docker compose up -d --force-recreate ctxd
PORT="${CTXD_PORT:-9091}"
echo "==> smoke: http://127.0.0.1:${PORT}/status"
sleep 3
curl -sf "http://127.0.0.1:${PORT}/status" | head -c 200
echo ""
docker compose ps postgres ctxd
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""Smoke test: one OAuth connector lists read+write tools on /mcp."""
import json, urllib.request, urllib.parse, hashlib, base64, secrets, http.client
from pathlib import Path
BASE = "http://127.0.0.1:9091"
env_path = Path("/mnt/ai-storage/projects/ctxd/app/.env")
approval_key = ""
for line in env_path.read_text().splitlines():
if line.startswith("OAUTH_APPROVAL_KEY="):
approval_key = line.split("=", 1)[1]
break
if not approval_key:
raise SystemExit("no OAUTH_APPROVAL_KEY")
class NR(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = urllib.request.build_opener(NR)
r = opener.open(
urllib.request.Request(
f"{BASE}/oauth/register",
data=json.dumps(
{
"redirect_uris": ["http://localhost:9999/cb"],
"client_name": "unified-mcp-test",
"scope": "ctxd.read ctxd.write",
}
).encode(),
headers={"Content-Type": "application/json"},
method="POST",
),
timeout=15,
)
client = json.loads(r.read())
cid, secret = client["client_id"], client["client_secret"]
cv = secrets.token_urlsafe(32)
cc = base64.urlsafe_b64encode(hashlib.sha256(cv.encode()).digest()).rstrip(b"=").decode()
params = {
"response_type": "code",
"client_id": cid,
"redirect_uri": "http://localhost:9999/cb",
"code_challenge": cc,
"code_challenge_method": "S256",
"scope": "ctxd.read ctxd.write",
"state": "t",
}
try:
opener.open(
urllib.request.Request(
f"{BASE}/oauth/authorize?" + urllib.parse.urlencode(params),
data=urllib.parse.urlencode({"approval_key": approval_key}).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"},
method="POST",
),
timeout=15,
)
except urllib.error.HTTPError as e:
loc = e.headers.get("Location", "")
code = dict(urllib.parse.parse_qsl(urllib.parse.urlparse(loc).query)).get("code", "")
r3 = opener.open(
urllib.request.Request(
f"{BASE}/oauth/token",
data=urllib.parse.urlencode(
{
"grant_type": "authorization_code",
"code": code,
"redirect_uri": "http://localhost:9999/cb",
"client_id": cid,
"client_secret": secret,
"code_verifier": cv,
}
).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"},
method="POST",
),
timeout=15,
)
token = json.loads(r3.read())["access_token"]
def mcp_post(path: str, body: dict, session_id: str | None = None) -> tuple[int, str, str | None]:
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
}
if session_id:
headers["mcp-session-id"] = session_id
conn = http.client.HTTPConnection("127.0.0.1", 9091, timeout=15)
conn.request("POST", path, body=json.dumps(body), headers=headers)
resp = conn.getresponse()
sid = resp.getheader("mcp-session-id")
data = resp.read(8000).decode(errors="replace")
conn.close()
return resp.status, data, sid
init_body = {
"jsonrpc": "2.0",
"id": 1,
"method": "initialize",
"params": {
"protocolVersion": "2025-03-26",
"capabilities": {},
"clientInfo": {"name": "unified-test", "version": "1"},
},
}
st, raw, sid = mcp_post("/mcp", init_body)
print("initialize:", st, "session:", sid)
if st != 200:
raise SystemExit(raw[:500])
if sid:
st2, raw2, _ = mcp_post(
"/mcp",
{"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}},
session_id=sid.split(",")[0].strip(),
)
else:
st2, raw2, _ = mcp_post("/mcp", {"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}})
print("tools/list:", st2)
names = []
for line in raw2.splitlines():
if line.startswith("data:"):
try:
payload = json.loads(line[5:].strip())
for t in payload.get("result", {}).get("tools", []):
names.append(t.get("name"))
except json.JSONDecodeError:
pass
if '"tools"' in raw2 and not names:
try:
payload = json.loads(raw2)
names = [t["name"] for t in payload.get("result", {}).get("tools", [])]
except json.JSONDecodeError:
pass
print("tools:", sorted(set(names)))
need = {"list_projects", "update_file", "sync_to_project"}
missing = need - set(names)
if missing:
raise SystemExit(f"missing tools: {missing}")
print("OK unified connector exposes read+write tools")
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""E2E: OAuth write token -> /write/sse and tools/list via SSE."""
import json, urllib.request, urllib.parse, hashlib, base64, secrets, http.client, time
from pathlib import Path
BASE = "http://127.0.0.1:9091"
env_path = Path("/mnt/ai-storage/projects/ctxd/app/.env")
approval_key = ""
for line in env_path.read_text().splitlines():
if line.startswith("OAUTH_APPROVAL_KEY="):
approval_key = line.split("=", 1)[1]
break
if not approval_key:
raise SystemExit("no OAUTH_APPROVAL_KEY")
class NR(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *a): return None
opener = urllib.request.build_opener(NR)
r = opener.open(urllib.request.Request(
f"{BASE}/oauth/register",
data=json.dumps({"redirect_uris": ["http://localhost:9999/cb"], "client_name": "write-test", "scope": "ctxd.read ctxd.write"}).encode(),
headers={"Content-Type": "application/json"}, method="POST"), timeout=15)
client = json.loads(r.read())
cid, secret = client["client_id"], client["client_secret"]
cv = secrets.token_urlsafe(32)
cc = base64.urlsafe_b64encode(hashlib.sha256(cv.encode()).digest()).rstrip(b"=").decode()
params = {"response_type": "code", "client_id": cid, "redirect_uri": "http://localhost:9999/cb",
"code_challenge": cc, "code_challenge_method": "S256", "scope": "ctxd.read ctxd.write", "state": "t"}
try:
opener.open(urllib.request.Request(f"{BASE}/oauth/authorize?" + urllib.parse.urlencode(params),
data=urllib.parse.urlencode({"approval_key": approval_key}).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"}, method="POST"), timeout=15)
except urllib.error.HTTPError as e:
loc = e.headers.get("Location", "")
code = dict(urllib.parse.parse_qsl(urllib.parse.urlparse(loc).query)).get("code", "")
r3 = opener.open(urllib.request.Request(f"{BASE}/oauth/token",
data=urllib.parse.urlencode({"grant_type": "authorization_code", "code": code, "redirect_uri": "http://localhost:9999/cb",
"client_id": cid, "client_secret": secret, "code_verifier": cv}).encode(),
headers={"Content-Type": "application/x-www-form-urlencoded"}, method="POST"), timeout=15)
token = json.loads(r3.read())["access_token"]
conn = http.client.HTTPConnection("127.0.0.1", 9091, timeout=5)
conn.request("GET", "/write/sse", headers={"Authorization": f"Bearer {token}"})
resp = conn.getresponse()
print("write/sse status:", resp.status, "ctype:", resp.getheader("content-type"))
chunk = resp.read(500).decode(errors="replace")
print("body[:200]:", chunk[:200])
conn.close()
# POST initialize to /write/messages (needs session from SSE - simplified check)
conn2 = http.client.HTTPConnection("127.0.0.1", 9091, timeout=10)
body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
conn2.request("POST", "/write/messages", body=body,
headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"})
r2 = conn2.getresponse()
print("write/messages POST:", r2.status, r2.read(300).decode(errors="replace")[:200])
conn2.close()
-5
View File
@@ -1,5 +0,0 @@
Metadata-Version: 2.4
Name: ctxd
Version: 0.1.0
Summary: Context daemon — single source of truth for multi-harness project context
Requires-Python: >=3.11
-13
View File
@@ -1,13 +0,0 @@
pyproject.toml
src/ctxd/__init__.py
src/ctxd/__main__.py
src/ctxd/cli.py
src/ctxd/config.py
src/ctxd/db.py
src/ctxd/mcp_stdio.py
src/ctxd/server.py
src/ctxd.egg-info/PKG-INFO
src/ctxd.egg-info/SOURCES.txt
src/ctxd.egg-info/dependency_links.txt
src/ctxd.egg-info/entry_points.txt
src/ctxd.egg-info/top_level.txt
@@ -1 +0,0 @@
-3
View File
@@ -1,3 +0,0 @@
[console_scripts]
ctx = ctxd:cli_entry
ctxd = ctxd:daemon_entry
-1
View File
@@ -1 +0,0 @@
ctxd
+7 -2
View File
@@ -4,7 +4,10 @@ import sys
if len(sys.argv) > 1 and sys.argv[1] in ('init', 'serve', 'project-list', 'project-create', if len(sys.argv) > 1 and sys.argv[1] in ('init', 'serve', 'project-list', 'project-create',
'read', 'cat', 'edit', 'search', 'sync', 'audit', 'read', 'cat', 'edit', 'search', 'sync', 'audit',
'user-list', 'user-create', 'import-vault'): 'user-list', 'user-create', 'import-vault',
'user-set-password', 'oauth-client-create',
'oauth-client-list', 'oauth-client-revoke',
'file-list', 'file-read'):
from ctxd.cli import cli_entry from ctxd.cli import cli_entry
cli_entry() cli_entry()
else: else:
@@ -12,7 +15,9 @@ else:
from ctxd.server import serve_sync from ctxd.server import serve_sync
cfg = CtxConfig.from_home() cfg = CtxConfig.from_home()
if not cfg.db_path.exists(): # When using PostgreSQL, the DB is external — skip the file check.
# When using SQLite, verify the db file exists.
if not cfg.use_postgres and not cfg.db_path.exists():
print("Not initialized. Run 'ctx init' first.", file=sys.stderr) print("Not initialized. Run 'ctx init' first.", file=sys.stderr)
sys.exit(1) sys.exit(1)
serve_sync(cfg) serve_sync(cfg)
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+40
View File
@@ -0,0 +1,40 @@
"""Password hashing for CTXD user accounts (stdlib only)."""
from __future__ import annotations
import hashlib
import secrets
PBKDF2_ITERATIONS = 600_000
SCHEME = "pbkdf2_sha256"
def hash_password(password: str) -> str:
if not password:
raise ValueError("password required")
salt = secrets.token_hex(16)
digest = hashlib.pbkdf2_hmac(
"sha256",
password.encode("utf-8"),
salt.encode("utf-8"),
PBKDF2_ITERATIONS,
)
return f"{SCHEME}${PBKDF2_ITERATIONS}${salt}${digest.hex()}"
def verify_password(password: str, token_hash: str | None) -> bool:
if not password or not token_hash:
return False
try:
scheme, iters_s, salt, expected_hex = token_hash.split("$", 3)
if scheme != SCHEME:
return False
digest = hashlib.pbkdf2_hmac(
"sha256",
password.encode("utf-8"),
salt.encode("utf-8"),
int(iters_s),
)
return secrets.compare_digest(digest.hex(), expected_hex)
except (ValueError, TypeError):
return False
+109 -10
View File
@@ -64,14 +64,15 @@ Multi-camera remote monitoring system.
def _seed_context(conn): def _seed_context(conn):
"""Insert seed project context with real newlines.""" """Insert seed project context with real newlines."""
conn.execute( from ctxd.db import _is_pg, _ph
"INSERT OR IGNORE INTO project_context (project_id, content, version, updated_by) VALUES (?, ?, 0, 'system')", ph = _ph(conn, 2)
('welcome', _WELCOME_CONTENT), placeholders = ph.split(", ")
) if _is_pg(conn):
conn.execute( sql = f"INSERT INTO project_context (project_id, content, version, updated_by) VALUES ({placeholders[0]}, {placeholders[1]}, 0, 'admin') ON CONFLICT DO NOTHING"
"INSERT OR IGNORE INTO project_context (project_id, content, version, updated_by) VALUES (?, ?, 0, 'system')", else:
('remote-rig', _REMOTE_RIG_CONTENT), sql = f"INSERT OR IGNORE INTO project_context (project_id, content, version, updated_by) VALUES ({placeholders[0]}, {placeholders[1]}, 0, 'admin')"
) conn.execute(sql, ('welcome', _WELCOME_CONTENT))
conn.execute(sql, ('remote-rig', _REMOTE_RIG_CONTENT))
def cmd_init(args): def cmd_init(args):
@@ -309,16 +310,87 @@ def cmd_user_create(args):
"""Create a new user.""" """Create a new user."""
conn = _db.init_db(CtxConfig.from_home(args.home)) conn = _db.init_db(CtxConfig.from_home(args.home))
try: try:
_db.user_create(conn, args.user_id, args.display_name, args.role) _db.user_create(conn, args.user_id, args.display_name, args.role, password=getattr(args, "password", None))
conn.commit() conn.commit()
print(f"✓ User '{args.user_id}' created.") print(f"✓ User '{args.user_id}' created.")
except Exception as e: except Exception as e:
conn.rollback() conn.rollback()
print(f"Error: {e}") print(f" {e}")
sys.exit(1)
finally: finally:
conn.close() conn.close()
def cmd_user_set_password(args):
"""Set or reset a user's Web UI password."""
conn = _db.init_db(CtxConfig.from_home(args.home))
try:
if _db.user_get(conn, args.user_id) is None:
print(f"✗ User '{args.user_id}' not found.")
sys.exit(1)
_db.user_set_password(conn, args.user_id, args.password)
conn.commit()
print(f"✓ Password set for '{args.user_id}'.")
except Exception as e:
conn.rollback()
print(f"{e}")
sys.exit(1)
finally:
conn.close()
def cmd_oauth_client_create(args):
"""Register an OAuth client for Claude (or other MCP connectors)."""
from .server import OAuthStore, CLAUDE_MCP_REDIRECT_URI
cfg = CtxConfig.from_home(args.home)
store = OAuthStore(cfg)
redirects = args.redirect_uri or [CLAUDE_MCP_REDIRECT_URI]
if isinstance(redirects, str):
redirects = [redirects]
client = store.register_client({
"client_name": args.name or "Claude MCP Client",
"redirect_uris": redirects,
"scopes": getattr(args, "scopes", None) or (
[s for s in (args.scope or "").split() if s]
or ["ctxd.read", "ctxd.write"]
),
})
issuer = (cfg.oauth_issuer or "").rstrip("/") or "https://ctxd.cubecraftcreations.com"
print(json.dumps({
"client_id": client["client_id"],
"client_secret": client["client_secret"],
"client_name": client.get("client_name"),
"redirect_uris": client.get("redirect_uris"),
"scope": client.get("scope"),
"connector_url": f"{issuer}/mcp",
"authorization_server": issuer,
"note": "Claude usually registers via POST /oauth/register automatically; save client_secret now — it is not shown again.",
}, indent=2))
def cmd_oauth_client_list(args):
"""List OAuth clients (no secrets)."""
from .server import OAuthStore
cfg = CtxConfig.from_home(args.home)
store = OAuthStore(cfg)
for c in store.list_clients_public():
print(f"{c.get('client_id')} {c.get('client_name', '')} scope={c.get('scope', '')} redirects={c.get('redirect_uris')}")
def cmd_oauth_client_revoke(args):
"""Revoke an OAuth client and invalidate its tokens."""
from .server import OAuthStore
cfg = CtxConfig.from_home(args.home)
store = OAuthStore(cfg)
if not store.revoke_client(args.client_id):
print(f"✗ client not found: {args.client_id}")
sys.exit(1)
print(f"✓ Revoked OAuth client {args.client_id}")
def cmd_import_vault(args): def cmd_import_vault(args):
"""Import context from an existing vault (e.g., OpenClawVault).""" """Import context from an existing vault (e.g., OpenClawVault)."""
cfg = CtxConfig.from_home(args.home) cfg = CtxConfig.from_home(args.home)
@@ -477,6 +549,33 @@ def build_parser() -> argparse.ArgumentParser:
sp.add_argument("user_id") sp.add_argument("user_id")
sp.add_argument("--display-name", "-n", required=True) sp.add_argument("--display-name", "-n", required=True)
sp.add_argument("--role", "-r", default="contributor", choices=["admin", "contributor", "service"]) sp.add_argument("--role", "-r", default="contributor", choices=["admin", "contributor", "service"])
sp.add_argument("--password", help="Optional Web UI login password")
sp.add_argument("--home")
# user set-password
sp = sub.add_parser("user-set-password", help="Set a user's Web UI password")
sp.set_defaults(func=cmd_user_set_password)
sp.add_argument("user_id")
sp.add_argument("--password", "-p", required=True)
sp.add_argument("--home")
# oauth-client-create
sp = sub.add_parser("oauth-client-create", help="Register OAuth client (Claude MCP); prints client_id and client_secret")
sp.set_defaults(func=cmd_oauth_client_create)
sp.add_argument("--name", "-n", default="Claude MCP Client", help="Client display name")
sp.add_argument("--redirect-uri", action="append", dest="redirect_uri", help="Redirect URI (default: Claude MCP callback; repeat for multiple)")
sp.add_argument("--scope", default="ctxd.read ctxd.write", help="Allowed scopes (space-separated: ctxd.read ctxd.write)")
sp.add_argument("--home")
# oauth-client-list
sp = sub.add_parser("oauth-client-list", help="List OAuth clients (no secrets)")
sp.set_defaults(func=cmd_oauth_client_list)
sp.add_argument("--home")
# oauth-client-revoke
sp = sub.add_parser("oauth-client-revoke", help="Revoke OAuth client and invalidate its tokens")
sp.set_defaults(func=cmd_oauth_client_revoke)
sp.add_argument("client_id", help="client_id to revoke (ctxd_…)")
sp.add_argument("--home") sp.add_argument("--home")
# import-vault # import-vault
+114 -11
View File
@@ -1,13 +1,39 @@
""" """
Configuration for ctxd — context daemon. Configuration for ctxd — context daemon.
Environment variables take precedence over ctxd.yaml, which takes precedence
over built-in defaults. This allows fully env-driven deployments (Docker,
Compose, Kubernetes) while preserving file-based config for local dev.
""" """
import os import os
from pathlib import Path from pathlib import Path
def _env_bool(key: str, fallback: bool = False) -> bool:
val = os.environ.get(key, "")
if not val:
return fallback
return val.lower() in ("1", "true", "yes", "on")
def _env_int(key: str, fallback: int) -> int:
val = os.environ.get(key, "")
if not val:
return fallback
try:
return int(val)
except ValueError:
return fallback
def _env_str(key: str, fallback: str = "") -> str:
return os.environ.get(key, fallback)
# Default home directory (~/.ctx) — overridable via CTXD_HOME env var # Default home directory (~/.ctx) — overridable via CTXD_HOME env var
DEFAULT_HOME = Path(os.environ.get("CTXD_HOME", Path.home() / ".ctx")) DEFAULT_HOME = Path(os.environ.get("CTXD_HOME", Path.home() / ".ctx"))
# Defaults for ctxd.yaml # Built-in defaults (lowest precedence)
DEFAULT_CONFIG = { DEFAULT_CONFIG = {
"server": { "server": {
"host": "0.0.0.0", "host": "0.0.0.0",
@@ -20,6 +46,18 @@ DEFAULT_CONFIG = {
"auth": { "auth": {
"enabled": False, "enabled": False,
"api_key": "", "api_key": "",
"external_readonly_key": "",
},
"oauth": {
"enabled": False,
"issuer": "",
"approval_key": "",
"approval_user_id": "admin",
"access_token_ttl_seconds": 3600,
"refresh_token_ttl_seconds": 2592000,
},
"web_sessions": {
"ttl_seconds": 604800,
}, },
"seed": { "seed": {
"admin_user": "admin", "admin_user": "admin",
@@ -31,7 +69,10 @@ DEFAULT_CONFIG = {
class CtxConfig: class CtxConfig:
"""Holds resolved paths and config for a ctxd runtime.""" """Holds resolved paths and config for a ctxd runtime.
Precedence: env var > ctxd.yaml value > built-in default.
"""
def __init__(self, home: Path | str | None = None, config: dict | None = None): def __init__(self, home: Path | str | None = None, config: dict | None = None):
resolved = Path(home) if home else DEFAULT_HOME resolved = Path(home) if home else DEFAULT_HOME
@@ -59,31 +100,91 @@ class CtxConfig:
def config_path(self) -> Path: def config_path(self) -> Path:
return self.home / "ctxd.yaml" return self.home / "ctxd.yaml"
# ── Config accessors ────────────────────────────────────────── @property
def oauth_state_path(self) -> Path:
return self.home / "oauth_state.json"
@property
def web_sessions_path(self) -> Path:
return self.home / "web_sessions.json"
# ── Database ──────────────────────────────────────────────────
@property
def database_url(self) -> str:
return _env_str("DATABASE_URL")
@property
def use_postgres(self) -> bool:
return bool(self.database_url)
# ── Server ────────────────────────────────────────────────────
@property @property
def host(self) -> str: def host(self) -> str:
return self._cfg.get("server", {}).get("host", "127.0.0.1") return _env_str("CTXD_HOST", self._cfg.get("server", {}).get("host", "0.0.0.0"))
@property @property
def port(self) -> int: def port(self) -> int:
return self._cfg.get("server", {}).get("port", 9091) return _env_int("CTXD_PORT", self._cfg.get("server", {}).get("port", 9091))
@property
def log_level(self) -> str:
return _env_str("LOG_LEVEL", "info")
@property
def demo_mode(self) -> bool:
return _env_bool("CTXD_DEMO_MODE", False)
# ── Snapshots ─────────────────────────────────────────────────
@property @property
def min_snapshots(self) -> int: def min_snapshots(self) -> int:
return self._cfg.get("snapshots", {}).get("min_keep", 5) return _env_int("SNAPSHOT_MIN_KEEP", self._cfg.get("snapshots", {}).get("min_keep", 5))
@property @property
def max_snapshots(self) -> int: def max_snapshots(self) -> int:
return self._cfg.get("snapshots", {}).get("max_keep", 25) return _env_int("SNAPSHOT_MAX_KEEP", self._cfg.get("snapshots", {}).get("max_keep", 25))
# ── Auth ────────────────────────────────────────────────────── # ── Auth ──────────────────────────────────────────────────────
@property @property
def auth_enabled(self) -> bool: def auth_enabled(self) -> bool:
return self._cfg.get("auth", {}).get("enabled", False) return _env_bool("CTXD_AUTH_ENABLED", self._cfg.get("auth", {}).get("enabled", False))
@property @property
def api_key(self) -> str: def api_key(self) -> str:
return self._cfg.get("auth", {}).get("api_key", "") return _env_str("CTXD_API_KEY", self._cfg.get("auth", {}).get("api_key", ""))
@property
def external_readonly_key(self) -> str:
return _env_str("CTXD_EXTERNAL_READONLY_KEY", self._cfg.get("auth", {}).get("external_readonly_key", ""))
# ── OAuth ─────────────────────────────────────────────────────
@property
def oauth_enabled(self) -> bool:
return _env_bool("OAUTH_ENABLED", self._cfg.get("oauth", {}).get("enabled", False))
@property
def oauth_issuer(self) -> str:
return _env_str("OAUTH_ISSUER", self._cfg.get("oauth", {}).get("issuer", ""))
@property
def oauth_approval_key(self) -> str:
return _env_str("OAUTH_APPROVAL_KEY", self._cfg.get("oauth", {}).get("approval_key", ""))
@property
def oauth_approval_user_id(self) -> str:
return _env_str("OAUTH_APPROVAL_USER_ID", self._cfg.get("oauth", {}).get("approval_user_id", "admin"))
@property
def oauth_access_token_ttl_seconds(self) -> int:
return _env_int("OAUTH_ACCESS_TOKEN_TTL", self._cfg.get("oauth", {}).get("access_token_ttl_seconds", 3600))
@property
def oauth_refresh_token_ttl_seconds(self) -> int:
return _env_int("OAUTH_REFRESH_TOKEN_TTL", self._cfg.get("oauth", {}).get("refresh_token_ttl_seconds", 2592000))
# ── Web Sessions ───────────────────────────────────────────────
@property
def web_session_ttl_seconds(self) -> int:
return _env_int("WEB_SESSION_TTL", self._cfg.get("web_sessions", {}).get("ttl_seconds", 604800))
# ── Bootstrap ───────────────────────────────────────────────── # ── Bootstrap ─────────────────────────────────────────────────
def ensure_dirs(self): def ensure_dirs(self):
@@ -93,7 +194,9 @@ class CtxConfig:
@classmethod @classmethod
def from_home(cls, home: Path | str | None = None) -> "CtxConfig": def from_home(cls, home: Path | str | None = None) -> "CtxConfig":
"""Load from ctxd.yaml if it exists, otherwise use defaults.""" """Load from ctxd.yaml if it exists, otherwise use defaults.
Env vars always take precedence over yaml values at read time.
"""
home = Path(home).resolve() if home else DEFAULT_HOME home = Path(home).resolve() if home else DEFAULT_HOME
cfg_path = home / "ctxd.yaml" cfg_path = home / "ctxd.yaml"
if cfg_path.exists(): if cfg_path.exists():
@@ -104,7 +207,7 @@ class CtxConfig:
return cls(home=str(home)) return cls(home=str(home))
def save(self): def save(self):
"""Write config to ctxd.yaml.""" """Write config to ctxd.yaml. Rarely needed in env-driven deployments."""
import yaml import yaml
self.ensure_dirs() self.ensure_dirs()
with open(self.config_path, "w") as f: with open(self.config_path, "w") as f:
+432 -110
View File
@@ -1,7 +1,10 @@
""" """
Database layer for ctxd — schema init, CRUD, workspace fork/merge, FTS, audit. Database layer for ctxd — schema init, CRUD, workspace fork/merge, FTS, audit.
All public methods take a sqlite3.Connection as the first argument so callers All public methods take a connection as the first argument so callers
control transactions. This module is stateless — all state is in SQLite. control transactions. This module is stateless — all state is in the DB.
Supports both PostgreSQL (via DATABASE_URL) and SQLite (fallback for local dev).
The public API is identical regardless of backend.
""" """
import json import json
import sqlite3 import sqlite3
@@ -11,15 +14,58 @@ from pathlib import Path
from typing import Any from typing import Any
from .config import CtxConfig from .config import CtxConfig
from .auth_password import hash_password, verify_password
# ── Schema ──────────────────────────────────────────────────────────────────── # ── Schema paths ─────────────────────────────────────────────────────────────
SCHEMA_PATH = Path(__file__).parent / "schema.sql" SCHEMA_PG_PATH = Path(__file__).parent / "schema.sql"
SCHEMA_SQLITE_PATH = Path(__file__).parent / "schema_sqlite.sql"
def init_db(cfg: CtxConfig) -> sqlite3.Connection: def init_db(cfg: CtxConfig):
"""Create ~/.ctx/ dirs + initialize the database from schema.sql.""" """Create ~/.ctx/ dirs + initialize the database.
If DATABASE_URL is set in the environment, connects to PostgreSQL.
Otherwise falls back to SQLite for local development.
Returns a connection object (psycopg.Connection or sqlite3.Connection).
"""
cfg.ensure_dirs() cfg.ensure_dirs()
if cfg.use_postgres:
return _init_pg(cfg)
else:
return _init_sqlite(cfg)
def _init_pg(cfg: CtxConfig):
"""Initialize PostgreSQL database from schema.sql."""
import psycopg
from psycopg.rows import dict_row
conn = psycopg.connect(cfg.database_url, row_factory=dict_row)
# Keep autocommit=False (default) — we use explicit commit() calls
# Check if schema is already initialized by looking for the users table
cur = conn.execute(
"SELECT EXISTS (SELECT FROM pg_tables WHERE schemaname = 'public' AND tablename = 'users')"
)
schema_exists = cur.fetchone()["exists"]
if not schema_exists:
with open(SCHEMA_PG_PATH) as f:
schema_sql = f.read()
# PostgreSQL can execute multiple statements in a single execute() call
conn.execute(schema_sql)
conn.commit()
else:
# Run migrations if needed
_migrate_pg(conn)
return conn
def _init_sqlite(cfg: CtxConfig) -> sqlite3.Connection:
"""Initialize SQLite database (fallback for local dev)."""
fresh = not cfg.db_path.exists() fresh = not cfg.db_path.exists()
conn = sqlite3.connect(str(cfg.db_path)) conn = sqlite3.connect(str(cfg.db_path))
conn.row_factory = sqlite3.Row conn.row_factory = sqlite3.Row
@@ -27,70 +73,248 @@ def init_db(cfg: CtxConfig) -> sqlite3.Connection:
conn.execute("PRAGMA foreign_keys = ON") conn.execute("PRAGMA foreign_keys = ON")
if fresh: if fresh:
with open(SCHEMA_PATH) as f: with open(SCHEMA_SQLITE_PATH) as f:
conn.executescript(f.read()) conn.executescript(f.read())
else: else:
# NOTE: SQLite cannot ALTER a FK constraint in place. Existing dev
# databases will NOT pick up FK changes in schema_sqlite.sql (e.g.
# ON DELETE SET NULL on user_id columns) — delete $CTXD_HOME/ctxd.db
# to recreate from the current schema. Only additive ADD COLUMN
# migrations are applied here.
# Migration: add metadata_tags column if it doesn't exist # Migration: add metadata_tags column if it doesn't exist
try: try:
conn.execute("ALTER TABLE projects ADD COLUMN metadata_tags TEXT DEFAULT '[]'") conn.execute("ALTER TABLE projects ADD COLUMN metadata_tags TEXT DEFAULT '[]'")
conn.commit() conn.commit()
except sqlite3.OperationalError: except sqlite3.OperationalError:
pass # column already exists pass
try:
conn.execute("ALTER TABLE users ADD COLUMN active INTEGER NOT NULL DEFAULT 1")
conn.commit()
except sqlite3.OperationalError:
pass
return conn return conn
def _migrate_pg(conn):
"""Run PostgreSQL migrations for existing databases."""
try:
conn.execute("ALTER TABLE projects ADD COLUMN IF NOT EXISTS metadata_tags TEXT DEFAULT '[]'")
conn.commit()
except Exception:
pass
try:
conn.execute("ALTER TABLE users ADD COLUMN IF NOT EXISTS active BOOLEAN NOT NULL DEFAULT TRUE")
conn.commit()
except Exception:
pass
def now() -> str: def now() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _is_pg(conn) -> bool:
"""Check if connection is a PostgreSQL connection."""
return not isinstance(conn, sqlite3.Connection)
def is_integrity_error(exc: BaseException) -> bool:
"""True if exc is a DB constraint violation (unique/primary-key, FK, check)
on either backend. Used to map duplicate-key errors to HTTP 409.
SQLite raises sqlite3.IntegrityError; psycopg raises subclasses of
psycopg.errors.IntegrityError (UniqueViolation, ForeignKeyViolation, ...).
"""
if isinstance(exc, sqlite3.IntegrityError):
return True
try:
import psycopg
except ImportError:
return False
return isinstance(exc, psycopg.errors.IntegrityError)
# ── Helpers ─────────────────────────────────────────────────────────────────── # ── Helpers ───────────────────────────────────────────────────────────────────
def _row_to_dict(row: sqlite3.Row | None) -> dict | None: def _row_to_dict(row) -> dict | None:
if row is None: if row is None:
return None return None
if isinstance(row, dict):
return dict(row)
return dict(row) return dict(row)
def _ph(conn, n: int = 1) -> str:
"""Return placeholder string for the current backend.
PostgreSQL uses %s, SQLite uses ?.
"""
if _is_pg(conn):
return ", ".join(["%s"] * n)
else:
return ", ".join(["?"] * n)
# ── Users ───────────────────────────────────────────────────────────────────── # ── Users ─────────────────────────────────────────────────────────────────────
def user_create(conn, user_id: str, display_name: str, role: str = "contributor"):
def user_create(conn, user_id: str, display_name: str, role: str = "contributor", password: str | None = None, active: bool = True):
token_hash = hash_password(password) if password else None
ph = _ph(conn, 5)
if _is_pg(conn):
active_val = active
else:
active_val = 1 if active else 0
conn.execute( conn.execute(
"INSERT INTO users (user_id, display_name, role) VALUES (?, ?, ?)", f"INSERT INTO users (user_id, display_name, role, token_hash, active) VALUES ({ph})",
(user_id, display_name, role), (user_id, display_name, role, token_hash, active_val),
)
def user_update(conn, user_id: str, *, display_name: str | None = None, role: str | None = None, active: bool | None = None) -> bool:
user = user_get(conn, user_id)
if not user:
return False
fields = []
values: list[Any] = []
ph = _ph(conn, 1)
if display_name is not None:
fields.append(f"display_name = {ph}")
values.append(display_name)
if role is not None:
fields.append(f"role = {ph}")
values.append(role)
if active is not None:
if _is_pg(conn):
fields.append(f"active = {ph}")
values.append(active)
else:
fields.append(f"active = {ph}")
values.append(1 if active else 0)
if not fields:
return True
if _is_pg(conn):
fields.append("updated_at = to_char(now() at time zone 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')")
else:
fields.append("updated_at = strftime('%Y-%m-%dT%H:%M:%SZ', 'now')")
values.append(user_id)
conn.execute(f"UPDATE users SET {', '.join(fields)} WHERE user_id = {ph}", values)
return True
def user_delete(conn, user_id: str) -> dict:
if user_get(conn, user_id) is None:
return {"ok": False, "error": "not_found"}
ph = _ph(conn, 1)
try:
conn.execute(f"DELETE FROM users WHERE user_id = {ph}", (user_id,))
return {"ok": True}
except Exception as e:
if _is_pg(conn):
import psycopg
# Roll back unconditionally so a failed DELETE never leaves the
# shared connection in an aborted-transaction state (see the 500
# cascade fix); only an FK violation maps to the soft response.
conn.rollback()
if isinstance(e, psycopg.errors.ForeignKeyViolation):
return {"ok": False, "error": "user_has_references", "hint": "Inactivate the user instead of deleting."}
raise
# SQLite: only an FK violation is the expected "still referenced" case.
# Anything else (I/O error, programming bug, corruption) must propagate
# to the global handler as a 500 rather than be masked as a soft 409.
if isinstance(e, sqlite3.IntegrityError):
return {"ok": False, "error": "user_has_references", "hint": "Inactivate the user instead of deleting."}
raise
def user_set_password(conn, user_id: str, password: str):
ph = _ph(conn, 2)
if _is_pg(conn):
ts_expr = "to_char(now() at time zone 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
else:
ts_expr = "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
conn.execute(
f"UPDATE users SET token_hash = {ph}, updated_at = {ts_expr} WHERE user_id = {ph}",
(hash_password(password), user_id),
) )
def user_get(conn, user_id: str) -> dict | None: def user_get(conn, user_id: str) -> dict | None:
ph = _ph(conn, 1)
return _row_to_dict(conn.execute( return _row_to_dict(conn.execute(
"SELECT * FROM users WHERE user_id = ?", (user_id,) f"SELECT * FROM users WHERE user_id = {ph}", (user_id,)
).fetchone()) ).fetchone())
def user_get_ci(conn, user_id: str) -> dict | None:
"""Case-insensitive user lookup (matches idx_users_lower)."""
ph = _ph(conn, 1)
return _row_to_dict(conn.execute(
f"SELECT * FROM users WHERE LOWER(user_id) = LOWER({ph})", (user_id.strip(),)
).fetchone())
def user_authenticate(conn, user_id: str, password: str) -> dict | None:
user = user_get_ci(conn, user_id)
if not user:
return None
# Handle both PostgreSQL BOOLEAN and SQLite INTEGER for active
active = user.get("active")
if active is None:
active = True # Default to active
if isinstance(active, int) and active == 0:
return None
if isinstance(active, bool) and not active:
return None
if not user.get("token_hash"):
return None
if not verify_password(password, user.get("token_hash")):
return None
return user
def user_list(conn) -> list[dict]: def user_list(conn) -> list[dict]:
return [dict(r) for r in conn.execute("SELECT * FROM users ORDER BY user_id").fetchall()] if _is_pg(conn):
return [dict(r) for r in conn.execute("SELECT * FROM users ORDER BY user_id").fetchall()]
else:
return [dict(r) for r in conn.execute("SELECT * FROM users ORDER BY user_id").fetchall()]
# ── Projects ────────────────────────────────────────────────────────────────── # ── Projects ──────────────────────────────────────────────────────────────────
def project_create(conn, project_id: str, display_name: str, description: str = ""): def project_create(conn, project_id: str, display_name: str, description: str = ""):
ph3 = _ph(conn, 3)
ph1 = _ph(conn, 1)
conn.execute( conn.execute(
"INSERT INTO projects (project_id, display_name, description) VALUES (?, ?, ?)", f"INSERT INTO projects (project_id, display_name, description) VALUES ({ph3})",
(project_id, display_name, description), (project_id, display_name, description),
) )
# Also create empty shared context # Also create empty shared context
conn.execute( conn.execute(
"INSERT INTO project_context (project_id, content, version) VALUES (?, '', 0)", f"INSERT INTO project_context (project_id, content, version) VALUES ({ph1}, '', 0)",
(project_id,), (project_id,),
) )
def project_get(conn, project_id: str) -> dict | None: def project_get(conn, project_id: str) -> dict | None:
ph = _ph(conn, 1)
return _row_to_dict(conn.execute( return _row_to_dict(conn.execute(
"SELECT * FROM projects WHERE project_id = ?", (project_id,) f"SELECT * FROM projects WHERE project_id = {ph}", (project_id,)
).fetchone()) ).fetchone())
def project_delete(conn, project_id: str) -> dict:
"""Delete a project and all its dependent rows (cascades).
audit_log.project_id is SET NULL via FK ON DELETE SET NULL.
Returns {'ok': True} or {'ok': False, 'error': ...}."""
if project_get(conn, project_id) is None:
return {"ok": False, "error": "not_found"}
ph = _ph(conn, 1)
conn.execute(f"DELETE FROM projects WHERE project_id = {ph}", (project_id,))
return {"ok": True}
def project_list(conn) -> list[dict]: def project_list(conn) -> list[dict]:
return [dict(r) for r in conn.execute( return [dict(r) for r in conn.execute(
"SELECT project_id, display_name, description, shared_version FROM projects ORDER BY project_id" "SELECT project_id, display_name, description, shared_version FROM projects ORDER BY project_id"
@@ -98,23 +322,30 @@ def project_list(conn) -> list[dict]:
def project_set_sync_path(conn, project_id: str, sync_path: str | None): def project_set_sync_path(conn, project_id: str, sync_path: str | None):
conn.execute( ph = _ph(conn, 2)
"UPDATE projects SET sync_path = ?, auto_sync = 1 WHERE project_id = ?", if _is_pg(conn):
(sync_path, project_id), conn.execute(
) f"UPDATE projects SET sync_path = {ph.split(', ')[0]}, auto_sync = TRUE WHERE project_id = {ph.split(', ')[1]}",
(sync_path, project_id),
)
else:
conn.execute(
f"UPDATE projects SET sync_path = ?, auto_sync = 1 WHERE project_id = ?",
(sync_path, project_id),
)
def project_get_tags(conn, project_id: str) -> list[str]: def project_get_tags(conn, project_id: str) -> list[str]:
"""Get project metadata tags as a list of strings.""" """Get project metadata tags as a list of strings."""
ph = _ph(conn, 1)
row = conn.execute( row = conn.execute(
"SELECT metadata_tags FROM projects WHERE project_id = ?", (project_id,) f"SELECT metadata_tags FROM projects WHERE project_id = {ph}", (project_id,)
).fetchone() ).fetchone()
if row is None: if row is None:
return [] return []
tags = row["metadata_tags"] tags = row["metadata_tags"]
if not tags: if not tags:
return [] return []
import json
try: try:
return json.loads(tags) return json.loads(tags)
except (json.JSONDecodeError, TypeError): except (json.JSONDecodeError, TypeError):
@@ -122,10 +353,9 @@ def project_get_tags(conn, project_id: str) -> list[str]:
def project_set_tags(conn, project_id: str, tags: list[str]): def project_set_tags(conn, project_id: str, tags: list[str]):
"""Set project metadata tags from a list of strings.""" ph = _ph(conn, 2)
import json
conn.execute( conn.execute(
"UPDATE projects SET metadata_tags = ? WHERE project_id = ?", f"UPDATE projects SET metadata_tags = {ph.split(', ')[0]} WHERE project_id = {ph.split(', ')[1]}",
(json.dumps(tags), project_id), (json.dumps(tags), project_id),
) )
@@ -139,7 +369,6 @@ def build_metadata_header(project_id: str, display_name: str | None = None,
TYPE: PROJECT CONTEXT, PROJECT, STATUS: ACTIVE, LAST-UPDATED, TAGS. TYPE: PROJECT CONTEXT, PROJECT, STATUS: ACTIVE, LAST-UPDATED, TAGS.
LAST-UPDATED uses the actual updated_at timestamp, falling back to today. LAST-UPDATED uses the actual updated_at timestamp, falling back to today.
TAGS uses the project's metadata_tags if provided, falling back to project name + CONTEXT.""" TAGS uses the project's metadata_tags if provided, falling back to project name + CONTEXT."""
from datetime import datetime, timezone
project_upper = (display_name or project_id).upper() project_upper = (display_name or project_id).upper()
last_updated = (updated_at or datetime.now(timezone.utc).strftime("%Y-%m-%d")) last_updated = (updated_at or datetime.now(timezone.utc).strftime("%Y-%m-%d"))
if "T" in last_updated: if "T" in last_updated:
@@ -186,10 +415,11 @@ def context_read(conn, project_id: str) -> dict | None:
Returns with metadata header prepended dynamically. Returns with metadata header prepended dynamically.
If content already has a header (including YAML frontmatter from vault imports), If content already has a header (including YAML frontmatter from vault imports),
it is replaced with the current dynamic header.""" it is replaced with the current dynamic header."""
ph = _ph(conn, 1)
row = conn.execute( row = conn.execute(
"SELECT pc.*, p.shared_version, p.display_name FROM project_context pc " f"SELECT pc.*, p.shared_version, p.display_name FROM project_context pc "
"JOIN projects p ON p.project_id = pc.project_id " f"JOIN projects p ON p.project_id = pc.project_id "
"WHERE pc.project_id = ?", (project_id,) f"WHERE pc.project_id = {ph}", (project_id,)
).fetchone() ).fetchone()
if row is None: if row is None:
return None return None
@@ -238,8 +468,9 @@ def context_update(conn, project_id: str, new_content: str, updated_by: str,
Returns {'ok': True, 'new_version': N} or {'ok': False, 'error': 'conflict', Returns {'ok': True, 'new_version': N} or {'ok': False, 'error': 'conflict',
'current_version': N}. 'current_version': N}.
""" """
ph = _ph(conn, 1)
cur = conn.execute( cur = conn.execute(
"SELECT shared_version FROM projects WHERE project_id = ?", f"SELECT shared_version FROM projects WHERE project_id = {ph}",
(project_id,) (project_id,)
) )
row = cur.fetchone() row = cur.fetchone()
@@ -262,15 +493,21 @@ def context_update(conn, project_id: str, new_content: str, updated_by: str,
_snapshot_take(conn, project_id, version_from=current_version, version_to=new_version) _snapshot_take(conn, project_id, version_from=current_version, version_to=new_version)
# Update project_context # Update project_context
ph5 = _ph(conn, 5)
if _is_pg(conn):
ts_expr = "to_char(now() at time zone 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
else:
ts_expr = "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
conn.execute( conn.execute(
"UPDATE project_context SET content = ?, version = ?, updated_by = ?, updated_at = ? " f"UPDATE project_context SET content = {_ph(conn,1)}, version = {_ph(conn,1)}, "
"WHERE project_id = ?", f"updated_by = {_ph(conn,1)}, updated_at = {ts_expr} "
(clean_content, new_version, updated_by, now(), project_id), f"WHERE project_id = {_ph(conn,1)}",
(clean_content, new_version, updated_by, project_id)
) )
# Bump shared version # Bump shared version
conn.execute( conn.execute(
"UPDATE projects SET shared_version = ? WHERE project_id = ?", f"UPDATE projects SET shared_version = {_ph(conn,1)} WHERE project_id = {_ph(conn,1)}",
(new_version, project_id), (new_version, project_id)
) )
return {"ok": True, "new_version": new_version, "content": clean_content} return {"ok": True, "new_version": new_version, "content": clean_content}
@@ -279,29 +516,38 @@ def context_update(conn, project_id: str, new_content: str, updated_by: str,
# ── User Profile ────────────────────────────────────────────────────────────── # ── User Profile ──────────────────────────────────────────────────────────────
def profile_read(conn, user_id: str) -> dict | None: def profile_read(conn, user_id: str) -> dict | None:
ph = _ph(conn, 1)
return _row_to_dict(conn.execute( return _row_to_dict(conn.execute(
"SELECT * FROM user_profiles WHERE user_id = ?", (user_id,) f"SELECT * FROM user_profiles WHERE user_id = {ph}", (user_id,)
).fetchone()) ).fetchone())
def profile_update(conn, user_id: str, content: str, base_version: int) -> dict: def profile_update(conn, user_id: str, content: str, base_version: int) -> dict:
ph = _ph(conn, 1)
cur = conn.execute( cur = conn.execute(
"SELECT version FROM user_profiles WHERE user_id = ?", (user_id,) f"SELECT version FROM user_profiles WHERE user_id = {ph}", (user_id,)
) )
row = cur.fetchone() row = cur.fetchone()
if row is None: if row is None:
# Create # Create
ph2 = _ph(conn, 2)
conn.execute( conn.execute(
"INSERT INTO user_profiles (user_id, content, version) VALUES (?, ?, 1)", f"INSERT INTO user_profiles (user_id, content, version) VALUES ({ph2}, 1)",
(user_id, content), (user_id, content)
) )
return {"ok": True, "new_version": 1} return {"ok": True, "new_version": 1}
current_version = row["version"] current_version = row["version"]
if base_version != current_version: if base_version != current_version:
return {"ok": False, "error": "conflict", "current_version": current_version} return {"ok": False, "error": "conflict", "current_version": current_version}
if _is_pg(conn):
ts_expr = "to_char(now() at time zone 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
else:
ts_expr = "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
ph4 = _ph(conn, 4)
conn.execute( conn.execute(
"UPDATE user_profiles SET content = ?, version = ?, updated_at = ? WHERE user_id = ?", f"UPDATE user_profiles SET content = {_ph(conn,1)}, version = {_ph(conn,1)}, "
(content, current_version + 1, now(), user_id), f"updated_at = {ts_expr} WHERE user_id = {_ph(conn,1)}",
(content, current_version + 1, user_id)
) )
return {"ok": True, "new_version": current_version + 1} return {"ok": True, "new_version": current_version + 1}
@@ -325,42 +571,51 @@ def workspace_fork(conn, user_id: str, project_id: str) -> dict:
shared_content = ctx["content"] if ctx else "" shared_content = ctx["content"] if ctx else ""
ws_id = str(uuid.uuid4()) ws_id = str(uuid.uuid4())
ph4 = _ph(conn, 4)
conn.execute( conn.execute(
"INSERT INTO user_workspaces (workspace_id, user_id, project_id, base_version) " f"INSERT INTO user_workspaces (workspace_id, user_id, project_id, base_version) "
"VALUES (?, ?, ?, ?)", f"VALUES ({ph4})",
(ws_id, user_id, project_id, base_version), (ws_id, user_id, project_id, base_version),
) )
# Seed workspace with current shared content # Seed workspace with current shared content
ph2 = _ph(conn, 2)
conn.execute( conn.execute(
"INSERT INTO workspace_files (workspace_id, file_path, content) VALUES (?, 'context.md', ?)", f"INSERT INTO workspace_files (workspace_id, file_path, content) VALUES ({ph2.split(', ')[0]}, 'context.md', {ph2.split(', ')[1]})",
(ws_id, shared_content), (ws_id, shared_content),
) )
return {"ok": True, "workspace_id": ws_id, "base_version": base_version} return {"ok": True, "workspace_id": ws_id, "base_version": base_version}
def workspace_get(conn, workspace_id: str) -> dict | None: def workspace_get(conn, workspace_id: str) -> dict | None:
ph = _ph(conn, 1)
return _row_to_dict(conn.execute( return _row_to_dict(conn.execute(
"SELECT * FROM user_workspaces WHERE workspace_id = ?", (workspace_id,) f"SELECT * FROM user_workspaces WHERE workspace_id = {ph}", (workspace_id,)
).fetchone()) ).fetchone())
def workspace_list_for_user(conn, user_id: str, project_id: str | None = None) -> list[dict]: def workspace_list_for_user(conn, user_id: str, project_id: str | None = None) -> list[dict]:
if project_id: if project_id:
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
rows = conn.execute( rows = conn.execute(
"SELECT * FROM user_workspaces WHERE user_id = ? AND project_id = ? ORDER BY created_at DESC", f"SELECT * FROM user_workspaces WHERE user_id = {placeholders[0]} AND project_id = {placeholders[1]} "
f"ORDER BY created_at DESC",
(user_id, project_id), (user_id, project_id),
).fetchall() ).fetchall()
else: else:
ph = _ph(conn, 1)
rows = conn.execute( rows = conn.execute(
"SELECT * FROM user_workspaces WHERE user_id = ? ORDER BY created_at DESC", f"SELECT * FROM user_workspaces WHERE user_id = {ph} ORDER BY created_at DESC",
(user_id,), (user_id,),
).fetchall() ).fetchall()
return [dict(r) for r in rows] return [dict(r) for r in rows]
def workspace_read_file(conn, workspace_id: str, file_path: str = "context.md") -> str | None: def workspace_read_file(conn, workspace_id: str, file_path: str = "context.md") -> str | None:
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
row = conn.execute( row = conn.execute(
"SELECT content FROM workspace_files WHERE workspace_id = ? AND file_path = ?", f"SELECT content FROM workspace_files WHERE workspace_id = {placeholders[0]} AND file_path = {placeholders[1]}",
(workspace_id, file_path), (workspace_id, file_path),
).fetchone() ).fetchone()
return row["content"] if row else None return row["content"] if row else None
@@ -368,19 +623,27 @@ def workspace_read_file(conn, workspace_id: str, file_path: str = "context.md")
def workspace_write_file(conn, workspace_id: str, content: str, def workspace_write_file(conn, workspace_id: str, content: str,
file_path: str = "context.md"): file_path: str = "context.md"):
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
existing = conn.execute( existing = conn.execute(
"SELECT 1 FROM workspace_files WHERE workspace_id = ? AND file_path = ?", f"SELECT 1 FROM workspace_files WHERE workspace_id = {placeholders[0]} AND file_path = {placeholders[1]}",
(workspace_id, file_path), (workspace_id, file_path),
).fetchone() ).fetchone()
if existing: if existing:
if _is_pg(conn):
ts_expr = "to_char(now() at time zone 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
else:
ts_expr = "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
conn.execute( conn.execute(
"UPDATE workspace_files SET content = ?, version = version + 1, updated_at = ? " f"UPDATE workspace_files SET content = {_ph(conn,1)}, version = version + 1, "
"WHERE workspace_id = ? AND file_path = ?", f"updated_at = {ts_expr} "
(content, now(), workspace_id, file_path), f"WHERE workspace_id = {_ph(conn,1)} AND file_path = {_ph(conn,1)}",
(content, workspace_id, file_path),
) )
else: else:
ph3 = _ph(conn, 3)
conn.execute( conn.execute(
"INSERT INTO workspace_files (workspace_id, file_path, content) VALUES (?, ?, ?)", f"INSERT INTO workspace_files (workspace_id, file_path, content) VALUES ({ph3})",
(workspace_id, file_path, content), (workspace_id, file_path, content),
) )
@@ -424,31 +687,35 @@ def workspace_submit(conn, workspace_id: str, submitted_by: str,
) )
if not result["ok"]: if not result["ok"]:
return result return result
ph = _ph(conn, 1)
conn.execute( conn.execute(
"UPDATE user_workspaces SET status = 'merged' WHERE workspace_id = ?", f"UPDATE user_workspaces SET status = 'merged' WHERE workspace_id = {ph}",
(workspace_id,), (workspace_id,),
) )
return {"ok": True, "action": "merged", **result} return {"ok": True, "action": "merged", **result}
else: else:
# Create pending change request # Create pending change request
req_id = str(uuid.uuid4()) req_id = str(uuid.uuid4())
ph7 = _ph(conn, 7)
conn.execute( conn.execute(
"INSERT INTO change_requests (request_id, workspace_id, project_id, " f"INSERT INTO change_requests (request_id, workspace_id, project_id, "
"submitted_by, target_version, base_version, diff_summary) " f"submitted_by, target_version, base_version, diff_summary) "
"VALUES (?, ?, ?, ?, ?, ?, ?)", f"VALUES ({ph7})",
(req_id, workspace_id, ws["project_id"], submitted_by, (req_id, workspace_id, ws["project_id"], submitted_by,
target_version, base_version, diff_summary), target_version, base_version, diff_summary),
) )
ph = _ph(conn, 1)
conn.execute( conn.execute(
"UPDATE user_workspaces SET status = 'submitted' WHERE workspace_id = ?", f"UPDATE user_workspaces SET status = 'submitted' WHERE workspace_id = {ph}",
(workspace_id,), (workspace_id,),
) )
return {"ok": True, "action": "submitted", "request_id": req_id} return {"ok": True, "action": "submitted", "request_id": req_id}
def workspace_abandon(conn, workspace_id: str): def workspace_abandon(conn, workspace_id: str):
ph = _ph(conn, 1)
conn.execute( conn.execute(
"UPDATE user_workspaces SET status = 'abandoned' WHERE workspace_id = ?", f"UPDATE user_workspaces SET status = 'abandoned' WHERE workspace_id = {ph}",
(workspace_id,), (workspace_id,),
) )
@@ -456,10 +723,11 @@ def workspace_abandon(conn, workspace_id: str):
def change_request_approve(conn, request_id: str, reviewer_id: str, def change_request_approve(conn, request_id: str, reviewer_id: str,
comments: str = "") -> dict: comments: str = "") -> dict:
"""Approve a change request and merge it into shared context.""" """Approve a change request and merge it into shared context."""
ph = _ph(conn, 1)
row = conn.execute( row = conn.execute(
"SELECT cr.*, ws.project_id FROM change_requests cr " f"SELECT cr.*, ws.project_id FROM change_requests cr "
"JOIN user_workspaces ws ON ws.workspace_id = cr.workspace_id " f"JOIN user_workspaces ws ON ws.workspace_id = cr.workspace_id "
"WHERE cr.request_id = ?", (request_id,) f"WHERE cr.request_id = {ph}", (request_id,)
).fetchone() ).fetchone()
if row is None: if row is None:
return {"ok": False, "error": "not_found"} return {"ok": False, "error": "not_found"}
@@ -467,8 +735,10 @@ def change_request_approve(conn, request_id: str, reviewer_id: str,
return {"ok": False, "error": f"status is {row['status']}"} return {"ok": False, "error": f"status is {row['status']}"}
# Record review # Record review
ph3 = _ph(conn, 3)
conn.execute( conn.execute(
"INSERT INTO reviews (request_id, reviewer_id, decision, comments) VALUES (?, ?, 'approved', ?)", f"INSERT INTO reviews (request_id, reviewer_id, decision, comments) "
f"VALUES ({ph3.split(', ')[0]}, {ph3.split(', ')[1]}, 'approved', {ph3.split(', ')[2]})",
(request_id, reviewer_id, comments), (request_id, reviewer_id, comments),
) )
@@ -483,12 +753,13 @@ def change_request_approve(conn, request_id: str, reviewer_id: str,
if not result["ok"]: if not result["ok"]:
return result return result
ph = _ph(conn, 1)
conn.execute( conn.execute(
"UPDATE change_requests SET status = 'merged' WHERE request_id = ?", f"UPDATE change_requests SET status = 'merged' WHERE request_id = {ph}",
(request_id,), (request_id,),
) )
conn.execute( conn.execute(
"UPDATE user_workspaces SET status = 'merged' WHERE workspace_id = ?", f"UPDATE user_workspaces SET status = 'merged' WHERE workspace_id = {ph}",
(row["workspace_id"],), (row["workspace_id"],),
) )
return {"ok": True, "action": "merged", **result} return {"ok": True, "action": "merged", **result}
@@ -510,18 +781,20 @@ def _snapshot_take(conn, project_id: str, version_from: int, version_to: int,
ts = now().replace(":", "-") ts = now().replace(":", "-")
storage_rel = f"{project_id}/{ts}__v{version_from}-{version_to}" storage_rel = f"{project_id}/{ts}__v{version_from}-{version_to}"
ph9 = _ph(conn, 9)
conn.execute( conn.execute(
"INSERT INTO snapshots (snapshot_id, project_id, user_id, workspace_id, " f"INSERT INTO snapshots (snapshot_id, project_id, user_id, workspace_id, "
"version_from, version_to, storage_path, content_hash, size_bytes) " f"version_from, version_to, storage_path, content_hash, size_bytes) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", f"VALUES ({ph9})",
(snap_id, project_id, user_id, workspace_id, (snap_id, project_id, user_id, workspace_id,
version_from, version_to, storage_rel, content_hash, len(content)), version_from, version_to, storage_rel, content_hash, len(content)),
) )
def snapshot_list(conn, project_id: str) -> list[dict]: def snapshot_list(conn, project_id: str) -> list[dict]:
ph = _ph(conn, 1)
rows = conn.execute( rows = conn.execute(
"SELECT * FROM snapshots WHERE project_id = ? ORDER BY created_at DESC", f"SELECT * FROM snapshots WHERE project_id = {ph} ORDER BY created_at DESC",
(project_id,), (project_id,),
).fetchall() ).fetchall()
return [dict(r) for r in rows] return [dict(r) for r in rows]
@@ -532,16 +805,19 @@ def snapshot_rotate(conn, project_id: str, max_keep: int = 25, min_keep: int = 5
Prune excess snapshots for a project, keeping at least min_keep. Prune excess snapshots for a project, keeping at least min_keep.
Returns count of pruned snapshots. Returns count of pruned snapshots.
""" """
ph = _ph(conn, 1)
rows = conn.execute( rows = conn.execute(
"SELECT snapshot_id FROM snapshots WHERE project_id = ? " f"SELECT snapshot_id FROM snapshots WHERE project_id = {ph} "
"ORDER BY created_at DESC", (project_id,) f"ORDER BY created_at DESC", (project_id,)
).fetchall() ).fetchall()
if len(rows) <= max_keep: if len(rows) <= max_keep:
return 0 return 0
keep = max(min_keep, max_keep) keep = max(min_keep, max_keep)
to_delete = [r["snapshot_id"] for r in rows[keep:]] to_delete = [r["snapshot_id"] for r in rows[keep:]]
for sid in to_delete: for sid in to_delete:
conn.execute("DELETE FROM snapshots WHERE snapshot_id = ?", (sid,)) conn.execute(
f"DELETE FROM snapshots WHERE snapshot_id = {_ph(conn,1)}", (sid,)
)
return len(to_delete) return len(to_delete)
@@ -551,10 +827,11 @@ def audit_log(conn, user_id: str, operation: str, summary: str,
agent_id: str = "ctx", project_id: str | None = None, agent_id: str = "ctx", project_id: str | None = None,
entity_type: str | None = None, entity_id: str | None = None, entity_type: str | None = None, entity_id: str | None = None,
details: dict | None = None): details: dict | None = None):
ph8 = _ph(conn, 8)
conn.execute( conn.execute(
"INSERT INTO audit_log (user_id, agent_id, project_id, operation, " f"INSERT INTO audit_log (user_id, agent_id, project_id, operation, "
"entity_type, entity_id, summary, details_json) " f"entity_type, entity_id, summary, details_json) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)", f"VALUES ({ph8})",
(user_id, agent_id, project_id, operation, (user_id, agent_id, project_id, operation,
entity_type, entity_id, summary, entity_type, entity_id, summary,
json.dumps(details) if details else None), json.dumps(details) if details else None),
@@ -569,7 +846,8 @@ def audit_query(conn, **filters) -> list[dict]:
for col in ("user_id", "project_id", "operation", "agent_id"): for col in ("user_id", "project_id", "operation", "agent_id"):
val = filters.get(col) val = filters.get(col)
if val: if val:
wheres.append(f"{col} = ?") ph = _ph(conn, 1)
wheres.append(f"{col} = {ph}")
params.append(val) params.append(val)
if wheres: if wheres:
parts.append("WHERE " + " AND ".join(wheres)) parts.append("WHERE " + " AND ".join(wheres))
@@ -584,13 +862,27 @@ def audit_query(conn, **filters) -> list[dict]:
def search(conn, query: str, limit: int = 10) -> list[dict]: def search(conn, query: str, limit: int = 10) -> list[dict]:
"""Full-text search across all indexed context content.""" """Full-text search across all indexed context content."""
rows = conn.execute( if _is_pg(conn):
"SELECT rowid, content, project_id, file_path, source_type, " ph = _ph(conn, 3)
"rank FROM fts_context WHERE fts_context MATCH ? " placeholders = ph.split(", ")
"ORDER BY rank LIMIT ?", rows = conn.execute(
(query, limit), f"SELECT content, project_id, file_path, source_type, "
).fetchall() f"ts_rank(tsv, plainto_tsquery('english', {placeholders[0]})) as rank "
return [dict(r) for r in rows] f"FROM fts_context WHERE tsv @@ plainto_tsquery('english', {placeholders[1]}) "
f"ORDER BY rank DESC LIMIT {placeholders[2]}",
(query, query, limit),
).fetchall()
return [dict(r) for r in rows]
else:
ph = _ph(conn, 2)
placeholders = ph.split(", ")
rows = conn.execute(
f"SELECT rowid, content, project_id, file_path, source_type, "
f"rank FROM fts_context WHERE fts_context MATCH {placeholders[0]} "
f"ORDER BY rank LIMIT {placeholders[1]}",
(query, limit),
).fetchall()
return [dict(r) for r in rows]
# ── Sync to project root ────────────────────────────────────────────────────── # ── Sync to project root ──────────────────────────────────────────────────────
@@ -677,9 +969,10 @@ def normalize_file_path(file_path: str) -> str:
def file_list(conn, project_id: str) -> list[dict]: def file_list(conn, project_id: str) -> list[dict]:
"""List all files for a project. Returns list of {file_id, file_path, version, updated_at, updated_by}.""" """List all files for a project. Returns list of {file_id, file_path, version, updated_at, updated_by}."""
ph = _ph(conn, 1)
rows = conn.execute( rows = conn.execute(
"SELECT file_id, file_path, version, updated_by, updated_at " f"SELECT file_id, file_path, version, updated_by, updated_at "
"FROM context_files WHERE project_id = ? ORDER BY file_path", f"FROM context_files WHERE project_id = {ph} ORDER BY file_path",
(project_id,) (project_id,)
).fetchall() ).fetchall()
return [_row_to_dict(r) for r in rows] return [_row_to_dict(r) for r in rows]
@@ -688,10 +981,12 @@ def file_list(conn, project_id: str) -> list[dict]:
def file_read(conn, project_id: str, file_path: str) -> dict | None: def file_read(conn, project_id: str, file_path: str) -> dict | None:
"""Read a single context file. Returns with dynamic metadata header prepended.""" """Read a single context file. Returns with dynamic metadata header prepended."""
file_path = normalize_file_path(file_path) file_path = normalize_file_path(file_path)
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
row = conn.execute( row = conn.execute(
"SELECT cf.*, p.display_name FROM context_files cf " f"SELECT cf.*, p.display_name FROM context_files cf "
"JOIN projects p ON p.project_id = cf.project_id " f"JOIN projects p ON p.project_id = cf.project_id "
"WHERE cf.project_id = ? AND cf.file_path = ?", f"WHERE cf.project_id = {placeholders[0]} AND cf.file_path = {placeholders[1]}",
(project_id, file_path) (project_id, file_path)
).fetchone() ).fetchone()
if row is None: if row is None:
@@ -731,8 +1026,10 @@ def file_create(conn, project_id: str, file_path: str, content: str = "",
file_path = normalize_file_path(file_path) file_path = normalize_file_path(file_path)
# Check if file already exists # Check if file already exists
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
existing = conn.execute( existing = conn.execute(
"SELECT file_id FROM context_files WHERE project_id = ? AND file_path = ?", f"SELECT file_id FROM context_files WHERE project_id = {placeholders[0]} AND file_path = {placeholders[1]}",
(project_id, file_path) (project_id, file_path)
).fetchone() ).fetchone()
if existing: if existing:
@@ -742,9 +1039,10 @@ def file_create(conn, project_id: str, file_path: str, content: str = "",
clean = strip_metadata_header(content) clean = strip_metadata_header(content)
clean = clean.lstrip("\n\r ").strip() clean = clean.lstrip("\n\r ").strip()
ph5 = _ph(conn, 5)
conn.execute( conn.execute(
"INSERT INTO context_files (project_id, file_path, content, version, updated_by) " f"INSERT INTO context_files (project_id, file_path, content, version, updated_by) "
"VALUES (?, ?, ?, 1, ?)", f"VALUES ({ph5.split(', ')[0]}, {ph5.split(', ')[1]}, {ph5.split(', ')[2]}, 1, {ph5.split(', ')[3]})",
(project_id, file_path, clean, updated_by) (project_id, file_path, clean, updated_by)
) )
audit_log(conn, updated_by, "create", f"Created file {file_path} in {project_id}", audit_log(conn, updated_by, "create", f"Created file {file_path} in {project_id}",
@@ -759,8 +1057,14 @@ def file_update(conn, project_id: str, file_path: str, new_content: str,
# Normalize # Normalize
file_path = normalize_file_path(file_path) file_path = normalize_file_path(file_path)
# Lock only files in the ctxd-docs project (documentation/guide)
if project_id == "ctxd-docs" and file_path in ("CONTEXT.MD", "LLM-CLIENT.MD"):
return {"ok": False, "error": "cannot_update_locked", "hint": f"{file_path} is locked in ctxd-docs — create a new file instead"}
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
row = conn.execute( row = conn.execute(
"SELECT version FROM context_files WHERE project_id = ? AND file_path = ?", f"SELECT version FROM context_files WHERE project_id = {placeholders[0]} AND file_path = {placeholders[1]}",
(project_id, file_path) (project_id, file_path)
).fetchone() ).fetchone()
if row is None: if row is None:
@@ -778,10 +1082,14 @@ def file_update(conn, project_id: str, file_path: str, new_content: str,
clean = clean.lstrip().strip() clean = clean.lstrip().strip()
new_version = current_version + 1 new_version = current_version + 1
if _is_pg(conn):
ts_expr = "to_char(now() at time zone 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
else:
ts_expr = "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
conn.execute( conn.execute(
"UPDATE context_files SET content = ?, version = ?, updated_by = ?, " f"UPDATE context_files SET content = {_ph(conn,1)}, version = {_ph(conn,1)}, "
"updated_at = strftime('%Y-%m-%dT%H:%M:%SZ', 'now') " f"updated_by = {_ph(conn,1)}, updated_at = {ts_expr} "
"WHERE project_id = ? AND file_path = ?", f"WHERE project_id = {_ph(conn,1)} AND file_path = {_ph(conn,1)}",
(clean, new_version, updated_by, project_id, file_path) (clean, new_version, updated_by, project_id, file_path)
) )
audit_log(conn, updated_by, "update", f"Updated {file_path} in {project_id} to v{new_version}", audit_log(conn, updated_by, "update", f"Updated {file_path} in {project_id} to v{new_version}",
@@ -789,26 +1097,32 @@ def file_update(conn, project_id: str, file_path: str, new_content: str,
return {"ok": True, "new_version": new_version} return {"ok": True, "new_version": new_version}
def file_delete(conn, project_id: str, file_path: str) -> dict: def file_delete(conn, project_id: str, file_path: str, deleted_by: str = "admin") -> dict:
"""Delete a context file. Returns {'ok': True} or {'ok': False, 'error': ...}.""" """Delete a context file. Returns {'ok': True} or {'ok': False, 'error': ...}."""
file_path = normalize_file_path(file_path) file_path = normalize_file_path(file_path)
# Don't allow deleting CONTEXT.md (it's the canonical synced file) # CONTEXT.MD cannot be deleted from any project (it's the minimum required file)
if file_path == "CONTEXT.MD": if file_path == "CONTEXT.MD":
return {"ok": False, "error": "cannot_delete_context"} return {"ok": False, "error": "cannot_delete_context"}
# LLM-CLIENT.MD locked in ctxd-docs only
if project_id == "ctxd-docs" and file_path == "LLM-CLIENT.MD":
return {"ok": False, "error": "cannot_delete_context"}
ph2 = _ph(conn, 2)
placeholders = ph2.split(", ")
row = conn.execute( row = conn.execute(
"SELECT file_id FROM context_files WHERE project_id = ? AND file_path = ?", f"SELECT file_id FROM context_files WHERE project_id = {placeholders[0]} AND file_path = {placeholders[1]}",
(project_id, file_path) (project_id, file_path)
).fetchone() ).fetchone()
if row is None: if row is None:
return {"ok": False, "error": "not_found"} return {"ok": False, "error": "not_found"}
conn.execute( conn.execute(
"DELETE FROM context_files WHERE project_id = ? AND file_path = ?", f"DELETE FROM context_files WHERE project_id = {_ph(conn,1)} AND file_path = {_ph(conn,1)}",
(project_id, file_path) (project_id, file_path)
) )
audit_log(conn, "admin", "delete", f"Deleted {file_path} from {project_id}", audit_log(conn, deleted_by, "delete", f"Deleted {file_path} from {project_id}",
project_id=project_id, entity_type="file", entity_id=file_path) project_id=project_id, entity_type="file", entity_id=file_path)
return {"ok": True} return {"ok": True}
@@ -822,9 +1136,10 @@ def compiled_read(conn, project_id: str) -> dict | None:
return None return None
# Get all files # Get all files
ph = _ph(conn, 1)
files = conn.execute( files = conn.execute(
"SELECT file_path, content, version, updated_at, updated_by " f"SELECT file_path, content, version, updated_at, updated_by "
"FROM context_files WHERE project_id = ? ORDER BY file_path", f"FROM context_files WHERE project_id = {ph} ORDER BY file_path",
(project_id,) (project_id,)
).fetchall() ).fetchall()
@@ -855,8 +1170,9 @@ def compiled_read(conn, project_id: str) -> dict | None:
) )
# Get the latest version from project_context (for version checking) # Get the latest version from project_context (for version checking)
ph = _ph(conn, 1)
ctx_row = conn.execute( ctx_row = conn.execute(
"SELECT version FROM project_context WHERE project_id = ?", f"SELECT version FROM project_context WHERE project_id = {ph}",
(project_id,) (project_id,)
).fetchone() ).fetchone()
version = ctx_row["version"] if ctx_row else 0 version = ctx_row["version"] if ctx_row else 0
@@ -873,25 +1189,29 @@ def ensure_default_files(conn, project_id: str):
"""Create default context files for a project if they don't exist. """Create default context files for a project if they don't exist.
Migrates existing single-context content into CONTEXT.md.""" Migrates existing single-context content into CONTEXT.md."""
# Check if any files already exist # Check if any files already exist
ph = _ph(conn, 1)
existing = conn.execute( existing = conn.execute(
"SELECT COUNT(*) as cnt FROM context_files WHERE project_id = ?", f"SELECT COUNT(*) as cnt FROM context_files WHERE project_id = {ph}",
(project_id,) (project_id,)
).fetchone() ).fetchone()
if existing and existing["cnt"] > 0: if existing and existing["cnt"] > 0:
return # Already has files return # Already has files
# Get existing single-context content to migrate into CONTEXT.md # Get existing single-context content to migrate into CONTEXT.md
ph = _ph(conn, 1)
ctx_row = conn.execute( ctx_row = conn.execute(
"SELECT content FROM project_context WHERE project_id = ?", f"SELECT content FROM project_context WHERE project_id = {ph}",
(project_id,) (project_id,)
).fetchone() ).fetchone()
existing_content = ctx_row["content"] if ctx_row else "" existing_content = ctx_row["content"] if ctx_row else ""
existing_content = strip_metadata_header(existing_content).strip() existing_content = strip_metadata_header(existing_content).strip()
# Create CONTEXT.md with existing content # Create CONTEXT.md with existing content
ph2 = _ph(conn, 2)
p = ph2.split(", ")
conn.execute( conn.execute(
"INSERT INTO context_files (project_id, file_path, content, version, updated_by) " f"INSERT INTO context_files (project_id, file_path, content, version, updated_by) "
"VALUES (?, 'CONTEXT.MD', ?, 1, 'admin')", f"VALUES ({p[0]}, 'CONTEXT.MD', {p[1]}, 1, 'admin')",
(project_id, existing_content) (project_id, existing_content)
) )
@@ -899,8 +1219,10 @@ def ensure_default_files(conn, project_id: str):
for fname in DEFAULT_FILES: for fname in DEFAULT_FILES:
if fname == "CONTEXT.md": if fname == "CONTEXT.md":
continue # Already created above continue # Already created above
ph2 = _ph(conn, 2)
p = ph2.split(", ")
conn.execute( conn.execute(
"INSERT INTO context_files (project_id, file_path, content, version, updated_by) " f"INSERT INTO context_files (project_id, file_path, content, version, updated_by) "
"VALUES (?, ?, '', 1, 'admin')", f"VALUES ({p[0]}, {p[1]}, '', 1, 'admin')",
(project_id, fname.upper()) (project_id, fname.upper())
) )
+309
View File
@@ -0,0 +1,309 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>CTXD — Context Dossier</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:ital,wght@0,400;0,600;0,700;1,400&display=swap" rel="stylesheet">
<style>
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
:root {
--bg: #1a1a1a;
--paper: #1e1e1e;
--ink: #d4d4d4;
--ink-dim: #888;
--accent: #e5c07b;
--accent3: #98c379;
--border: #2a2a2a;
--border-light: #333;
--input-bg: #222;
--hover: #252525;
--danger: #e06c75;
--font: 'JetBrains Mono', 'Fira Code', 'Cascadia Code', 'IBM Plex Mono', monospace;
}
html { height: 100%; font-size: 15px; }
body {
font-family: var(--font);
background: var(--bg);
color: var(--ink);
height: 100vh;
height: 100dvh;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 1.5rem;
}
.landing {
max-width: 420px;
width: 100%;
text-align: center;
}
.landing h1 {
font-size: 1.4rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.15em;
color: var(--accent);
margin-bottom: 0.35rem;
}
.landing .subtitle {
font-size: 0.7rem;
text-transform: uppercase;
letter-spacing: 0.1em;
color: var(--ink-dim);
margin-bottom: 2rem;
}
.landing .description {
font-size: 0.75rem;
color: var(--ink-dim);
line-height: 1.6;
margin-bottom: 2rem;
text-align: left;
}
.landing .description code {
color: var(--accent);
background: var(--input-bg);
padding: 0.1rem 0.3rem;
font-size: 0.7rem;
}
.login-card {
background: var(--paper);
border: 1px solid var(--border-light);
padding: 1.5rem;
text-align: left;
}
.login-card h2 {
font-size: 0.85rem;
text-transform: uppercase;
letter-spacing: 0.1em;
color: var(--accent);
margin-bottom: 1rem;
}
.login-card label {
display: block;
font-size: 0.65rem;
text-transform: uppercase;
letter-spacing: 0.08em;
color: var(--ink-dim);
margin-bottom: 0.3rem;
margin-top: 0.75rem;
}
.login-card input {
width: 100%;
font-family: var(--font);
font-size: 0.8rem;
padding: 0.55rem 0.65rem;
background: var(--input-bg);
border: 1px solid var(--border);
color: var(--ink);
outline: none;
}
.login-card input:focus { border-color: var(--accent); }
.login-card .actions {
display: flex;
gap: 0.5rem;
margin-top: 1rem;
}
.login-card button {
font-family: var(--font);
font-size: 0.7rem;
text-transform: uppercase;
letter-spacing: 0.08em;
padding: 0.5rem 0.85rem;
cursor: pointer;
border: 1px solid var(--border-light);
background: none;
color: var(--ink);
}
.login-card button:hover { background: var(--hover); }
.login-card button.primary {
background: var(--accent);
color: var(--bg);
border-color: var(--accent);
flex: 1;
}
.login-card button.primary:hover { background: #d4ae5c; }
.toast {
position: fixed;
bottom: 1.5rem;
left: 50%;
transform: translateX(-50%);
background: var(--paper);
border: 1px solid var(--border-light);
color: var(--ink);
font-family: var(--font);
font-size: 0.7rem;
padding: 0.6rem 1rem;
opacity: 0;
transition: opacity 0.2s;
pointer-events: none;
z-index: 200;
}
.toast.show { opacity: 1; }
.toast.error { border-color: var(--danger); color: var(--danger); }
.toast.success { border-color: var(--accent3); color: var(--accent3); }
.links {
margin-top: 1.5rem;
font-size: 0.65rem;
color: var(--ink-dim);
line-height: 1.8;
}
.links a {
color: var(--accent);
text-decoration: none;
}
.links a:hover { text-decoration: underline; }
.status {
margin-top: 1rem;
font-size: 0.6rem;
color: var(--ink-dim);
text-transform: uppercase;
letter-spacing: 0.06em;
}
.status .dot {
display: inline-block;
width: 5px; height: 5px;
border-radius: 50%;
background: var(--accent3);
margin-right: 0.3rem;
vertical-align: middle;
}
</style>
</head>
<body>
<div class="landing">
<h1>CTXD</h1>
<div class="subtitle">Context Dossier</div>
<div class="description">
Single source of truth for multi-harness project context.
One canonical <code>AGENTS.md</code> per project, served to
Claude, Hermes, Codex, Cursor, and any OAuth-capable MCP client.
</div>
<div class="login-card" id="login-card">
<h2>sign in</h2>
<label>user id</label>
<input type="text" id="user-id" placeholder="e.g. admin" autocomplete="username" autocorrect="off" autocapitalize="off" spellcheck="false" onkeydown="if(event.key==='Enter')submitLogin()">
<label>password</label>
<input type="password" id="password" placeholder="password" autocomplete="current-password" onkeydown="if(event.key==='Enter')submitLogin()">
<div class="actions">
<button class="primary" onclick="submitLogin()">sign in</button>
</div>
</div>
<div class="links">
<a href="/.well-known/oauth-authorization-server">OAuth discovery</a> &middot;
<a href="/readonly/sse">read-only MCP</a> &middot;
<a href="/write/sse">write MCP</a>
</div>
<div class="status" id="status">
<span class="dot"></span> connected
</div>
</div>
<div class="toast" id="toast"></div>
<script>
const API = '';
function showToast(msg, type) {
const el = document.getElementById('toast');
el.textContent = msg;
el.className = 'toast show' + (type ? ' ' + type : '');
clearTimeout(window._tt);
window._tt = setTimeout(() => el.classList.remove('show'), 3500);
}
async function submitLogin() {
const uid = document.getElementById('user-id').value.trim();
const pw = document.getElementById('password').value;
if (!uid || !pw) {
showToast('user id and password required', 'error');
return;
}
try {
const res = await fetch('/auth/login', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ user_id: uid, password: pw }),
});
if (res.ok) {
const data = await res.json();
localStorage.setItem('ctxd_session_token', data.token);
showToast('signed in — loading dashboard', 'success');
setTimeout(() => { window.location.href = '/'; }, 500);
} else {
let detail = '';
try { detail = (await res.json()).error || ''; } catch (_) {}
if (res.status === 401) {
showToast(detail === 'invalid credentials' ? 'invalid user id or password' : ('login failed: ' + (detail || res.status)), 'error');
} else if (res.status === 403 && detail === 'account inactive') {
showToast('account inactive — contact an admin', 'error');
} else {
showToast('login failed (' + res.status + ')', 'error');
}
}
} catch (e) {
showToast('network error — check connection', 'error');
}
}
// Check if already signed in (cookie-based — server will redirect to dashboard)
// The "signed in" card is only shown if the server served the landing page
// despite a valid cookie, which shouldn't happen. If it does, offer a redirect.
(async () => {
// If the server served the landing page, we're not authenticated via cookie.
// Try localStorage token as fallback (for backward compat with old sessions).
const token = localStorage.getItem('ctxd_session_token');
if (token) {
try {
const res = await fetch('/auth/me', { headers: { Authorization: 'Bearer ' + token } });
if (res.ok) {
// Token works via Bearer but cookie wasn't set — force redirect with token in cookie
// Re-login to get the cookie set, or just redirect (the dashboard JS uses Bearer too)
document.getElementById('login-card').innerHTML = '<h2>signed in</h2><p style="font-size:0.75rem;color:var(--ink-dim);margin-bottom:0.75rem">You are signed in.</p><div class="actions"><button class="primary" onclick="window.location.href=\'/\'">open dashboard</button></div>';
}
} catch (_) {}
}
})();
// Status check
(async () => {
try {
await fetch('/status');
document.getElementById('status').innerHTML = '<span class="dot"></span> connected';
} catch (_) {
document.getElementById('status').innerHTML = 'disconnected';
}
})();
</script>
</body>
</html>
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""
Migrate data from SQLite to PostgreSQL.
Usage:
DATABASE_URL=postgresql://ctxd:ctxd_local_dev@localhost:5432/ctxd \
SQLITE_PATH=/data/ctxd.db \
python3 -m ctxd.migrate_sqlite_to_pg
Or inside the Docker container:
docker exec dossier python3 -m ctxd.migrate_sqlite_to_pg
Environment variables:
DATABASE_URL — PostgreSQL connection string (required)
SQLITE_PATH — Path to SQLite DB file (default: /data/ctxd.db)
"""
import os
import sqlite3
import sys
try:
import psycopg
from psycopg.rows import dict_row
except ImportError:
print("ERROR: psycopg is required. Install with: pip install psycopg[binary]")
sys.exit(1)
SQLITE_PATH = os.environ.get("SQLITE_PATH", "/data/ctxd.db")
PG_URL = os.environ.get("DATABASE_URL", "")
# Tables in migration order (parents before children for FK safety)
TABLES = [
"users",
"projects",
"project_permissions",
"project_context",
"context_files",
"user_profiles",
"user_workspaces",
"workspace_files",
"change_requests",
"reviews",
"snapshots",
"audit_log",
]
# Columns that need type conversion from SQLite INTEGER to PostgreSQL BOOLEAN
BOOL_COLUMNS = {
"users": ["active"],
"projects": ["auto_sync"],
}
def migrate():
if not PG_URL:
print("ERROR: DATABASE_URL environment variable not set")
print("Example: DATABASE_URL=postgresql://ctxd:ctxd_local_dev@localhost:5432/ctxd")
sys.exit(1)
if not os.path.exists(SQLITE_PATH):
print(f"ERROR: SQLite database not found at {SQLITE_PATH}")
sys.exit(1)
print(f"Migrating from SQLite ({SQLITE_PATH}) to PostgreSQL ({PG_URL})")
print()
# Connect to SQLite
sconn = sqlite3.connect(SQLITE_PATH)
sconn.row_factory = sqlite3.Row
# Connect to PostgreSQL
pconn = psycopg.connect(PG_URL, row_factory=dict_row)
pconn.autocommit = False
# Step 1: Clear all data from PostgreSQL tables
print("Clearing existing PostgreSQL data...")
pconn.execute(
"TRUNCATE TABLE fts_context, audit_log, reviews, change_requests, "
"workspace_files, user_workspaces, context_files, project_context, "
"snapshots, user_profiles, project_permissions, projects, users CASCADE"
)
pconn.commit()
print(" Done.")
print()
# Step 2: Migrate each table
total_rows = 0
for table in TABLES:
# Get column names from SQLite
try:
cur = sconn.execute(f"PRAGMA table_info({table})")
columns = [row["name"] for row in cur.fetchall()]
except sqlite3.OperationalError:
print(f" {table}: table not found in SQLite, skipping")
continue
if not columns:
print(f" {table}: no columns found, skipping")
continue
# Read all rows from SQLite
col_str = ", ".join(columns)
try:
cur = sconn.execute(f"SELECT {col_str} FROM {table}")
rows = cur.fetchall()
except sqlite3.OperationalError:
print(f" {table}: error reading from SQLite, skipping")
continue
if not rows:
print(f" {table}: 0 rows")
continue
# Insert into PostgreSQL
val_str = ", ".join(["%s"] * len(columns))
insert_sql = f'INSERT INTO {table} ({col_str}) VALUES ({val_str}) ON CONFLICT DO NOTHING'
bool_cols = BOOL_COLUMNS.get(table, [])
count = 0
for row in rows:
values = []
for col in columns:
val = row[col]
# Convert integer 0/1 to boolean for PostgreSQL BOOLEAN columns
if col in bool_cols and val is not None:
val = bool(val)
values.append(val)
try:
pconn.execute(insert_sql, values)
count += 1
except Exception as fk_err:
# Skip rows with FK violations (e.g. orphaned snapshots)
pconn.rollback()
count_skipped = count_skipped + 1 if 'count_skipped' in dir() else 1
continue
pconn.commit()
total_rows += count
print(f" {table}: {count} rows migrated")
# Step 3: Rebuild FTS index
print()
print("Rebuilding FTS index...")
pconn.execute("DELETE FROM fts_context")
pconn.commit()
# Re-populate FTS from source tables
pconn.execute("""
INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
SELECT project_id, content, project_id, 'context.md', 'project_context',
to_tsvector('english', content)
FROM project_context
WHERE content != ''
""")
fts_pc = pconn.cursor().rowcount
pconn.execute("""
INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
SELECT file_id::text, content, project_id, file_path, 'context_file',
to_tsvector('english', content)
FROM context_files
WHERE content != ''
""")
fts_cf = pconn.cursor().rowcount
pconn.execute("""
INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
SELECT user_id, content, '~user', user_id, 'user_profile',
to_tsvector('english', content)
FROM user_profiles
WHERE content != ''
""")
fts_up = pconn.cursor().rowcount
pconn.commit()
print(f" project_context: {fts_pc} entries")
print(f" context_files: {fts_cf} entries")
print(f" user_profiles: {fts_up} entries")
# Step 4: Reset SERIAL sequences to max(existing_id) + 1
print()
print("Resetting SERIAL sequences...")
for table, col in [
("audit_log", "entry_id"),
("context_files", "file_id"),
("project_permissions", "id"),
("reviews", "review_id"),
("workspace_files", "file_id"),
]:
try:
pconn.execute(f"SELECT setval(pg_get_serial_sequence('{table}', '{col}'), COALESCE((SELECT MAX({col}) FROM {table}), 1))")
print(f" {table}.{col}: seq reset")
except Exception as e:
print(f" {table}.{col}: skipped ({e})")
pconn.commit()
print()
print("=" * 60)
print(f"Migration complete! {total_rows} total rows migrated.")
print("=" * 60)
sconn.close()
pconn.close()
if __name__ == "__main__":
migrate()
+84
View File
@@ -0,0 +1,84 @@
"""
Migration: alter user FK constraints to ON DELETE SET NULL.
PostgreSQL doesn't support ALTER CONSTRAINT inline — you must drop and
re-add the constraint. This script does that for all user_id FKs that
were originally created as RESTRICT (no ON DELETE action).
Run once against the production database:
docker exec ctxd python3 -m ctxd.migrate_user_fk_set_null
Safe to run multiple times (skips if constraint already has ON DELETE SET NULL).
"""
import os
import psycopg
# Tables + constraints to fix. constraint_name is the auto-generated PG name.
FKS_TO_FIX = [
("project_context", "project_context_updated_by_fkey", "updated_by"),
("context_files", "context_files_updated_by_fkey", "updated_by"),
("change_requests", "change_requests_submitted_by_fkey", "submitted_by"),
("reviews", "reviews_reviewer_id_fkey", "reviewer_id"),
("audit_log", "audit_log_user_id_fkey", "user_id"),
]
# Columns that were NOT NULL but must be nullable for ON DELETE SET NULL to work
DROP_NOT_NULL = [
("audit_log", "user_id"),
("change_requests", "submitted_by"),
("reviews", "reviewer_id"),
]
def main():
url = os.environ.get("DATABASE_URL")
if not url:
print("DATABASE_URL not set — nothing to do (SQLite mode)")
return
conn = psycopg.connect(url)
conn.autocommit = True
# Step 1: drop NOT NULL where needed (ON DELETE SET NULL requires nullable)
for table, col in DROP_NOT_NULL:
row = conn.execute(
f"SELECT is_nullable FROM information_schema.columns WHERE table_name = %s AND column_name = %s",
(table, col),
).fetchone()
if row and row[0] == "NO":
print(f" DROP NOT NULL {table}.{col}")
conn.execute(f"ALTER TABLE {table} ALTER COLUMN {col} DROP NOT NULL")
print(f" DONE {table}.{col}")
else:
print(f" SKIP {table}.{col}: already nullable")
# Step 2: fix FK constraints
for table, constraint, col in FKS_TO_FIX:
# Check if the constraint exists and what its current ON DELETE action is
row = conn.execute(
"""SELECT confdeltype FROM pg_constraint
WHERE conname = %s AND contype = 'f'""",
(constraint,),
).fetchone()
if not row:
print(f" SKIP {constraint}: not found (already migrated or table missing)")
continue
if row[0] == "n": # 'n' = SET NULL
print(f" SKIP {constraint}: already ON DELETE SET NULL")
continue
# `col` comes straight from the FKS_TO_FIX tuple — don't re-derive it
# from the constraint name, which breaks for manually-named constraints.
print(f" ALTER {table}.{col} ({constraint}): {row[0]} -> SET NULL")
conn.execute(f'ALTER TABLE {table} DROP CONSTRAINT {constraint}')
conn.execute(
f'ALTER TABLE {table} ADD CONSTRAINT {constraint} '
f'FOREIGN KEY ({col}) REFERENCES users(user_id) ON DELETE SET NULL'
)
print(f" DONE {constraint}")
conn.close()
print("Migration complete.")
if __name__ == "__main__":
main()
+174 -121
View File
@@ -1,23 +1,20 @@
-- ============================================================================ -- ============================================================================
-- ctxd — Context Daemon Schema -- ctxd — Context Daemon Schema
-- SQLite 3.x, WAL mode, FTS5 -- PostgreSQL 16
-- ============================================================================ -- ============================================================================
-- WAL for concurrent reads during writes; foreign keys enforced
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
-- ============================================================================ -- ============================================================================
-- USERS -- USERS
-- ============================================================================ -- ============================================================================
CREATE TABLE users ( CREATE TABLE users (
user_id TEXT PRIMARY KEY, -- uuid or "joshua", "polly", "hermes-gateway" user_id TEXT PRIMARY KEY,
display_name TEXT NOT NULL, display_name TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'contributor' role TEXT NOT NULL DEFAULT 'contributor'
CHECK (role IN ('admin', 'contributor', 'service')), CHECK (role IN ('admin', 'contributor', 'service')),
token_hash TEXT, -- NULL = no auth (localhost/trusted) token_hash TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), active BOOLEAN NOT NULL DEFAULT TRUE,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')) created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
CREATE UNIQUE INDEX idx_users_lower ON users (LOWER(user_id)); CREATE UNIQUE INDEX idx_users_lower ON users (LOWER(user_id));
@@ -26,22 +23,22 @@ CREATE UNIQUE INDEX idx_users_lower ON users (LOWER(user_id));
-- PROJECTS -- PROJECTS
-- ============================================================================ -- ============================================================================
CREATE TABLE projects ( CREATE TABLE projects (
project_id TEXT PRIMARY KEY, -- uuid or slug "remote-rig" project_id TEXT PRIMARY KEY,
display_name TEXT NOT NULL, display_name TEXT NOT NULL,
description TEXT, description TEXT,
metadata_tags TEXT DEFAULT '[]', -- JSON array of tag strings e.g. '["ARCHITECTURE","3D-PRINTING"]' metadata_tags TEXT DEFAULT '[]',
shared_version INTEGER NOT NULL DEFAULT 0, -- monotonically increasing shared_version INTEGER NOT NULL DEFAULT 0,
auto_sync INTEGER NOT NULL DEFAULT 0, -- boolean: auto-write AGENTS.md to sync_path auto_sync BOOLEAN NOT NULL DEFAULT FALSE,
sync_path TEXT, -- absolute path to project root (nullable) sync_path TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')) updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
-- ============================================================================ -- ============================================================================
-- PROJECT PERMISSIONS (admin overrides all) -- PROJECT PERMISSIONS (admin overrides all)
-- ============================================================================ -- ============================================================================
CREATE TABLE project_permissions ( CREATE TABLE project_permissions (
id INTEGER PRIMARY KEY AUTOINCREMENT, id SERIAL PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE, project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
user_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE, user_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE,
permission TEXT NOT NULL DEFAULT 'editor' permission TEXT NOT NULL DEFAULT 'editor'
@@ -54,9 +51,9 @@ CREATE TABLE project_permissions (
-- ============================================================================ -- ============================================================================
CREATE TABLE user_profiles ( CREATE TABLE user_profiles (
user_id TEXT PRIMARY KEY REFERENCES users(user_id) ON DELETE CASCADE, user_id TEXT PRIMARY KEY REFERENCES users(user_id) ON DELETE CASCADE,
content TEXT NOT NULL DEFAULT '', -- markdown content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1, version INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')) updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
-- ============================================================================ -- ============================================================================
@@ -66,10 +63,10 @@ CREATE TABLE user_profiles (
-- ============================================================================ -- ============================================================================
CREATE TABLE project_context ( CREATE TABLE project_context (
project_id TEXT PRIMARY KEY REFERENCES projects(project_id) ON DELETE CASCADE, project_id TEXT PRIMARY KEY REFERENCES projects(project_id) ON DELETE CASCADE,
content TEXT NOT NULL DEFAULT '', -- compiled markdown content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 0, -- mirrors projects.shared_version version INTEGER NOT NULL DEFAULT 0,
updated_by TEXT REFERENCES users(user_id), updated_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')) updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
-- ============================================================================ -- ============================================================================
@@ -78,13 +75,13 @@ CREATE TABLE project_context (
-- version tracks this file's edit count (independent of the shared version). -- version tracks this file's edit count (independent of the shared version).
-- ============================================================================ -- ============================================================================
CREATE TABLE context_files ( CREATE TABLE context_files (
file_id INTEGER PRIMARY KEY AUTOINCREMENT, file_id SERIAL PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE, project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
file_path TEXT NOT NULL, -- "decisions/001-use-go.md" file_path TEXT NOT NULL,
content TEXT NOT NULL DEFAULT '', content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1, -- per-file edit counter version INTEGER NOT NULL DEFAULT 1,
updated_by TEXT REFERENCES users(user_id), updated_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
UNIQUE(project_id, file_path) UNIQUE(project_id, file_path)
); );
@@ -94,16 +91,16 @@ CREATE TABLE context_files (
-- shared version they started from. current_version tracks their edits. -- shared version they started from. current_version tracks their edits.
-- ============================================================================ -- ============================================================================
CREATE TABLE user_workspaces ( CREATE TABLE user_workspaces (
workspace_id TEXT PRIMARY KEY, -- uuid workspace_id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE, user_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE, project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'in_progress' status TEXT NOT NULL DEFAULT 'in_progress'
CHECK (status IN ('in_progress', 'submitted', 'merged', 'abandoned')), CHECK (status IN ('in_progress', 'submitted', 'merged', 'abandoned')),
base_version INTEGER NOT NULL, -- shared version at fork time base_version INTEGER NOT NULL,
current_version INTEGER NOT NULL DEFAULT 1, current_version INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
UNIQUE(user_id, project_id, status) -- one active workspace per user per project UNIQUE(user_id, project_id, status)
); );
-- ============================================================================ -- ============================================================================
@@ -111,12 +108,12 @@ CREATE TABLE user_workspaces (
-- Mirrors the same file_path as context_files but in the user's workspace. -- Mirrors the same file_path as context_files but in the user's workspace.
-- ============================================================================ -- ============================================================================
CREATE TABLE workspace_files ( CREATE TABLE workspace_files (
file_id INTEGER PRIMARY KEY AUTOINCREMENT, file_id SERIAL PRIMARY KEY,
workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE, workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE,
file_path TEXT NOT NULL, file_path TEXT NOT NULL,
content TEXT NOT NULL DEFAULT '', content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1, version INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
UNIQUE(workspace_id, file_path) UNIQUE(workspace_id, file_path)
); );
@@ -124,30 +121,29 @@ CREATE TABLE workspace_files (
-- CHANGE REQUESTS — submit / review / merge workflow -- CHANGE REQUESTS — submit / review / merge workflow
-- ============================================================================ -- ============================================================================
CREATE TABLE change_requests ( CREATE TABLE change_requests (
request_id TEXT PRIMARY KEY, -- uuid request_id TEXT PRIMARY KEY,
workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE, workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE,
project_id TEXT NOT NULL REFERENCES projects(project_id), project_id TEXT NOT NULL REFERENCES projects(project_id),
submitted_by TEXT NOT NULL REFERENCES users(user_id), submitted_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
status TEXT NOT NULL DEFAULT 'pending' status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'approved', 'rejected', 'merged')), CHECK (status IN ('pending', 'approved', 'rejected', 'merged')),
-- Snapshot of what changed, stored inline so reviews survive workspace mutation diff_summary TEXT,
diff_summary TEXT, -- free-text summary of changes target_version INTEGER NOT NULL,
target_version INTEGER NOT NULL, -- the shared version this would bump to base_version INTEGER NOT NULL,
base_version INTEGER NOT NULL, -- the shared version they forked from created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), updated_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
); );
-- ============================================================================ -- ============================================================================
-- REVIEWS — approvals/rejections on change requests -- REVIEWS — approvals/rejections on change requests
-- ============================================================================ -- ============================================================================
CREATE TABLE reviews ( CREATE TABLE reviews (
review_id INTEGER PRIMARY KEY AUTOINCREMENT, review_id SERIAL PRIMARY KEY,
request_id TEXT NOT NULL REFERENCES change_requests(request_id) ON DELETE CASCADE, request_id TEXT NOT NULL REFERENCES change_requests(request_id) ON DELETE CASCADE,
reviewer_id TEXT NOT NULL REFERENCES users(user_id), reviewer_id TEXT REFERENCES users(user_id) ON DELETE SET NULL,
decision TEXT NOT NULL CHECK (decision IN ('approved', 'rejected')), decision TEXT NOT NULL CHECK (decision IN ('approved', 'rejected')),
comments TEXT, comments TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"'),
UNIQUE(request_id, reviewer_id) UNIQUE(request_id, reviewer_id)
); );
@@ -156,31 +152,29 @@ CREATE TABLE reviews (
-- Stored as files on disk at the path in storage_path. -- Stored as files on disk at the path in storage_path.
-- ============================================================================ -- ============================================================================
CREATE TABLE snapshots ( CREATE TABLE snapshots (
snapshot_id TEXT PRIMARY KEY, -- uuid snapshot_id TEXT PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE, project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
-- NULL user_id = snapshot of the shared copy; non-NULL = snapshot of a user workspace
user_id TEXT REFERENCES users(user_id) ON DELETE CASCADE, user_id TEXT REFERENCES users(user_id) ON DELETE CASCADE,
workspace_id TEXT REFERENCES user_workspaces(workspace_id) ON DELETE SET NULL, workspace_id TEXT REFERENCES user_workspaces(workspace_id) ON DELETE SET NULL,
version_from INTEGER, -- version range this snapshot covers version_from INTEGER,
version_to INTEGER, version_to INTEGER,
storage_path TEXT NOT NULL, -- relative to ~/.ctx/snapshots/ storage_path TEXT NOT NULL,
content_hash TEXT NOT NULL, -- sha256 of the compiled markdown content_hash TEXT NOT NULL,
size_bytes INTEGER NOT NULL DEFAULT 0, size_bytes INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')) created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
-- Index for snapshot rotation queries
CREATE INDEX idx_snapshots_cleanup ON snapshots (project_id, user_id, created_at); CREATE INDEX idx_snapshots_cleanup ON snapshots (project_id, user_id, created_at);
-- ============================================================================ -- ============================================================================
-- AUDIT LOG — append-only (INSERT only, never UPDATE or DELETE) -- AUDIT LOG — append-only (INSERT only, never UPDATE or DELETE)
-- ============================================================================ -- ============================================================================
CREATE TABLE audit_log ( CREATE TABLE audit_log (
entry_id INTEGER PRIMARY KEY AUTOINCREMENT, entry_id SERIAL PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(user_id), user_id TEXT REFERENCES users(user_id) ON DELETE SET NULL,
agent_id TEXT NOT NULL DEFAULT 'cli', -- "hermes", "claude-code", "ctx" agent_id TEXT NOT NULL DEFAULT 'cli',
session_id TEXT, -- opaque session identifier session_id TEXT,
project_id TEXT REFERENCES projects(project_id), project_id TEXT REFERENCES projects(project_id) ON DELETE SET NULL,
operation TEXT NOT NULL operation TEXT NOT NULL
CHECK (operation IN ( CHECK (operation IN (
'read', 'update', 'create', 'delete', 'read', 'update', 'create', 'delete',
@@ -188,105 +182,164 @@ CREATE TABLE audit_log (
'sync', 'search', 'export', 'restore', 'sync', 'search', 'export', 'restore',
'login', 'logout', 'import' 'login', 'logout', 'import'
)), )),
entity_type TEXT, -- 'project', 'workspace', 'change_request', 'snapshot', 'user_profile' entity_type TEXT,
entity_id TEXT, -- polymorphic reference entity_id TEXT,
summary TEXT NOT NULL, -- human-readable: "Updated camera-node wiring section" summary TEXT NOT NULL,
details_json TEXT, -- structured payload: diff, version numbers, etc. details_json TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')) created_at TEXT NOT NULL DEFAULT to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
); );
-- Audit queries by user, project, or time range
CREATE INDEX idx_audit_user ON audit_log (user_id, created_at); CREATE INDEX idx_audit_user ON audit_log (user_id, created_at);
CREATE INDEX idx_audit_project ON audit_log (project_id, created_at); CREATE INDEX idx_audit_project ON audit_log (project_id, created_at);
CREATE INDEX idx_audit_agent ON audit_log (agent_id, created_at); CREATE INDEX idx_audit_agent ON audit_log (agent_id, created_at);
CREATE INDEX idx_audit_op ON audit_log (operation, created_at); CREATE INDEX idx_audit_op ON audit_log (operation, created_at);
-- Trigger: audit_log is append-only enforce no updates or deletes at the DB level -- Note: audit_log append-only enforcement is handled at the application layer.
CREATE TRIGGER tr_audit_log_no_update -- DB-level BEFORE UPDATE/DELETE triggers conflict with FK ON DELETE SET NULL
BEFORE UPDATE ON audit_log -- cascades from projects, which internally issue UPDATE statements.
BEGIN
SELECT RAISE(ABORT, 'audit_log is append-only — no updates allowed');
END;
CREATE TRIGGER tr_audit_log_no_delete
BEFORE DELETE ON audit_log
BEGIN
SELECT RAISE(ABORT, 'audit_log is append-only — no deletes allowed');
END;
-- ============================================================================ -- ============================================================================
-- FULL-TEXT SEARCH (FTS5) -- FULL-TEXT SEARCH (tsvector with GIN index)
-- Separate FTS table with triggers to keep index in sync with source tables.
-- ============================================================================ -- ============================================================================
CREATE VIRTUAL TABLE fts_context USING fts5( CREATE TABLE fts_context (
content, id SERIAL PRIMARY KEY,
project_id UNINDEXED, source_id TEXT NOT NULL,
file_path UNINDEXED, content TEXT NOT NULL,
source_type UNINDEXED, -- 'project_context', 'context_file', 'user_profile', 'workspace_file' project_id TEXT NOT NULL,
tokenize='porter unicode61' file_path TEXT NOT NULL,
source_type TEXT NOT NULL,
tsv tsvector NOT NULL,
UNIQUE(source_type, source_id)
); );
-- Triggers to keep FTS index in sync with project_context CREATE INDEX idx_fts_context_tsv ON fts_context USING GIN (tsv);
CREATE TRIGGER tr_fts_project_context_insert AFTER INSERT ON project_context CREATE INDEX idx_fts_context_project ON fts_context (project_id);
BEGIN CREATE INDEX idx_fts_context_source ON fts_context (source_type, source_id);
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.rowid, NEW.content, NEW.project_id, 'context.md', 'project_context');
END;
CREATE TRIGGER tr_fts_project_context_update AFTER UPDATE ON project_context -- ── Trigger functions for project_context ───────────────────────────────────
BEGIN
DELETE FROM fts_context WHERE rowid = OLD.rowid;
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.rowid, NEW.content, NEW.project_id, 'context.md', 'project_context');
END;
CREATE TRIGGER tr_fts_project_context_delete AFTER DELETE ON project_context CREATE OR REPLACE FUNCTION fts_pc_insert() RETURNS TRIGGER AS $$
BEGIN BEGIN
DELETE FROM fts_context WHERE rowid = OLD.rowid; INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
VALUES (NEW.project_id, NEW.content, NEW.project_id, 'context.md', 'project_context',
to_tsvector('english', NEW.content));
RETURN NEW;
END; END;
$$ LANGUAGE plpgsql;
-- Triggers for context_files CREATE TRIGGER tr_fts_pc_insert AFTER INSERT ON project_context
CREATE TRIGGER tr_fts_context_files_insert AFTER INSERT ON context_files FOR EACH ROW EXECUTE FUNCTION fts_pc_insert();
BEGIN
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.file_id + 1000000, NEW.content, NEW.project_id, NEW.file_path, 'context_file');
END;
CREATE TRIGGER tr_fts_context_files_update AFTER UPDATE ON context_files CREATE OR REPLACE FUNCTION fts_pc_update() RETURNS TRIGGER AS $$
BEGIN BEGIN
DELETE FROM fts_context WHERE rowid = OLD.file_id + 1000000; DELETE FROM fts_context WHERE source_type = 'project_context' AND source_id = OLD.project_id;
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type) INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
VALUES (NEW.file_id + 1000000, NEW.content, NEW.project_id, NEW.file_path, 'context_file'); VALUES (NEW.project_id, NEW.content, NEW.project_id, 'context.md', 'project_context',
to_tsvector('english', NEW.content));
RETURN NEW;
END; END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_context_files_delete AFTER DELETE ON context_files CREATE TRIGGER tr_fts_pc_update AFTER UPDATE ON project_context
BEGIN FOR EACH ROW EXECUTE FUNCTION fts_pc_update();
DELETE FROM fts_context WHERE rowid = OLD.file_id + 1000000;
END;
-- Triggers for user_profiles CREATE OR REPLACE FUNCTION fts_pc_delete() RETURNS TRIGGER AS $$
CREATE TRIGGER tr_fts_user_profiles_insert AFTER INSERT ON user_profiles
BEGIN BEGIN
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type) DELETE FROM fts_context WHERE source_type = 'project_context' AND source_id = OLD.project_id;
VALUES (NEW.rowid + 2000000, NEW.content, '~user~', NEW.user_id, 'user_profile'); RETURN OLD;
END; END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_user_profiles_update AFTER UPDATE ON user_profiles CREATE TRIGGER tr_fts_pc_delete AFTER DELETE ON project_context
FOR EACH ROW EXECUTE FUNCTION fts_pc_delete();
-- ── Trigger functions for context_files ─────────────────────────────────────
CREATE OR REPLACE FUNCTION fts_cf_insert() RETURNS TRIGGER AS $$
BEGIN BEGIN
DELETE FROM fts_context WHERE rowid = OLD.rowid + 2000000; INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type) VALUES (NEW.file_id::text, NEW.content, NEW.project_id, NEW.file_path, 'context_file',
VALUES (NEW.rowid + 2000000, NEW.content, '~user~', NEW.user_id, 'user_profile'); to_tsvector('english', NEW.content));
RETURN NEW;
END; END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_cf_insert AFTER INSERT ON context_files
FOR EACH ROW EXECUTE FUNCTION fts_cf_insert();
CREATE OR REPLACE FUNCTION fts_cf_update() RETURNS TRIGGER AS $$
BEGIN
DELETE FROM fts_context WHERE source_type = 'context_file' AND source_id = OLD.file_id::text;
INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
VALUES (NEW.file_id::text, NEW.content, NEW.project_id, NEW.file_path, 'context_file',
to_tsvector('english', NEW.content));
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_cf_update AFTER UPDATE ON context_files
FOR EACH ROW EXECUTE FUNCTION fts_cf_update();
CREATE OR REPLACE FUNCTION fts_cf_delete() RETURNS TRIGGER AS $$
BEGIN
DELETE FROM fts_context WHERE source_type = 'context_file' AND source_id = OLD.file_id::text;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_cf_delete AFTER DELETE ON context_files
FOR EACH ROW EXECUTE FUNCTION fts_cf_delete();
-- ── Trigger functions for user_profiles ─────────────────────────────────────
CREATE OR REPLACE FUNCTION fts_up_insert() RETURNS TRIGGER AS $$
BEGIN
INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
VALUES (NEW.user_id, NEW.content, '~user', NEW.user_id, 'user_profile',
to_tsvector('english', NEW.content));
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_up_insert AFTER INSERT ON user_profiles
FOR EACH ROW EXECUTE FUNCTION fts_up_insert();
CREATE OR REPLACE FUNCTION fts_up_update() RETURNS TRIGGER AS $$
BEGIN
DELETE FROM fts_context WHERE source_type = 'user_profile' AND source_id = OLD.user_id;
INSERT INTO fts_context (source_id, content, project_id, file_path, source_type, tsv)
VALUES (NEW.user_id, NEW.content, '~user', NEW.user_id, 'user_profile',
to_tsvector('english', NEW.content));
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_up_update AFTER UPDATE ON user_profiles
FOR EACH ROW EXECUTE FUNCTION fts_up_update();
CREATE OR REPLACE FUNCTION fts_up_delete() RETURNS TRIGGER AS $$
BEGIN
DELETE FROM fts_context WHERE source_type = 'user_profile' AND source_id = OLD.user_id;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER tr_fts_up_delete AFTER DELETE ON user_profiles
FOR EACH ROW EXECUTE FUNCTION fts_up_delete();
-- ============================================================================ -- ============================================================================
-- SEED DATA (for development / first-run) -- SEED DATA (for development / first-run)
-- ============================================================================ -- ============================================================================
INSERT INTO users (user_id, display_name, role) VALUES INSERT INTO users (user_id, display_name, role) VALUES
('admin', 'Administrator', 'admin'), ('admin', 'Administrator', 'admin'),
('hermes-gateway', 'Hermes Agent', 'service'); ('hermes-gateway', 'Hermes Agent', 'service')
ON CONFLICT DO NOTHING;
INSERT INTO projects (project_id, display_name, description) VALUES INSERT INTO projects (project_id, display_name, description) VALUES
('welcome', 'Welcome', 'Getting started guide and documentation for ctxd'), ('welcome', 'Welcome', 'Getting started guide and documentation for ctxd'),
('remote-rig', 'RemoteRig', 'Multi-camera remote monitoring system'); ('remote-rig', 'RemoteRig', 'Multi-camera remote monitoring system')
ON CONFLICT DO NOTHING;
-- Project context is seeded by the Python init code (cmd_init) -- Project context is seeded by the Python init code (cmd_init)
-- to ensure real newlines, not literal backslash-n from SQL strings. -- to ensure real newlines, not literal backslash-n from SQL strings.
+275
View File
@@ -0,0 +1,275 @@
-- ============================================================================
-- ctxd — Context Daemon Schema
-- SQLite 3.x, WAL mode, FTS5
-- (Legacy fallback for local dev when DATABASE_URL is not set)
-- ============================================================================
-- WAL for concurrent reads during writes; foreign keys enforced
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
-- ============================================================================
-- USERS
-- ============================================================================
CREATE TABLE users (
user_id TEXT PRIMARY KEY,
display_name TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'contributor'
CHECK (role IN ('admin', 'contributor', 'service')),
token_hash TEXT,
active INTEGER NOT NULL DEFAULT 1 CHECK (active IN (0, 1)),
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
CREATE UNIQUE INDEX idx_users_lower ON users (LOWER(user_id));
-- ============================================================================
-- PROJECTS
-- ============================================================================
CREATE TABLE projects (
project_id TEXT PRIMARY KEY,
display_name TEXT NOT NULL,
description TEXT,
metadata_tags TEXT DEFAULT '[]',
shared_version INTEGER NOT NULL DEFAULT 0,
auto_sync INTEGER NOT NULL DEFAULT 0,
sync_path TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
-- ============================================================================
-- PROJECT PERMISSIONS (admin overrides all)
-- ============================================================================
CREATE TABLE project_permissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
user_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE,
permission TEXT NOT NULL DEFAULT 'editor'
CHECK (permission IN ('owner', 'editor', 'viewer')),
UNIQUE(project_id, user_id)
);
-- ============================================================================
-- USER PROFILES (personal context — timezone, preferences, style)
-- ============================================================================
CREATE TABLE user_profiles (
user_id TEXT PRIMARY KEY REFERENCES users(user_id) ON DELETE CASCADE,
content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
-- ============================================================================
-- PROJECT CONTEXT — THE AUTHORITATIVE SHARED COPY
-- ============================================================================
CREATE TABLE project_context (
project_id TEXT PRIMARY KEY REFERENCES projects(project_id) ON DELETE CASCADE,
content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 0,
updated_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
-- ============================================================================
-- CONTEXT FILES — individual files within a project
-- ============================================================================
CREATE TABLE context_files (
file_id INTEGER PRIMARY KEY AUTOINCREMENT,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1,
updated_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(project_id, file_path)
);
-- ============================================================================
-- USER WORKSPACES — per-user forks of a project
-- ============================================================================
CREATE TABLE user_workspaces (
workspace_id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'in_progress'
CHECK (status IN ('in_progress', 'submitted', 'merged', 'abandoned')),
base_version INTEGER NOT NULL,
current_version INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(user_id, project_id, status)
);
-- ============================================================================
-- WORKSPACE FILES — per-user fork of context_files
-- ============================================================================
CREATE TABLE workspace_files (
file_id INTEGER PRIMARY KEY AUTOINCREMENT,
workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
content TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(workspace_id, file_path)
);
-- ============================================================================
-- CHANGE REQUESTS — submit / review / merge workflow
-- ============================================================================
CREATE TABLE change_requests (
request_id TEXT PRIMARY KEY,
workspace_id TEXT NOT NULL REFERENCES user_workspaces(workspace_id) ON DELETE CASCADE,
project_id TEXT NOT NULL REFERENCES projects(project_id),
submitted_by TEXT REFERENCES users(user_id) ON DELETE SET NULL,
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'approved', 'rejected', 'merged')),
diff_summary TEXT,
target_version INTEGER NOT NULL,
base_version INTEGER NOT NULL,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
-- ============================================================================
-- REVIEWS — approvals/rejections on change requests
-- ============================================================================
CREATE TABLE reviews (
review_id INTEGER PRIMARY KEY AUTOINCREMENT,
request_id TEXT NOT NULL REFERENCES change_requests(request_id) ON DELETE CASCADE,
reviewer_id TEXT REFERENCES users(user_id) ON DELETE SET NULL,
decision TEXT NOT NULL CHECK (decision IN ('approved', 'rejected')),
comments TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(request_id, reviewer_id)
);
-- ============================================================================
-- SNAPSHOTS — point-in-time copies of project or workspace content
-- ============================================================================
CREATE TABLE snapshots (
snapshot_id TEXT PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(project_id) ON DELETE CASCADE,
user_id TEXT REFERENCES users(user_id) ON DELETE CASCADE,
workspace_id TEXT REFERENCES user_workspaces(workspace_id) ON DELETE SET NULL,
version_from INTEGER,
version_to INTEGER,
storage_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
size_bytes INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
CREATE INDEX idx_snapshots_cleanup ON snapshots (project_id, user_id, created_at);
-- ============================================================================
-- AUDIT LOG — append-only (INSERT only, never UPDATE or DELETE)
-- ============================================================================
CREATE TABLE audit_log (
entry_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id TEXT REFERENCES users(user_id) ON DELETE SET NULL,
agent_id TEXT NOT NULL DEFAULT 'cli',
session_id TEXT,
project_id TEXT REFERENCES projects(project_id),
operation TEXT NOT NULL
CHECK (operation IN (
'read', 'update', 'create', 'delete',
'submit', 'approve', 'reject', 'merge',
'sync', 'search', 'export', 'restore',
'login', 'logout', 'import'
)),
entity_type TEXT,
entity_id TEXT,
summary TEXT NOT NULL,
details_json TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
CREATE INDEX idx_audit_user ON audit_log (user_id, created_at);
CREATE INDEX idx_audit_project ON audit_log (project_id, created_at);
CREATE INDEX idx_audit_agent ON audit_log (agent_id, created_at);
CREATE INDEX idx_audit_op ON audit_log (operation, created_at);
-- Note: audit_log append-only enforcement is handled at the application layer.
-- DB-level BEFORE UPDATE/DELETE triggers conflict with FK ON DELETE SET NULL
-- (deleting a user must set audit_log.user_id to NULL, which is an UPDATE).
-- This mirrors schema.sql, which dropped the equivalent PG triggers for the
-- same reason. db.py only ever INSERTs into audit_log.
-- ============================================================================
-- FULL-TEXT SEARCH (FTS5)
-- ============================================================================
CREATE VIRTUAL TABLE fts_context USING fts5(
content,
project_id UNINDEXED,
file_path UNINDEXED,
source_type UNINDEXED,
tokenize='porter unicode61'
);
-- Triggers to keep FTS index in sync with project_context
CREATE TRIGGER tr_fts_project_context_insert AFTER INSERT ON project_context
BEGIN
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.rowid, NEW.content, NEW.project_id, 'context.md', 'project_context');
END;
CREATE TRIGGER tr_fts_project_context_update AFTER UPDATE ON project_context
BEGIN
DELETE FROM fts_context WHERE rowid = OLD.rowid;
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.rowid, NEW.content, NEW.project_id, 'context.md', 'project_context');
END;
CREATE TRIGGER tr_fts_project_context_delete AFTER DELETE ON project_context
BEGIN
DELETE FROM fts_context WHERE rowid = OLD.rowid;
END;
-- Triggers for context_files
CREATE TRIGGER tr_fts_context_files_insert AFTER INSERT ON context_files
BEGIN
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.file_id + 1000000, NEW.content, NEW.project_id, NEW.file_path, 'context_file');
END;
CREATE TRIGGER tr_fts_context_files_update AFTER UPDATE ON context_files
BEGIN
DELETE FROM fts_context WHERE rowid = OLD.file_id + 1000000;
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.file_id + 1000000, NEW.content, NEW.project_id, NEW.file_path, 'context_file');
END;
CREATE TRIGGER tr_fts_context_files_delete AFTER DELETE ON context_files
BEGIN
DELETE FROM fts_context WHERE rowid = OLD.file_id + 1000000;
END;
-- Triggers for user_profiles
CREATE TRIGGER tr_fts_user_profiles_insert AFTER INSERT ON user_profiles
BEGIN
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.rowid + 2000000, NEW.content, '~user', NEW.user_id, 'user_profile');
END;
CREATE TRIGGER tr_fts_user_profiles_update AFTER UPDATE ON user_profiles
BEGIN
DELETE FROM fts_context WHERE rowid = OLD.rowid + 2000000;
INSERT INTO fts_context(rowid, content, project_id, file_path, source_type)
VALUES (NEW.rowid + 2000000, NEW.content, '~user', NEW.user_id, 'user_profile');
END;
-- ============================================================================
-- SEED DATA (for development / first-run)
-- ============================================================================
INSERT INTO users (user_id, display_name, role) VALUES
('admin', 'Administrator', 'admin'),
('hermes-gateway', 'Hermes Agent', 'service');
INSERT INTO projects (project_id, display_name, description) VALUES
('welcome', 'Welcome', 'Getting started guide and documentation for ctxd'),
('remote-rig', 'RemoteRig', 'Multi-camera remote monitoring system');
-- Project context is seeded by the Python init code (cmd_init)
-- to ensure real newlines, not literal backslash-n from SQL strings.
+1257 -69
View File
File diff suppressed because it is too large Load Diff
+1036 -37
View File
File diff suppressed because it is too large Load Diff
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
View File
-14
View File
@@ -1,14 +0,0 @@
server:
host: 0.0.0.0
port: 9091
snapshots:
min_keep: 5
max_keep: 25
auth:
enabled: true
api_key: Fa50-7cg5x6ObyFjJCjBYQkaNhGF5gMBJluVY6C1OwE
seed:
admin_user: admin
admin_display: Administrator
service_user: hermes-gateway
service_display: Hermes Agent
+147
View File
@@ -0,0 +1,147 @@
# I Built a Context Server to Fight My ADHD-Induced Context Sprawl
**How replacing 7 scattered config files with a single source of truth saved my sanity (and my tokens).**
---
If you work with AI tools the way I do — Claude Desktop for research, Claude Code for implementation, Codex CLI for reviews, Cursor for quick edits, and Hermes Agent for orchestration — you have a problem you might not have named yet.
I call it **context sprawl**. And if you have ADHD, it's worse than you think.
## The Problem
Every AI harness has its own context file convention:
| Harness | File |
|---------|------|
| Hermes | `AGENTS.md` |
| Claude Code | `CLAUDE.md` |
| Cursor | `.cursorrules` |
| Copilot | `.github/copilot-instructions.md` |
| Codex CLI | `CODEX.md` |
| Windsurf | `.windsurfrules` |
| Continue.dev | `.continuerc` |
When you work on the same project across multiple harnesses, you end up with N copies of the same project context. They drift. One says the build system is Go; another still says Python. One knows about the new OAuth flow; another doesn't. The LLM reads stale instructions and makes confident, wrong decisions.
For someone with ADHD, this is a specific kind of hell:
- You **forget** which file you last updated
- You **avoid** updating because it means updating 4-6 files
- You **lose track** of which harness has the "right" version
- You **context-switch** between tools and can't remember what the project actually needs
- The friction of maintaining context across tools becomes so high that you **stop maintaining context at all**
And then every LLM session starts from scratch — re-asking questions you already answered, re-discovering decisions you already made, re-making mistakes you already fixed.
## The Tipping Point
I was working on a project called RemoteRig — a multi-camera remote monitoring system built in Go with ESP32-C6 camera nodes. I had context in:
- `AGENTS.md` (Hermes)
- `CLAUDE.md` (Claude Code)
- `.cursorrules` (Cursor)
- `CODEX.md` (Codex CLI)
- A Notion page (for humans, but also where I drafted things)
- My own memory of what I'd told each tool (which is the worst place to store anything when you have ADHD)
I'd update `AGENTS.md` with a new architecture decision, forget to update `CLAUDE.md`, and then Claude Code would suggest something I'd already ruled out. I'd spend the first 20 minutes of every session re-explaining the project. Sometimes I'd update the wrong file and wonder why Codex was still using the old Go module structure.
The context was sprawled across the filesystem like a hoarder's garage, and I was the only one who knew where anything was — except I didn't, because ADHD.
## The Solution: CTXD (Context Dossier)
I built **CTXD** — a single source of truth for project context that every AI harness can read from and write to.
### Architecture
```
Context Dossier (Docker container, port 9091)
├── PostgreSQL 16 backend
├── Multi-file context per project:
│ ├── CONTEXT.MD ← canonical overview, synced as AGENTS.md to repos
│ ├── DECISIONS.MD ← architecture decisions, rationale
│ ├── RUNBOOKS.MD ← deploy, troubleshoot, operate procedures
│ ├── PROMPTS.MD ← project-specific prompts for different harnesses
│ └── GLOSSARY.MD ← project-specific terms, acronyms
├── OAuth 2.0 authorization server
│ ├── Dynamic Client Registration (Claude auto-registers)
│ ├── Authorization Code + PKCE
│ └── Scopes: ctxd.read, ctxd.write
├── Streamable HTTP MCP endpoints:
│ ├── /readonly/mcp (read-only, OAuth ctxd.read)
│ ├── /write/mcp (read-write, OAuth ctxd.write)
│ └── /mcp (internal, shared API key)
├── Web UI (per-user login, admin panel)
├── Version-checked writes (optimistic concurrency)
├── Append-only audit log
├── Point-in-time snapshots (auto-rotated)
└── Full-text search (PostgreSQL tsvector + GIN)
```
### How it works
1. **One canonical file per project**`CONTEXT.MD` is the source of truth. CTXD syncs it to `AGENTS.md` in the project root and creates symlinks (`CLAUDE.md`, `.cursorrules`, `CODEX.md``AGENTS.md`). Every harness reads the same file.
2. **LLMs connect via MCP** — Claude Desktop connects to `https://ctxd.yourdomain.com/readonly/mcp` and auto-discovers OAuth. No manual config, no API keys in URLs. It can read project context, search across all projects, and list individual files.
3. **LLMs can write context** — With a `ctxd.write`-scoped OAuth token, an LLM can update files with version checking. If two agents try to edit simultaneously, the second one gets a `version_conflict` and knows to re-read, merge, and retry.
4. **The compiled view** — When an LLM calls `get_project_context("remote-rig")`, it gets all files concatenated into a single document with a metadata header. One call, complete context, no guessing.
5. **A locked client guide** — There's a `get_client_guide()` MCP tool that returns a locked `LLM-CLIENT.MD` file. Every LLM session starts by reading this guide, which explains how to connect, what tools are available, and what the discipline rules are. The file is immutable — no LLM can modify its own instruction manual.
### What it solves for ADHD
- **One place to update** — Change `CONTEXT.MD` once. Every harness gets it via sync or MCP. No more updating 6 files.
- **No remembering required** — The LLM reads context at the start of every session. You don't have to remember what you told it last time.
- **Audit trail** — Every read, write, and search is logged. When you can't remember what changed (and you can't, because ADHD), the audit log tells you.
- **Version checking** — Optimistic concurrency means no silent overwrites. If you and an LLM edit at the same time, the conflict is caught, not lost.
- **Search** — Full-text search across all projects. "What did I decide about the camera node?" → search → answer.
- **Snapshots** — Automatic point-in-time backups before every edit. Roll back if something goes wrong.
## The Build
I built CTXD over a series of focused sessions with Hermes Agent — itself powered by the same MCP infrastructure. Some of the tools that made this possible:
- **Hermes Agent** for orchestration and the MCP framework
- **PostgreSQL 16** as the primary database (migrated from SQLite)
- **MCP SDK 1.28** with Streamable HTTP transport (replaced the older SSE pattern)
- **OAuth 2.0** with DCR, PKCE, and per-scope access control
- **Docker Compose** for deployment — env-driven, no config files needed
- **Traefik** for public TLS termination and routing
The development was iterative and honest. I hit real bugs: the `set -e` in the entrypoint script killed the container because a PostgreSQL schema check returned exit code 1. The `__main__.py` checked for a SQLite file that doesn't exist when using PostgreSQL, causing an infinite re-initialization loop. The FTS5 search query used `?` placeholders instead of `%s` for psycopg. Each one was a "oh, of course" moment — the kind you only get by actually building and deploying.
## What I Learned
**Context sprawl is an accessibility problem.** For neurotypical developers, maintaining 6 config files is annoying. For someone with ADHD, it's a barrier that leads to giving up on context entirely — and then every AI interaction starts from zero.
**The single source of truth pattern isn't just about consistency.** It's about reducing the cognitive load of context management to one action: update `CONTEXT.MD`. That's one decision, one file, one habit — something even an ADHD brain can maintain.
**OAuth + MCP is the right protocol stack for this.** LLMs don't need API keys in environment variables. They need to discover a resource, authenticate via OAuth, and use standardized tools. The fact that Claude Desktop can connect to CTXD with zero manual configuration — just a URL — is the difference between "I should set this up" and "I'll just re-explain the project again."
**PostgreSQL matters more than you think.** I started with SQLite, which was fine for one user. But the moment you want concurrent agents writing context, proper backups, and full-text search that actually works — PostgreSQL is the right tool. The migration was a real engineering task (dual-backend support, schema rewrite, FTS5 → tsvector), and it was worth every hour.
## Where It's Going
CTXD is now running in production at `ctxd.cubecraftcreations.com`, serving context to Claude Desktop, Hermes, and any OAuth-capable MCP client. It backs:
- RemoteRig (Go, ESP32-C6)
- ExtrudeX (3D printing control system)
- Tracehound (network diagnostic tool)
- Control Center (homelab dashboard)
The code is open and documented. There's a static demo at `ctxd-demo` that shows the UI without needing a database. There's a full README, an LLM.txt for automated deployment, and a SKILL.md that LLMs can read to understand how to use the server.
## The Takeaway
If you work across multiple AI tools and find yourself re-explaining your project every session — you have context sprawl. If you have ADHD, context sprawl isn't just inefficient; it's a wall between you and productive AI-assisted work.
Building CTXD was my answer. It turned 6 files into 1, turned "remember what I told Claude" into "Claude reads the context server," and turned "I'll update the context later" into "I update one file and every tool gets it."
One source of truth. One habit. Zero context sprawl.
---
*CTXD is open source. The production repo is at [CubeCraft-Creations/CTXD](https://code.cubecraftlabs.com/CubeCraft-Creations/CTXD), with a static demo at [CTXD-Demo](https://code.cubecraftlabs.com/CubeCraft-Creations/CTXD-Demo). Built by Joshua at CubeCraft Creations.*