ai_runs logging, redaction and retention #60

2026-10-05T09:35:18-04:00

dev_agent commented

2026-10-05 09:35:18 -04:00

Maestro task: 53e382d8-1db3-4555-a31d-e44cf55cc98d (Feature 09 - AI Harness and Skills Framework).

Done-condition

Every harness Run (including the repair turn) writes an ai_runs row: skill, skill_version, provider, model, temperature, status (ok|repaired|bad_output|provider_error), latency_ms, tokens in/out when reported, error text, and redacted request/response bodies. Redaction removes the decrypted API key, Authorization/x-api-key values, Google Chat webhook URLs and URLs with key-like query parameters. Purge(olderThan) deletes rows older than 30 days. Tests assert one row per run with correct status, a seeded key and webhook URL never appear in stored rows, and Purge removes only old rows. Go quality floor passes.

Evidence

New internal/ai/harness/redact.go + runs.go + runs_test.go; migration 0006_ai_runs_status.sql and migrations/queries/ai_runs.sql with regenerated internal/db/ai_runs.sql.go; harness.go writes a row per run.
gofmt/vet/go test -race/build pass; make sqlc-check no drift; no new dependency.
Purge is not scheduled yet (the scheduler is Feature 20).

Maestro task: 53e382d8-1db3-4555-a31d-e44cf55cc98d (Feature 09 - AI Harness and Skills Framework). ## Done-condition Every harness Run (including the repair turn) writes an ai_runs row: skill, skill_version, provider, model, temperature, status (ok|repaired|bad_output|provider_error), latency_ms, tokens in/out when reported, error text, and redacted request/response bodies. Redaction removes the decrypted API key, Authorization/x-api-key values, Google Chat webhook URLs and URLs with key-like query parameters. Purge(olderThan) deletes rows older than 30 days. Tests assert one row per run with correct status, a seeded key and webhook URL never appear in stored rows, and Purge removes only old rows. Go quality floor passes. ## Evidence - New internal/ai/harness/redact.go + runs.go + runs_test.go; migration 0006_ai_runs_status.sql and migrations/queries/ai_runs.sql with regenerated internal/db/ai_runs.sql.go; harness.go writes a row per run. - gofmt/vet/go test -race/build pass; make sqlc-check no drift; no new dependency. - Purge is not scheduled yet (the scheduler is Feature 20).

dev_agent added 1 commit 2026-10-05 09:35:19 -04:00

Record every harness run in ai_runs with redaction and purge

CI / ci (push) Successful in 1m35s

Details

CI / ci (pull_request) Successful in 1m32s

Details

b0cf6fb951

harness.Run now writes one ai_runs row per run that reaches the provider,
covering the repair turn: skill, version, provider, model, temperature,
status (ok, repaired, bad_output, provider_error), latency, summed token
counts when reported, error text and the per-turn request and response
bodies. Every stored string is redacted first: the decrypted API key,
Authorization and API-key header values, Google Chat webhook URLs and
URLs with key-like query or fragment parameters or user info. JSON is
decoded and its strings redacted, so escapes cannot hide a credential.
A row that cannot be written fails the run.

Purge deletes rows created before a cutoff (RunRetention is 30 days),
clearing suggestion_batches and week_plans references in the same
transaction.

Migration 0006 rebuilds ai_runs with the new status CHECK and a
created_at index; a pre-harness running or error row stops it.

Maestro task 53e382d8-1db3-4555-a31d-e44cf55cc98d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review_agent requested changes

2026-10-05 09:39:12 -04:00

Dismissed

review_agent left a comment

Independent Codex gpt-6-sol review: CHANGES_REQUIRED for this exact commit.

overseer added 1 commit 2026-10-05 09:45:49 -04:00

Build the run's provider from the key it redacts; record render failures

CI / ci (pull_request) Successful in 1m33s

Details

CI / ci (push) Successful in 1m33s

Details

6a7c040657

Review findings on b0cf6fb:

1. Config took a ready-made Provider and a separate, optional APIKey used
   only for redaction, so a provider holding a decrypted key could run
   with APIKey empty and a reply quoting that key unlabelled was stored.
   Config now holds the household's llm.Settings, the decrypted APIKey and
   a NewProvider constructor (presets.FromSettings in production). Run
   builds the provider from exactly that key and redacts the same key, so
   the two cannot differ. ProviderName, Model and Temperature come from
   Settings. A new test runs presets.FromSettings against a server that
   echoes the bearer key unlabelled in the reply text, reply JSON and
   model name, across a repair turn, and asserts the stored row never
   holds it; with redaction disabled the test fails.

2. A render failure wrote no ai_runs row. Every run with a valid Config
   now writes one: a render failure is bad_output and a provider that
   cannot be built is provider_error, each with the error text and empty
   request and response lists. TestRunRenderError now expects exactly one
   such row. A Config error (no store, provider name or model) still
   writes none, since the row cannot be written without them.

Redaction rules, the four statuses for provider-reaching runs (one row,
both turns) and Purge are unchanged.

Maestro task 53e382d8-1db3-4555-a31d-e44cf55cc98d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review_agent approved these changes

2026-10-05 09:50:55 -04:00

review_agent left a comment

Independent Codex gpt-6-sol review: APPROVED for this exact commit.

approver_agent merged commit 87316d8b70 into feature/d4a8ed83-9cee-459c-aca3-efd1ccf17ecd-ai-harness

2026-10-05 09:55:06 -04:00

dev_agent referenced this pull request

2026-10-05 10:43:06 -04:00

Feature 09: AI harness and skills framework #62

dev_agent referenced this pull request

2026-10-05 11:20:53 -04:00

Promote dev to main: Features 08-09 #63