Skill harness interface and repair retry #53

2026-10-04T17:11:28-04:00

dev_agent commented

2026-10-04 17:11:28 -04:00

Task 38e7bf98-f1d1-4301-b73b-7ec4b51e8b68. internal/ai/harness Skill interface and Run pipeline with exactly one repair turn, ErrBadOutput after a second failure, provider errors passed through, and a trivial test skill. Head 9ed5cb1192.

Task 38e7bf98-f1d1-4301-b73b-7ec4b51e8b68. internal/ai/harness Skill interface and Run pipeline with exactly one repair turn, ErrBadOutput after a second failure, provider errors passed through, and a trivial test skill. Head 9ed5cb119232bf67c07820454942ab9a8d5e2fd7.

dev_agent added 1 commit 2026-10-04 17:11:29 -04:00

Add skill harness with one repair retry

CI / ci (push) Successful in 1m31s

Details

CI / ci (pull_request) Successful in 1m30s

Details

Independent review Changes required — 2 medium finding(s) (#53)

Details

9ed5cb1192

internal/ai/harness defines the generic Skill interface (Name, Version,
Render, Schema, Validate, PostProcess) and Run, which renders the prompt,
calls Provider.Generate with the skill's schema and the household
temperature, strictly decodes and validates the reply, and on failure sends
exactly one repair turn quoting the problems. A second failure returns a
*BadOutputError matching llm.ErrBadOutput with both problem lists. Provider
errors other than ErrBadOutput pass through unchanged.

Maestro task 38e7bf98-f1d1-4301-b73b-7ec4b51e8b68.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review_agent reviewed

2026-10-05 07:48:28 -04:00

review_agent left a comment

Independent review — Codex `gpt-6-sol`, reasoning high

Verdict: CHANGES_REQUIRED
Reviewed SHA: 9ed5cb119232bf67c07820454942ab9a8d5e2fd7

{
  "verdict": "CHANGES_REQUIRED",
  "reviewed_sha": "9ed5cb119232bf67c07820454942ab9a8d5e2fd7",
  "summary": "The Go quality checks pass. The harness makes at most one repair attempt, passes other provider errors through unchanged, and correctly treats an adapter-returned ErrBadOutput as repairable because it denotes an unusable model reply. No new dependency or production logging was added. Two output integrity cases need correction.",
  "findings": [
    {
      "id": "F1",
      "severity": "medium",
      "location": "internal/ai/harness/harness.go:116",
      "failure_case": "A successful run can return output that fails the skill's validation.",
      "evidence": "The harness validates before PostProcess and returns its result without validating again. The included echo skill accepts an echo containing only spaces, then trims it to an empty string.",
      "suggested_correction": "Ensure the final processed output is validated, and include any resulting validation failure in the one-repair flow. Add a whitespace-only test."
    },
    {
      "id": "F2",
      "severity": "medium",
      "location": "internal/ai/harness/harness.go:128",
      "failure_case": "When an Anthropic reply contains both text and invalid tool JSON, the repair turn presents the text as the previous assistant reply while omitting the JSON that failed validation.",
      "evidence": "The Anthropic adapter can populate both Response.Text and Response.JSON; attempt prefers Text whenever it is nonempty.",
      "suggested_correction": "Prefer Response.JSON as the previous assistant content when structured output was returned, and test a response containing both fields."
    }
  ]
}

Posted unaltered by review_agent. This is transport, not a second review.

## Independent review — Codex `gpt-6-sol`, reasoning high Verdict: **CHANGES_REQUIRED** Reviewed SHA: `9ed5cb119232bf67c07820454942ab9a8d5e2fd7` ```json { "verdict": "CHANGES_REQUIRED", "reviewed_sha": "9ed5cb119232bf67c07820454942ab9a8d5e2fd7", "summary": "The Go quality checks pass. The harness makes at most one repair attempt, passes other provider errors through unchanged, and correctly treats an adapter-returned ErrBadOutput as repairable because it denotes an unusable model reply. No new dependency or production logging was added. Two output integrity cases need correction.", "findings": [ { "id": "F1", "severity": "medium", "location": "internal/ai/harness/harness.go:116", "failure_case": "A successful run can return output that fails the skill's validation.", "evidence": "The harness validates before PostProcess and returns its result without validating again. The included echo skill accepts an echo containing only spaces, then trims it to an empty string.", "suggested_correction": "Ensure the final processed output is validated, and include any resulting validation failure in the one-repair flow. Add a whitespace-only test." }, { "id": "F2", "severity": "medium", "location": "internal/ai/harness/harness.go:128", "failure_case": "When an Anthropic reply contains both text and invalid tool JSON, the repair turn presents the text as the previous assistant reply while omitting the JSON that failed validation.", "evidence": "The Anthropic adapter can populate both Response.Text and Response.JSON; attempt prefers Text whenever it is nonempty.", "suggested_correction": "Prefer Response.JSON as the previous assistant content when structured output was returned, and test a response containing both fields." } ] } ``` --- Posted unaltered by `review_agent`. This is transport, not a second review.

overseer added 1 commit 2026-10-05 07:50:23 -04:00

Validate post-processed skill output and quote failed JSON in repair

CI / ci (push) Successful in 1m33s

Details

CI / ci (pull_request) Successful in 1m32s

Details

7be4879f6d

The harness now validates a skill's output again after PostProcess, so
output that post-processing makes invalid (e.g. a whitespace-only echo
trimmed to empty) enters the single repair turn instead of returning as
a success. The repair turn now quotes Response.JSON as the previous
assistant reply whenever structured output was returned, falling back to
Response.Text only when there is none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review_agent approved these changes

2026-10-05 07:52:21 -04:00

review_agent left a comment

Independent Codex gpt-6-sol review: APPROVED for this exact commit.

approver_agent merged commit 55c15b1b3a into feature/d4a8ed83-9cee-459c-aca3-efd1ccf17ecd-ai-harness

2026-10-05 07:55:34 -04:00

dev_agent referenced this pull request

2026-10-05 10:43:06 -04:00

Feature 09: AI harness and skills framework #62

dev_agent referenced this pull request

2026-10-05 11:20:53 -04:00

Promote dev to main: Features 08-09 #63