Name normalization and dedupe #59

Member

Maestro task: f046ec6c-dbe2-4b12-bc95-3081c07714eb (Feature 09 - AI Harness and Skills Framework).

Done-condition

harness.Normalize lowercases, strips punctuation and accents, removes filler words (easy, best, homemade, quick, simple, the, a, recipe, classic) and collapses whitespace; harness.Similar(a,b) uses token-set similarity with a 0.85 threshold; a Deduper built from Regulars, Quest log, pending suggestions and declined names answers IsDuplicate(name). Table tests include the five named pairs (incl. the negative) and accented names. Go quality floor passes.

Evidence

  • New internal/ai/harness/dedupe.go + dedupe_test.go; docs/ARCHITECTURE.md updated.
  • gofmt/vet/go test -race/build pass; no new dependency.

Out of scope, noted: dinners.normalized_name stays populated by existing callers, not harness.Normalize.

Maestro task: f046ec6c-dbe2-4b12-bc95-3081c07714eb (Feature 09 - AI Harness and Skills Framework). ## Done-condition harness.Normalize lowercases, strips punctuation and accents, removes filler words (easy, best, homemade, quick, simple, the, a, recipe, classic) and collapses whitespace; harness.Similar(a,b) uses token-set similarity with a 0.85 threshold; a Deduper built from Regulars, Quest log, pending suggestions and declined names answers IsDuplicate(name). Table tests include the five named pairs (incl. the negative) and accented names. Go quality floor passes. ## Evidence - New internal/ai/harness/dedupe.go + dedupe_test.go; docs/ARCHITECTURE.md updated. - gofmt/vet/go test -race/build pass; no new dependency. Out of scope, noted: dinners.normalized_name stays populated by existing callers, not harness.Normalize.
dev_agent added 1 commit 2026-10-05 08:40:04 -04:00
Name normalization and dedupe
CI / ci (push) Successful in 1m32s
CI / ci (pull_request) Successful in 1m33s
1e0b1fc75e
Add harness.Normalize (lowercase, fold accents, strip punctuation, drop
filler words, collapse whitespace), harness.Similar (token-set Jaccard
similarity, threshold 0.85) and a Deduper built from the Regulars, Quest
log, pending suggestions and passed names that answers IsDuplicate.

Maestro task: f046ec6c-dbe2-4b12-bc95-3081c07714eb

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review_agent requested changes 2026-10-05 08:42:26 -04:00
Dismissed
review_agent left a comment
Member

Independent Codex gpt-6-sol review: CHANGES_REQUIRED for this exact commit.

Independent Codex gpt-6-sol review: CHANGES_REQUIRED for this exact commit.
overseer added 1 commit 2026-10-05 08:52:45 -04:00
Fold names by Unicode decomposition in Normalize
CI / ci (push) Successful in 2m4s
CI / ci (pull_request) Successful in 1m32s
7729488cde
Replace the hand-written accent map with letterFolds, a table generated
offline by gen_folds.py from Python's unicodedata: every code point whose
canonical decomposition contains combining marks maps to its base letters,
plus stroked Latin letters and ligatures. Combining marks are still dropped,
so composed and decomposed spellings ("Phở" and "Phở") now
normalize equal. No new dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review_agent requested changes 2026-10-05 08:57:43 -04:00
Dismissed
review_agent left a comment
Member

Independent Codex gpt-6-sol review: CHANGES_REQUIRED for this exact commit.

Independent Codex gpt-6-sol review: CHANGES_REQUIRED for this exact commit.
overseer added 1 commit 2026-10-05 09:11:05 -04:00
Reduce letter folds to a fixed point in gen_folds.py
CI / ci (push) Successful in 1m32s
CI / ci (pull_request) Successful in 1m33s
d2d5e9c742
A fold result could itself be foldable: ǿ folded to ø and ǽ to æ, while
the canonically equivalent ø+U+0301 and æ+U+0301 reached o and ae.
Normalize writes a result without folding it again, so the generator now
folds every result letter by letter until it settles (and fails if it does
not), making the committed table fully reduced. TestLetterFoldsFullyReduced
pins that, and the canonical-equivalence tests now cover ǿ and ǽ through
Normalize, Similar and Deduper.IsDuplicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review_agent left a comment
Member

Independent Codex gpt-6-sol review: APPROVED for this exact commit.

Independent Codex gpt-6-sol review: APPROVED for this exact commit.
approver_agent merged commit 5d5404b3cb into feature/d4a8ed83-9cee-459c-aca3-efd1ccf17ecd-ai-harness 2026-10-05 09:16:17 -04:00
Sign in to join this conversation.
No Reviewers
No labels
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Maniacs_Mansion/MealQuest#59