Fold names by Unicode decomposition in Normalize

Replace the hand-written accent map with letterFolds, a table generated offline by gen_folds.py from Python's unicodedata: every code point whose canonical decomposition contains combining marks maps to its base letters, plus stroked Latin letters and ligatures. Combining marks are still dropped, so composed and decomposed spellings ("Phở" and "Phở") now normalize equal. No new dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:50:42 -04:00
parent 1e0b1fc75e
commit 7729488cde
5 changed files with 1314 additions and 21 deletions
@@ -165,8 +165,10 @@ automatic** (D-021).
  character budget trims the oldest plans first, then the oldest passes;
  Regulars and the Quest log are never trimmed.
 - **Name dedupe** (`harness.Normalize`, `harness.Similar`, `harness.Deduper`):
-  a dinner name is normalized by lowercasing, folding accents, stripping
-  punctuation, dropping filler words (easy, best, homemade, quick, simple,
+  a dinner name is normalized by lowercasing, dropping combining marks and
+  folding precomposed letters to their base letters through a table generated
+  from Unicode data (`gen_folds.py`), so composed and decomposed spellings
+  match, stripping punctuation, dropping filler words (easy, best, homemade, quick, simple,
  the, a, recipe, classic) and collapsing whitespace. Two names are the same
  dinner when the token-set (Jaccard) similarity of their words is at least
  0.85. A `Deduper` over the Regulars, Quest log, pending suggestions and