Reduce letter folds to a fixed point in gen_folds.py
A fold result could itself be foldable: ǿ folded to ø and ǽ to æ, while the canonically equivalent ø+U+0301 and æ+U+0301 reached o and ae. Normalize writes a result without folding it again, so the generator now folds every result letter by letter until it settles (and fails if it does not), making the committed table fully reduced. TestLetterFoldsFullyReduced pins that, and the canonical-equivalence tests now cover ǿ and ǽ through Normalize, Similar and Deduper.IsDuplicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,11 @@ The table maps a code point to its lowercased base letters:
|
||||
WITH ..." map to x (ø -> o, ł -> l, đ -> d);
|
||||
- the Latin ligatures and letters listed in EXTRA map as given.
|
||||
|
||||
Each result is then folded again, letter by letter, until nothing changes
|
||||
(ǿ -> ø -> o, ǽ -> æ -> ae). Normalize writes a result without folding it
|
||||
again, so the table must already be fully reduced; the generator fails if a
|
||||
result does not settle.
|
||||
|
||||
Hangul syllables are left out: their decomposition is into conjoining jamo,
|
||||
not combining marks.
|
||||
"""
|
||||
@@ -67,12 +72,23 @@ def fold(cp):
|
||||
return None
|
||||
|
||||
|
||||
def reduce(folds, s):
|
||||
"""Folds every letter of s until nothing changes."""
|
||||
for _ in range(len(folds)):
|
||||
nxt = "".join(folds.get(c, c) for c in s)
|
||||
if nxt == s:
|
||||
return s
|
||||
s = nxt
|
||||
sys.exit("gen_folds.py: fold of %r does not settle" % s)
|
||||
|
||||
|
||||
def main():
|
||||
entries = []
|
||||
folds = {}
|
||||
for cp in range(sys.maxunicode + 1):
|
||||
folded = fold(cp)
|
||||
if folded is not None:
|
||||
entries.append((cp, folded))
|
||||
folds[chr(cp)] = folded
|
||||
entries = [(ord(ch), reduce(folds, folded)) for ch, folded in folds.items()]
|
||||
out = [
|
||||
"// Code generated by gen_folds.py from Unicode "
|
||||
+ unicodedata.unidata_version + "; DO NOT EDIT.",
|
||||
|
||||
Reference in New Issue
Block a user