M2.5: dictionary hyphenation, letter/glyph justification, keeps

Browse Source

- hyphen: bundled en-US dictionary from the public-domain Moby Hyphenator II
  list (Grady Ward, Gutenberg #3204), front-coded + deflated (433 KB), with
  case/apostrophe/punctuation/compound handling and inflection inheritance;
  our own Liang patterns (19.5k, 129 KB) trained by a patgen-style trainer
  (hyphen::patgen, examples/hyphgen.rs) for unknown words; heuristic last.
  Discretionary hyphens replace automatic points.
- breaker: prioritised word/letter/glyph-scaling glue in Knuth-Plass and
  layout; uncapped badness; hyphen limit as a hard constraint (hyphen-count
  states); hyphenation zone for ragged text (both composers); ragged
  hyphenation measured correctly; optical margin alignment; balance ragged
  lines; Knuth's node pruning.
- compose: Keep Options (keep with next, keep all lines, keep first/last)
  via paragraph snapshots; desired spacing applied to all alignments;
  justified last lines shrink instead of overflowing; word shaping and
  hyphenation caches (300k-char story: ~40 ms warm, ~60 ms cold).
- tests: golden corpus, perf (ignored), keeps, balance, optical, zone,
  limit, letter/glyph scaling, discretionary hyphens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

...

This commit is contained in:

Brandon Thomas

2026-10-01 12:04:43 -04:00

co-authored by Claude Opus 5.5

parent 9fdd61a390

commit 83915e2fc2

17 changed files with 4032 additions and 327 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

assets/hyphenation/en-us.dic

BIN

Binary file not shown.

assets/hyphenation/en-us.pat

+1633

File diff suppressed because it is too large Load Diff