The Hungarian Hunspell dictionary declares SET UTF-8 but writes its flags as single bytes, uses AF aliases and 158 distinct flags. Read flag fields byte by byte unless FLAG UTF-8 is set, expand AF aliases, skip AM, apply ICONV and IGNORE, honour KEEPCASE, and widen flag masks to 256 bits. Compounding and NEEDAFFIX stay unsupported for now.
Wall-clock bounds of 1-2 s failed on loaded runners. Each bound is now 10 s, at least 10x the
fixed code's time and well under the old code's on the same input (measured by running the
pre-fix code):
- self-continuing rules: 40,000 rules (was 16,000); ~1 ms now, old checker 14 s at 16,000.
- many strips per suffix: 200,000 rules (was 30,000); 0.4 s now, old index 3 s at 30,000 and
11 min at 300,000.
- forms of self-continuing rules: the MAX_FORMS count catches the old code (9 million forms);
the time bound is a backstop.
- overlong Turkish words: ~20 ms now; without the caps it ran past 5 minutes.
- capped Hunspell reads: no clock; the "not a regular file" error is the proof (the old reader
never returned from /dev/zero).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CX1hMKX28764Lrx5moxs1o
Asked to confirm a non-UTF-8 Turkish dictionary still loads: before this PR
tr_TR was read with read_to_string, so a non-UTF-8 file never loaded; now it
goes through read_capped and decode, which didn't know ISO 8859-9 (Latin-5,
the encoding of older tr_TR dictionaries) and refused it as unsupported.
decode now reads `SET ISO8859-9`, and a test writes an ISO 8859-9 .aff/.dic
to disk and finds words with ğ ı ş İ ç ö ü, as stems and with suffixes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CX1hMKX28764Lrx5moxs1o
Audit finding 4, and the rest of #3661.
- dict.rs: `Dictionary::from_bytes` inflates with `decompress_to_vec_with_limit` (64 MiB; the
bundled en-US one is ~1 MB) and caps the words it keeps (bytes and count): front coding lets a
2-byte line spell out a 255-byte word, so the inflated size alone doesn't bound storage.
- hunspell.rs: one `read_capped` reader — regular files only, size checked on the open handle and
the read itself bounded by `take(MAX_FILE + 1)`, so no TOCTOU between metadata and read and no
endless read from a FIFO or device.
- tr: the Turkish loader reads through `read_capped` and decodes with `hunspell::decode` (it
read both files whole, with no cap at all).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Proofing: say when a dictionary is installed but unreadable, with file and reason (#2830)
* proof test: use SHIFT_JIS as the unsupported encoding (ISO 8859-1 is decoded since #2904)
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Indonesian spelling reads the installed Hunspell id_ID dictionary at runtime
(Debian/Ubuntu hunspell-id, LibreOffice dict-id, the Dictionaries folder or
WORDCRAFT_DICTIONARIES), like Czech, Polish and Portuguese: nothing is bundled,
and without a dictionary Indonesian text gets no spelling marks.
- ProofLang::Id for id, id-ID, ind and the legacy in tag
- suggestions reuse the dictionary-driven Czech ranking (MAP/REP/TRY/KEY of
the loaded dictionary), nothing Indonesian hard-coded
- hyphenated reduplication accepted whole or by parts (kupu-kupu, anak-anak);
hyphenated clitics after capitals/abbreviations (KTP-nya, se-Indonesia)
- generic grammar only: repeated word (offers anak-anak first), space before
punctuation, double space, with Indonesian messages
- Review > Language, status bar, Set Proofing Language, File > Options >
Proofing list Indonesian; name translated in every catalog
- tests use a hand-written synthetic dictionary; engine end-to-end test
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Same pattern as Portuguese (#797) and Polish (#346): the cs_CZ
dictionary is read from the system at runtime (e.g. Debian/Ubuntu
hunspell-cs), nothing is bundled. Suggestions use the dictionary's own
MAP/REP/TRY/KEY; grammar stays generic (repeated word, spacing), with
digit ratios like "3 : 1" exempt. The Czech interface now writes new
documents (and opened .txt/.md) in cs-CZ.
Engine fixes the Czech dictionary needs, also used by pl/pt:
- PFX/SFX header lines may carry a trailing comment (cs_CZ has
"SFX y Y 80 #…"); rule lines are left literal;
- NOSUGGEST stems are accepted when typed but never offered;
- KEY is parsed for neighbour-key suggestions.
Tests use a hand-written synthetic Czech dictionary only; an
end-to-end engine test mirrors the Portuguese one.
Co-authored-by: Losminek <301412321+Losminek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
* Russian proofing: spelling, suggestions, grammar and word division
Runs tagged ru get Russian tools instead of none:
- spelling against an installed ru_RU Hunspell dictionary (not bundled;
LibreOffice dict-ru folders searched; KOI8-R and Windows-1251 decoded),
with ё optional, hyphenated words and solid compounds understood, не with
adjectives, participles and verbal nouns, prefixed verbs, abbreviations
with endings and camel case left alone, plus our supplement of modern
words with their declensions (assets/spelling/ru-extra.txt);
- suggestions tuned for Russian: е/ё, letter groups, common confusions,
missing spaces, typos and a stem scan;
- grammar rules with Russian messages (repeated words, spaces, sentence
capitals with Russian abbreviations, likely missing commas before
clause openers, doubled punctuation);
- Russian word division from the spelling rules, written as code.
* Options: Russian in the dictionaries list (ProofLang::Ru arm)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Магда М.Г. <mail@mgmagda.ru>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* proof: Turkish spelling, suggestions, grammar and hyphenation
* Turkish: dictionary tests skip when no Turkish Hunspell dictionary is installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* M15: Polish interface, and Polish spelling, grammar and hyphenation
Interface
- Polish is the tenth interface language ("Polski", pl.tsv, 1,045 entries);
pl, pl-PL and pl_PL.UTF-8 system locales select it automatically.
- Right-click menu labels now go through translation in every language.
- Review > Language is a menu (English US/UK, Polski, Do not check spelling
or grammar); the status bar shows the language at the caret.
- With a Polish interface, new blank documents and opened .txt/.md files
default to pl-PL.
Proofing
- Each run is checked in its own language (w:lang, resolved through
styles): squiggles, F7, review.issues, suggestions and hyphenation.
Text in languages without proofing tools is no longer flagged as English.
- Polish spelling from the SJP.PL Hunspell dictionary (Apache-2.0 option),
read by a new Rust affix engine (crates/proof/src/affix.rs); 875 KB asset,
~4.5M word forms. Regenerate with the pl_dictionary example.
- Polish suggestions: case, missing diacritics, rz/ż, ó/u, ch/h confusions,
missing spaces (napewno -> na pewno), edit distance.
- Conservative Polish grammar rules: repeated words, spacing, doubled
punctuation, sentence capitals, missing comma before że/który/ponieważ/
aby/żeby/jeśli…, we/ze forms.
- Polish hyphenation with the hyph-utf8 hyph-pl.tex patterns (MIT option);
pattern keys widened to 20 letters, TeX exceptions parsed; the heuristic
no longer treats ł, ż, ś… as vowels.
review.language validates and normalizes tags, can set noProof alone and
the document default; format.state reports lang and noProof.
Tests for every part, docs and ATTRIBUTION.md/NOTICE updated;
cargo xtask ci passes (fmt, clippy, test, assets, layers, wasm).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Layout: rustfmt the test module order left unformatted by #391
cargo fmt --check failed on main after #391 added mod tests_grid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Review fixes for #346: read Polish spelling from the installed Hunspell dictionary
- Polish spelling uses the Hunspell pl_PL dictionary installed on the
system (OS packages, Homebrew, LibreOffice, ~/.local/share/hunspell, or
WORDCRAFT_DICTIONARIES) instead of a bundled 870 KB SJP.PL file;
without one, Polish text gets no spelling marks, like Word without the
language's proofing tools. .aff/.dic are size-capped and decoded from
UTF-8 or ISO 8859-2 (crates/proof/src/hunspell.rs).
- The dictionary can be replaced at runtime (install_dictionary) or
re-searched (reload_dictionary); an empty one accepts every word.
- Tests use small synthetic dictionaries written for them, so they don't
depend on what the machine has installed.
- The bundled pl.dic, its ATTRIBUTION/NOTICE entries and the generator
example are removed; docs describe the installed dictionary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* #100: affix — accept FLAG UTF-8 and DEFAULT flag modes
Per-character flag modes are what the reader already parses; only the
multi-character encodings (long, num) stay errors. VERO pt_BR declares
FLAG UTF-8.
* #100: affix — widen flag masks to u128 (128 flag capacity)
VERO pt_BR needs 106 distinct flags; the u64 bitmask capped at 64.
Serialization is unaffected: WCAFF1 stores flag characters and
re-derives the masks on load.
* #100: affix — support FORBIDDENWORD
VERO pt_BR marks spellings it must reject with a forbidden flag (ý).
Stems carrying it are listed but never correct and never inflect.
* #100: affix — continuation classes in check
SFX A 0 able/B: the form derived by A carries flag B, licensing a
further suffix of class B (workable -> workables) while the bare stem
cannot take one. form_has_flag walks the chain depth-bounded
(MAX_CHAIN = 3); with no continuations declared it degenerates to the
stem lookup, so existing dictionaries are unaffected. VERO pt_BR uses
26 continuation rules.
* #100: affix — continuation classes in forms
Enumerate chained suffix forms (workable -> workables) under the same
MAX_CHAIN bound the checker walks, so every form the checker accepts
is listed and vice versa.
* #100: affix — round-trip continuations and FORBIDDENWORD through WCAFF1
The compact format keeps its WCAFF1 magic: continuation classes ride
on the rule's appended field (as in the .aff source) and the forbidden
flag becomes a FORBIDDENWORD line. Old files read as before.
* #100: affix — strip a UTF-8 BOM from .aff/.dic input
VERO's files open with a BOM; glued to the count line it parsed as a
stem (one garbage entry per load) and to SET it hid the directive.
* #782: affix — the module doc still rejected what the code now reads
Continuation classes and FORBIDDENWORD are supported; the header said
Rejected for both.
* #100: pt — Portuguese spelling, following the Polish module
ProofLang::Pt resolves pt/pt-BR/pt-PT/por tags; the pt module reads the
installed Hunspell pt_BR dictionary (VERO) through the affix engine —
FLAG UTF-8 flags, continuation classes and FORBIDDENWORD spellings all
exercised by the synthetic test dictionary. Hyphenated compounds are
checked part by part (guarda-roupa, ex-marido); without a dictionary
installed, Portuguese text gets no marks. Spelling message in
Portuguese; suggestions and grammar land next.
* #100: pt — Portuguese suggestions
Rounds of rising cost, driven by the dictionary's own tables: missing
accents from MAP (coraçao -> coração, saude -> saúde), the REP
confusions (seje -> seja, excessão -> exceção), a missing space after a
short function word (nãoé -> não é, dagua -> da água), one typo over
TRY, and an accent-folding stem scan — the Polish rounds, retuned for
Portuguese.
* #100: pt — Portuguese grammar flags
Conservative rules with Portuguese messages: repeated words (except
the emphatic doubles Portuguese allows), space before punctuation and
double spaces, sentence capitals knowing the Portuguese abbreviations
and dialogue dashes, and doubled punctuation. No comma rule: que opens
a clause only sometimes, so a wrong flag would cost more than a missed
one.
* #100: pt — Portuguese hyphenation
hyph-utf8's hyph-pt.tex bundled unmodified (BSD 3-clause, ATTRIBUTION
row with source and hash): 426 Liang patterns and 2 exception words;
falls back to the generic heuristic outside their reach.
* #100: pt — Review › Language offers Português (Brasil), and a pt-br interface writes pt-BR
The menu's fourth entry sets the run language; the status bar names
Portuguese; editing_lang is now allowed for pt-br (wordcraft-proof
proofs it), so new documents under a Portuguese interface are
Portuguese by default.
* #100: pt — engine test: Portuguese proofing through the review commands
review.language pt-BR normalizes the tag, review.issues reports
Portuguese spelling and grammar with the lang field, F7 and Change
apply a suggestion, Add to Dictionary works for Portuguese.
* #100: pt — Portuguese in the mixed-language proofing tests and docs
check_text proofs a Portuguese span with the Portuguese message; the
never-panic loops and the module docs cover the third language.
* #100: docs — Portuguese proofing in the gap, parity and roadmap docs
Gap 8 narrowed (English, Polish and Portuguese), localization parity
gains a Portuguese row and a revision entry, target-app-parity's
Proofing row moves 40% -> 42%, and the ROADMAP progress log records
what landed.
* #100: hunspell — LibreOffice extension folders follow the language
The search paths hardcoded LibreOffice's dict-pl; a pt_BR user whose
only dictionary is LibreOffice's dict-pt-BR extension — on Windows the
only realistic on-disk source — got silently no spelling. Both the
primary-language folder and the dashed full name are now tried, so
Polish keeps dict-pl and Portuguese finds dict-pt-BR.
* #100: pt — review fixes
Suggestion ties break toward the change nearest the word's start
(saúde before saudê, proven against a fixture that has both); the
Language menu's label is the English source "Portuguese (Brazil)" with
translations in all twelve catalogs (as Polish is); the attribution
counts pt.pat's 2 exception words; the pt fixture drops a duplicate
stem; docs/roadmap.md catches up with the landing; the throwaway spike
is gone.
* Review fixes for #782: two-suffix chain limit, module doc
- MAX_CHAIN = 2 counts suffixes in a row, matching Hunspell's two-suffix limit
(the old bound of 3 allowed up to four); test pins that a third is rejected.
- Module doc: say probes allocate small scratch strings instead of claiming none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Review fixes for #797: NOTICE for pt.pat, no indexing in the LibreOffice folder helper
- NOTICE reproduces hyph-pt.tex's BSD 3-Clause copyright notice, conditions and
disclaimer, which the licence requires for binary redistributions.
- hunspell::lo_ext_dirs builds its list without indexing (never-crash rule).
- Catalogs: main already translates "Portuguese (Brazil)" in every language, so the
duplicate rows the branch added are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: witkowski01 <konradwitkowski01@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
* M15: Polish interface, and Polish spelling, grammar and hyphenation
Interface
- Polish is the tenth interface language ("Polski", pl.tsv, 1,045 entries);
pl, pl-PL and pl_PL.UTF-8 system locales select it automatically.
- Right-click menu labels now go through translation in every language.
- Review > Language is a menu (English US/UK, Polski, Do not check spelling
or grammar); the status bar shows the language at the caret.
- With a Polish interface, new blank documents and opened .txt/.md files
default to pl-PL.
Proofing
- Each run is checked in its own language (w:lang, resolved through
styles): squiggles, F7, review.issues, suggestions and hyphenation.
Text in languages without proofing tools is no longer flagged as English.
- Polish spelling from the SJP.PL Hunspell dictionary (Apache-2.0 option),
read by a new Rust affix engine (crates/proof/src/affix.rs); 875 KB asset,
~4.5M word forms. Regenerate with the pl_dictionary example.
- Polish suggestions: case, missing diacritics, rz/ż, ó/u, ch/h confusions,
missing spaces (napewno -> na pewno), edit distance.
- Conservative Polish grammar rules: repeated words, spacing, doubled
punctuation, sentence capitals, missing comma before że/który/ponieważ/
aby/żeby/jeśli…, we/ze forms.
- Polish hyphenation with the hyph-utf8 hyph-pl.tex patterns (MIT option);
pattern keys widened to 20 letters, TeX exceptions parsed; the heuristic
no longer treats ł, ż, ś… as vowels.
review.language validates and normalizes tags, can set noProof alone and
the document default; format.state reports lang and noProof.
Tests for every part, docs and ATTRIBUTION.md/NOTICE updated;
cargo xtask ci passes (fmt, clippy, test, assets, layers, wasm).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Layout: rustfmt the test module order left unformatted by #391
cargo fmt --check failed on main after #391 added mod tests_grid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Review fixes for #346: read Polish spelling from the installed Hunspell dictionary
- Polish spelling uses the Hunspell pl_PL dictionary installed on the
system (OS packages, Homebrew, LibreOffice, ~/.local/share/hunspell, or
WORDCRAFT_DICTIONARIES) instead of a bundled 870 KB SJP.PL file;
without one, Polish text gets no spelling marks, like Word without the
language's proofing tools. .aff/.dic are size-capped and decoded from
UTF-8 or ISO 8859-2 (crates/proof/src/hunspell.rs).
- The dictionary can be replaced at runtime (install_dictionary) or
re-searched (reload_dictionary); an empty one accepts every word.
- Tests use small synthetic dictionaries written for them, so they don't
depend on what the machine has installed.
- The bundled pl.dic, its ATTRIBUTION/NOTICE entries and the generator
example are removed; docs describe the installed dictionary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* M15: Polish catalog covers the interface text merged from main
The merge of main (98456ac) brought ~575 new interface strings (charts,
SmartArt, content controls, Developer tab, AutoCorrect, outline view,
protection, compare, print, accessibility, references and Language/
Hyphenation dialogs, Draw additions). pl.tsv lacked them, so
polish_covers_every_key_of_every_other_catalog and
every_tab_and_command_is_translated failed. All are now translated.
cargo xtask ci: all 6 steps pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Skaywalker