mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-24 10:08:00 -06:00
main
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tanish MankuandClaude Opus 5
|
99941daa5a |
docs(roadmap): record the day's two changes
`CLAUDE.md` asks for a progress-log line and a status line at the end of a session. - The Annex D.2 alias codes, in both tables (`13e3154a`). - Reading order in the text oracle (`50cbf5fa`), with gap 12's revision history. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
50cbf5fa19 |
feat(oracle): score reading order, which word F1 cannot see
`cargo xtask text-oracle` reported one number: word F1 against `pdftotext`. F1 is a bag of words. It catches a character read wrong, a glyph not read at all, a font we cannot map -- and it is blind to order, so a page emitted back to front scores a perfect 1.000. Reading order is most of what text extraction has left to get wrong: a column read into the middle of its neighbour (#2387), a figure's labels threaded through a paragraph (#2586), a form's value parted from its label. The hole is not hypothetical. Reviewing #2431, a change that moves text and alters nothing else, **word F1 called 26 of the 29 files whose output changed identical**, while scoring the word sequence separated 18 improvements from 6 regressions and named the shape of the regressions (forms: `prefilled_f1040.pdf` 0.672 -> 0.481, a checkbox's number parted from its label). The harness could not see the only thing that change does. So a second score per file: the longest common subsequence of the two word sequences, as a share of the longer one. It shows up in the first three corpus files -- `Test-plusminus.pdf` scores F1 1.000 and reading order 0.875, the same forty words in a title block that we read down the column and `pdftotext` reads across. Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages is plenty of signal; the run says how many files that bit, so the cap cannot quietly become the measurement. It reports; it does not gate, and like the pixel score it is a *disagreement* with Poppler rather than a verdict -- in the #2431 measurement the same split scored as a win on one file and a loss on another purely because `pdftotext` makes it in one place and not the other. The function's doc comment, `crates/oracle/README.md` and `docs/gaps.md` gap 12 all say so. `words` now takes the sequence rather than re-splitting the text, so both scores read the same tokens. `docs/file-format-parity.md` claimed plain-text export was good "(word-F1 0.98)" in a row whose own column is reading order; it now says the two are scored separately. Verified: `cargo xtask text-oracle --limit 60` end to end; `cargo test -p xtask` 49 passed; fmt and clippy clean. The unit test was watched failing with the score blinded to order (sorting both sides made it 1.000 where it must be below 0.7). The corpus-wide first reading is not in this commit: the machine is shared with a parallel build and the full sweep was stopped twice for memory. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
13e3154ab3 |
fix(fonts): read the Annex D.2 alias codes as the ink they draw
#2305 fixed WinAnsi 0xAD in `crates/fonts`, and closed with two loose ends it named: the user-visible extraction path goes through hayro, not that table, so it still reported the invisible character; and the same Annex D.2 note makes 0xA0 the `space` glyph. This finishes both, and makes the two tables in the product agree about the same byte. WinAnsiEncoding gives codes 0xA0 and 0xAD the glyph names `space` and `hyphen` -- the names it also gives 0x20 and 0x2D -- and MacRomanEncoding gives 0xCA `space`. The tables called them `nbspace` and `sfthyphen`, whose Adobe Glyph List values U+00A0 and U+00AD are invisible formatting characters, so extraction hid ink the reader can see: "well-known" drawn with the 0xAD byte matched no search and no redaction pattern, and a non-breaking space did not match the space a reader typed. Evidence. Two sources transcribe the table the same way (an Annex D implementation's docs, and the WinAnsi.enc encoding vector R ships, which gives `/space` at octal 240 and `/hyphen` at 255), and a black-box check settles the rest: on a synthetic fixture pdftotext reads the base-encoded codes as `a-b a b`, and keeps U+00AD for a code an explicit `/Differences` names `sfthyphen`. We now do both. Over the 989-file pdf.js corpus the soft hyphens we extract fall from 117 to 13 and the non-breaking spaces from 30 to 26; everything left comes from `/ToUnicode` maps, which say what they mean and are not touched. Scope. The base table only, in the text direction only. - A `/Differences` entry naming `sfthyphen` or `nbspace` is a deliberate statement about one code and keeps its glyph list value, as pdftotext does. - Glyph lookup is untouched, so a font program's own `sfthyphen` program is still found first. In hayro that is why the alias sits in the Unicode path rather than in the generated encoding table: `code_to_ps_name` is shared with lookup. - Writing is unaffected: `encode` takes the lowest code showing a character, so ' ' and '-' still write 0x20 and 0x2D, and a caller holding a literal U+00A0 still writes 0xA0. Tests. `fonts` pins both tables and the whole decode/encode path including the `/Differences` case; `render` pins the extraction a reader actually gets, over WinAnsi, MacRoman and `/Differences`. Each was watched failing with each cell and each match arm reverted in turn, so no assertion is passing by accident. fonts 70, render 296+4, redact 49, a11y 43 and edit 156 are green, with fmt, clippy, `xtask layers` (37 crates) and `xtask assets` clean. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
41d4c15849 |
fix(fonts): read WinAnsi 0xAD as a hyphen, so search and redaction can see it (#2305)
A page that draws a discretionary hyphen through WinAnsiEncoding's 0xAD extracts, in our map, as U+00AD SOFT HYPHEN. That character is invisible, so text the reader can plainly see does not match a search for it, and `redact`'s pattern matching -- which runs on exactly this map -- does not match it either. The same silent-failure shape as reading Symbol as Latin: the text is there, the tool cannot see it. Annex D.2 gives codes 0x2D and 0xAD the same glyph name, so the byte a Windows codepage calls a soft hyphen draws, and should extract as, an ordinary hyphen. Changed in the reading table only: writing is unaffected, because win_ansi_byte maps 0xA0..0xFF by identity, so '-' still takes the canonical 0x2D and a caller holding a literal U+00AD still writes 0xAD. The round-trip test could no longer hold for that code -- two codes now mean one character -- so it states the aliasing instead of asserting a bijection, and a new test pins the behaviour and the reason. **This needs the owner's reading of the spec before merging.** The evidence is not unanimous: a PDF library's documentation cites Annex D.2 for the glyph name `hyphen` at 0xAD, and Poppler extracts U+002D, but the vendored hayro maps 0xAD to the glyph name `sfthyphen` and reports U+00AD, normalising to `hyphen` only when looking the glyph up in the font program. Two implementations disagree and I cannot read the normative table from here. Note this does not change `pdfcraft-cli text`, which maps through hayro, not this table; the user-visible extraction path carries the same question separately. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
ac89a11391 |
fix(xtask): the text oracle scored us zero for honouring a no-copy document
A PDF can forbid copying its text. `pdfcraft-cli text` honours that and refuses, naming the owner password as the way through. pdftotext ignores the flag and extracts anyway, so the oracle was comparing our refusal against its text and scoring 0.000 -- measuring obedience, not quality. bug1815476.pdf is the case: a one-page Spanish form we render perfectly, whose text we decline to extract and pdftotext hands over regardless. It sat at the bottom of the worst list looking like a font bug, which is what sent me looking at it. Those files are now skipped and counted, not scored, and the count is printed rather than passed over in silence, on the same rule the oracle crate exists for: a comparison that did not happen must not look like one that succeeded. Over the first 250 corpus files that moves the mean from 0.811 to 0.854 and drops one file from the sample, with the median unchanged at 0.991. Permissions come from `Document::permissions`, so this is not a match on our own error text. A file that cannot be opened at all counts as permitted, because that is a different problem and dropping it quietly would hide it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
f802257be6 |
docs(fonts): the README still called the standard-14 widths an estimate
F1 replaced the character-class approximation with the published standard-14 metrics, and `helvetica_width` has delegated to `Std14::Helvetica` ever since. The README kept describing it as PdfCraft's own estimate and, more importantly, said "No metrics file or font program from any vendor is bundled" -- which stopped being true when the generated tables landed. They are Adobe-authored non-visual technical data, the item AGENTS.md 1.1 permits, carried in std14_metrics.rs with adobe_data = true in ATTRIBUTION.toml. A contributor reading the README would have concluded there was no vendor data here at all, which is the one kind of error that matters in a provenance claim. Says what is bundled now, and what still is not: widths and encodings, never a font program or a typeface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
e0c63546bb |
test(render): pin that every JBIG2 conformance fixture decodes to the same picture
The render oracle put four JBIG2 fixtures at the top of its disagreement list against Poppler, and the obvious reading was that hayro-jbig2, which carries local patches, had a bug. Investigating it found the reverse. The 96 `bitmap-*.pdf` corpus files are a conformance suite: one small drawing encoded through every corner of JBIG2 -- each generic-region template, custom AT pixels, TPGDON and TPGRON, refinement, halftone with and without skip, symbol dictionaries with Huffman tables, MMR, the composition operators, random access, unknown-height segments. Those features change how the bytes are compressed, not what the picture is, so a correct decoder renders all 96 identically. That is a real test with no reference image and no external oracle: the suite checks itself. We render 1 distinct image across all 96. Poppler renders 8: it draws a blank page for the seven halftone-skip and symbol fixtures, and its output changes when TPGRON is switched on, which a correct decoder's cannot. So roughly a fifth of Poppler's JBIG2 output is wrong, the top of the oracle's worst list is its bug rather than ours, and Poppler is not a usable oracle for JBIG2. Watched failing by admitting one unrelated file to the set: the test fails and names it as the odd one out rather than merely reporting a count. The expectation holds of the pinned corpus. Whoever re-pins it to a newer pdf.js should re-check this rather than assume it still holds, which the test's own documentation says. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
5e08aa8719 |
fix(oracle): say what the render-oracle score does and does not mean
Investigating the worst file of the first 250 turned the headline finding around. bug1552113.pdf scores 84%, and the cause is Poppler: it paints the link's `/Border [0 0 112]` as a 112-unit band across a 250x50 page, hiding the two lines of text. The fixture is named "this text should be visible", and in our rendering it is. The oracle is not ground truth, and the top of its worst list needs a person to look at both images before it is read as a bug of ours. That led to a plausible change that the measurement rejected. Poppler synthesises appearances for annotations that have none, where we draw only what the document carries -- `/AP`-less link borders are already recorded as not covered under view.render-annotation-appearances -- so hiding annotations looked like the way to measure page content alone. It is worse: it then scores Poppler for every appearance we do draw. Over the same 250 files, shown gives mean 2.16%, p90 5.35% and 25 files within 0.1%; hidden gives 2.18%, 5.55% and 22, with annotation-tx3.pdf jumping to 45.64% and annotation-tx2.pdf to 18.79%. The two biases nearly cancel. So the default is unchanged and `--hide-annotations` is an opt-in for explaining a single file, with the numbers written down so the experiment is not repeated. Verified that the default reproduces the merged baseline exactly and that the flag moves it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
f988cf87ed |
feat(oracle): measure rendering against Poppler, and turn the pixel test on
Gap 12 of docs/gaps.md: until rendering is measured, every claim about its quality is a judgement. Text extraction had an oracle and written output had qpdf; rendering had nothing. `cargo xtask render-oracle` renders page 1 of every corpus file with our CLI and with Poppler's pdftoppm and reports how far apart they are. The metric is the point. A raw pixel comparison is useless here -- two correct renderers disagree on every glyph edge, so it calls a perfect page badly wrong and says nothing about whether a shading is missing. `pixels::compare` counts a pixel as differing only when it differs from every pixel in the other image's 3x3 neighbourhood by more than a tolerance, in both directions, taking the worse. A one-pixel shift is forgiven; absent or miscoloured content is not. First reading, 250 files at 72 dpi, tolerance 8: 229 compared, median 0.89%, mean 2.16%, p90 5.35%, 25 within 0.1%. It reports rather than gates -- the corpus is not on CI, and a first reading of a borrowed renderer is a baseline to improve on, not a bar to clear. The worst results cluster on JBIG2 `bitmap-*` files and on ZapfDingbats, which aims the renderer work at something specific for the first time. 15 files could not be compared because the two renderers round the page box to different pixel sizes; they are reported as unmeasurable rather than scored. Verified against the knob's bounds end to end: tolerance 255 scores every file 0.00%, tolerance 0 takes the median to 0.95% and p90 to 26.36%. Also turns on `extract_type3_shared_xobjects_matches_poppler_pixels`, which had never run anywhere. Poppler is now installed on the Linux runner, and the test is version-robust by construction -- both sides of its comparison go through the same pdftoppm, so it is about our two documents rather than about Poppler's output. Verified in all three states: runs and passes with pdftoppm present, skips without it, fails without it when PDFCRAFT_ORACLES_REQUIRED is set. `run_capture` keeps its exact behaviour, including leaving the buffer empty on non-UTF-8 output; the byte-oriented capture an image needs is a second entry point on the same timeout logic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
50629eb6bf |
feat(testkit): one way to write a synthetic PDF fixture
55 files hand-rolled the same serialiser -- number the objects, record each one's offset, emit a classic cross-reference table and a trailer -- 111 times over. The object bodies are properly the business of the crate under test; the envelope around them is not, and every copy is a place for a fixture to be quietly wrong in a way that makes a test prove less than it appears to. `pdfcraft-testkit` holds that envelope. It has no workspace dependencies on purpose, so the crates at the bottom of the layering can use it without a cycle, and it returns bytes rather than a parsed document for the same reason. The builder is byte-identical to the serialiser it replaces, asserted against a verbatim copy of the old code across versions and trailer variations, because a migration that changes a fixture's bytes silently changes what the test using it proves. Watched failing by breaking the free-entry generation: three tests caught it. The one intended difference is that an empty stream dictionary gives `<< /Length N >>` rather than a doubled space. Migrates cos, measure and redact: 111 copies in 55 files becomes 108 in 53. The rest are follow-ups rather than one sweep -- ui-egui alone is 27 of those files and is the most contested area in the repository. Some copies should stay: a test about a malformed cross-reference table needs to write its own bytes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
0754740fd9 |
feat(oracle): make a missing external oracle loud instead of a silent pass
Every test that wanted an external oracle probed for it by hand and chose for itself what to do when it was missing. The common choice -- print a line and return -- reports as a pass, so a suite with no oracles installed looked exactly like one in which every oracle agreed. That was not hypothetical. No CI runner installed qpdf, so `qpdf_agrees` had never executed on a runner, while README.md said saved output was "independently checked with qpdf". For R2-R4 that test is the only check of owner-password authentication (Algorithm 7) that does not go through our own code. `pdfcraft-oracle` holds the probes and one rule: a missing oracle is a declared skip or a failure, never a silent pass. `PDFCRAFT_ORACLES_REQUIRED` picks which, on the same convention as `CRAFT_FONTS_REQUIRED`. CI installs qpdf and sets it, so a skip notice there is now a red build. The crate has no panicking path of its own; `or_skip!` expands its assert into the caller, which is test code. Verified by running the qpdf tests against qpdf 12.4.2 for the first time -- they pass, so the README claim is now both enforced and true. Both gate directions were watched: absent and unrequired skips with a notice, absent and required fails with the install command in the message. Only qpdf is installed in CI, because it is the only tool the test job uses. The Poppler pixel comparison in crates/organize stays ignored until a change turns it on and verifies it there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
727555ca29 |
fix(xtask): let a standalone crate dev-depend on test support
The standalone rule ran before the testkit exemption and returned early, so `cos`, `geom`, `filters`, `crypt` and `arlington` could not use `pdfcraft-testkit` or `pdfcraft-oracle` even as a dev-dependency. Both crates were already registered in the layering table as `Class::Testkit`, so the first real use of either from a standalone crate would have been rejected. A dev-dependency is not part of the graph a standalone crate publishes, which is what the rule protects; a normal dependency on test support, and any dependency on a product crate, are still rejected. The existing test covered neither direction of the standalone case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude
|
d9607ec3be |
F3: one documented substitution chain, and report what it chose and why
A font the document does not embed was replaced by whatever the caller happened to pick, and nothing told the user. `doc_info` said only `embedded: false`, which is why "my Chinese PDF shows nothing" has been a mystery rather than a sentence. `crates/fonts/src/fallback.rs` applies one chain, in a documented order, and returns what it chose with a reason: 1. embedded -> nothing substituted; 2. /BaseFont names a standard-14 face or a known alias (exact metrics); 3. craft-fonts holds that exact family; 4. /CIDSystemInfo /Ordering names a script we have a face for; 5. the ordering names a script we have no licensed face for -- report none, so nothing is drawn rather than the wrong glyphs; 6. otherwise the nearest of the fourteen by descriptor flags. It delegates to the existing per-script selectors instead of restating their order, so each keeps one source of truth. It answers the dictionary question only; "which face has a glyph for this character?" depends on the string being drawn and stays where the drawing happens. `doc_info` gains `substitute` and `substitution` per font. Both null means the document's own glyphs are shown; a null `substitute` with a reason means no face was available at all. That is Acrobat's Actual Font, minus the interface, which `parity/` now records as still missing. Two findings on the way: `Std14::from_base_font` matches loosely, so "SomeHouseSerif" resolved to Times by its name containing "serif". The face is a fair guess but reporting it as "names one of the standard 14" would have been untrue of a font nobody has heard of. `Std14::recognise` now says whether a name matched an alias or only a family hint, and the two report differently. `from_base_font` is unchanged. The generated table moved to `crates/fonts/src/std14_metrics.rs`, data alone, and the ATTRIBUTION entry and AGENTS.md 1.1 line follow it. F1 put the hash on a file holding both the table and the code reading it, so this commit's ordinary edits tripped the gate -- and a hash that churns on unrelated changes gets re-attested reflexively, which is the protection gone. Verified both directions after the split: editing std14.rs passes the gate, changing one width in std14_metrics.rs fails it. Every new test was watched failing first: removing the script step breaks the two unit tests and the doc_info integration test. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Tanish MankuandClaude
|
0a09864833 |
F1, F2: adapt to main and re-pin the Arabic layout golden
Rebase onto
|
||
|
Tanish MankuandClaude
|
ce9a1d1014 |
F2: read glyph names by the naming algorithm, and stop reading Symbol as Latin
Two gaps in code-to-Unicode, both found by probing the current behaviour rather than by reading the code. A font with no /Widths and no /ToUnicode fell back to StandardEncoding whatever it was, so /Symbol code 0x61 -- `alpha` in Symbol's own built-in encoding -- decoded as "a". ZapfDingbats and MacExpert were read as Latin the same way. That map is what `redact` matches patterns against, so a wrong character there is a silent redaction failure, and a wrong character is worse than no character. Symbol, ZapfDingbats and MacExpert now yield no mapping at all unless the font names a base encoding, which is always honoured as the deliberate statement it is. This keys on the base font *name*, not the /FontDescriptor symbolic flag. A great many ordinary Latin TrueType subsets set that flag and still rely on a Latin encoding, so suppressing on the flag would lose real text across a large share of real documents. A test pins that restraint so it is not "fixed" into a regression later. Glyph names now follow the published naming algorithm instead of two of its cases: a name is cut at its first `.` (`a.sc` is `a`), split on `_` into components (`f_i` is "fi"), and `uniXXXX` carries any number of four-digit values (`uni00660069` is "fi"). Names are resolved whole or not at all, so half a ligature is never returned. `glyph_text` returns a String for this; `glyph_unicode` stays as the single-character view. Hex parsing rejects the leading `+` that `from_str_radix` would otherwise accept, and a lone surrogate half. Both halves were verified by reverting each fix and watching the matching tests fail. The parity note on protect.redact-patterns now says Symbol-encoded text is skipped rather than mis-decoded. Still open, and an owner decision: Symbol and ZapfDingbats could be mapped properly with a glyph-name-to-Unicode table, but the vendored Adobe Glyph List is not on the AGENTS.md 1.1 closed list. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Tanish MankuandClaude
|
117430fade |
F1: measure an added item with the face it is written as
`Family::width` chose its standard-14 face from `bold` alone and passed `italic: false`, while `Family::base_font` writes the item out as the italic or oblique face. Every italic item was therefore laid out with upright metrics: Times-Italic `A` is 611 against Times-Roman's 722, so boxes, wrapping and centring were all sized from the wrong face. The measure now takes `italic` and the five call sites pass the item's own flag, which they already had in scope. The test walks all twelve style combinations and asserts the written face and the measured face are the same one, so the two can't drift apart again. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
aaa19d87af |
F1: put the generated table on the AGENTS.md 1.1 closed list
Isolated so it can be reverted on its own. AGENTS.md 1.1 keeps a closed list of Adobe-authored non-visual technical data, and adding to it needs the owner's explicit approval, which was given on 2026-10-08. The item itself -- the standard-14 font metrics and encoding tables from the PDF specification -- was already on that list; what is new is holding a copy first-party rather than only through hayro, which is what lets generated appearances measure with it and what survives hayro being retired in M2.8. The gate only understood the list for dependency-bundled files, and rejected adobe_data on a repository file outright. It now accepts it for a closed list of first-party paths, so crates/fonts/src/std14.rs carries an ATTRIBUTION entry with its SHA-256: regenerating the table fails the gate until the entry is updated, which makes a change to Adobe-derived data a deliberate act rather than an ordinary edit. Worth noting that .rs is not an asset extension, so the table would never have been scanned at all. Entering it on the list is a choice to be checked, not a requirement the gate imposed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude Opus 5
|
d0d1ad8192 |
F1: exact standard-14 metrics
pdfcraft_fonts::helvetica_width was a fifteen-line guess by character class, and seven crates generated appearance streams through it: annot, edit, forms, ocr, redact, sign and xfa. Times was Helvetica x 0.9, bold was x 1.05, Courier a flat 0.6. Against the published metrics a Times capital A is 722, so that estimate said 600 -- a 17% error on a letter, compounding across a line. Std14 carries the real advance widths for all fourteen faces, indexed by character code: WinAnsiEncoding for the twelve text fonts, which is what we write into the appearance streams we generate, and their own built-in encodings for Symbol and ZapfDingbats. The tables are generated by joining the published encodings with the per-glyph widths. helvetica_width now delegates, so every call site gained exact Helvetica with no change. The three places that faked another face by scaling it -- Family in edit::added, the paragraph measure in edit::text, and the redaction overlay fonts -- now select the right face instead. So does Metrics::width in fonts::pdf, which measures existing document text when a font carries no /Widths; that path feeds redaction's glyph removal, where a wrong width means cutting the wrong glyphs. Tests spot-check the table against values typed in from the specification by hand, so it is verified against the spec rather than against whatever produced it; plus Courier being fixed pitch everywhere, full printable-ASCII and Latin-1 coverage, Symbol and ZapfDingbats not being WinAnsi, /BaseFont aliases and subset prefixes resolving, and hostile text staying finite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tanish MankuandClaude
|
681548a47f |
Add a PR template, and record the review practice and external-PR bar
Replaces this PR's earlier contents. It previously added a committed `docs/plan/` tree -- a status baseline, an architecture doc and an execution plan. Since then `main` has grown `docs/gaps.md`, `docs/target-app-parity.md`, `docs/roadmap.md` and `docs/architecture.md`, which are actively maintained (gaps.md was reviewed 2026-10-10 and updated 2026-10-11) and cover the same ground better, because they sit in the flow contributors already touch. Keeping a parallel plan would have meant two sources of truth and a document that rots; it had already gone conflicting twice in three days. So the plan is dropped and only the parts with no equivalent anywhere remain. `.github/pull_request_template.md` is new -- the repository had none, with 167 PRs open and 66 of them conflicting. It leads with the two things a contributor cannot be expected to know and that cannot be undone after a merge: provenance (no Adobe asset, no committed font, no copied GPL/AGPL code, behaviour from the spec rather than from reading another implementation) and the never-crash rules. It also asks for a test that has been *seen to fail* before the fix, and says plainly that per-crate verification is preferred because CI runs the workspace anyway. CLAUDE.md gains three things it did not record: the four-zoom review habit (clause, function, file, project) with the tests that mirror it; the external-PR bar; and a note that builds and tests should be scoped per crate, because a full workspace build has locked up a development machine and CI covers it. The commit rule now also says that `main` moves fast enough for PRs to rot -- 396 commits in three days -- and that conflict risk tracks how contested the files are rather than how large the change is. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
|
0a8b230ca0 |
D1: pin and verify the pdf.js corpus (#339)
xtask corpus cloned mozilla/pdf.js at HEAD with no commit pin and no content check, and the corpus tests returned early when the directory was missing. So the corpus sweep measured a moving target, xtask/baselines/pdfjs.json was compared against whatever upstream happened to be that day, and a damaged or tampered checkout produced green results. AGENTS.md §2 asks for fixtures fetched pinned by commit and sha256-verified. Verification is two layers. The commit fixes which tree we ask for; the manifest digest fixes what landed on disk: one '<sha256> <path>' line per file, sorted by path, then one sha256 over that text. Git object ids are SHA-1, so the manifest is what makes this a sha256 check. The pin is 5d923c9 with 1455 files, which matches what the GitHub tree API reports for test/pdfs. The checkout sets core.autocrlf=false and core.eol=lf. This repository has autocrlf=true globally, so without that git would rewrite line endings on checkout and the same commit would hash differently on Windows than on Linux; the pin has to mean the same thing everywhere. A successful verification writes corpus/pdfjs/.pdfcraft-corpus.json. The corpus tests now skip only when nothing is fetched, and fail loudly when a checkout exists without that stamp. xtask check verifies before sweeping, so the baseline is never compared against an unknown tree. Checked end to end: the unrecorded-pin guard refuses to fetch; --update-pin fetches and prints the constants; appending one byte to one of the 1455 files is detected and repaired by re-fetching; removing the stamp makes the corpus tests fail with a message saying what to run. Against the verified corpus the tests pass: 964 files ok and 7 password cases. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
+1
|
cecbfad0ae |
D2, D3: fuzz every tool's arguments, and prove root confinement (#351)
* XFA: choice lists, signature, password, image and barcode fields; PNG and GIF pictures The xfa crate lays out the field kinds that were left blank or shown as text fields: - Choice lists become combo boxes (editable with textEntry) or list boxes (open="always"; multi-select with open="multiSelect"), with the shown items and the saved values of <items save="1"> as /Opt pairs, /V, /I and /DV as saved values, several on separate lines in the data. Field::saved_value maps shown text to saved values for the layout and the script tree alike, so scripts read what is saved. - Password fields are password text fields; signature fields are /Sig fields the app can sign. - Image fields show the picture the data (base64) or the template holds; barcode fields show their value as text, with a warning that no bars are drawn. - PNG and GIF pictures (fields and draws) are decoded to Flate image XObjects with a soft mask for transparency; JPEGs stay as they were. Sizes come from the headers, so layout never decodes; decoding is capped at 40 million pixels and failures are warnings, not errors. Engine: multi-select list values from scripts and data are one per line. README and parity notes updated (form.xfa-dynamic). Hardened after an adversarial review pass: picture sizes refused from the header past 20 million pixels or 65,535 a side, the GIF decoder given limits (a frame far larger than its screen is refused before allocation), template pictures read whole up to 32 MB, pictures decoded once and remembered in the catalog by content hash so re-layouts reuse them, data pictures that aren't images reported by field; choice values matched against saved values first, several values deduplicated and capped, /I ascending and unique, /DV through the same mapping, <value> children read, the item cap reported; the reopen path takes another viewer's choice values (several per line); CMYK JPEGs inverted only with Adobe APP14. * D5: README for cos and engine, and an honest cos status line Thirteen crates have no README, although the session protocol tells every session to read the README of the crate it is touching. These are the two biggest: cos is where every hostile file lands first, and engine is the whole API surface a frontend gets. Each states its layer and the rules that bind it, sketches the API against the real signatures, and ends with what is deliberately not done. The cos README spells out the clone-is-a-snapshot property and why both writers return bytes rather than taking a sink; the engine README spells out the four-step editing model that makes what you see what Save writes. Also corrects the module doc on cos/src/lib.rs, which still said encryption and object-stream output for full saves were not done. Both shipped: security.rs handles R2-R6 with crypt filters and /EFF, and SaveOptions::object_streams defaults to true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * D2, D3: fuzz every tool's arguments, and prove root confinement AGENTS.md §3 and §4 treat tool arguments as untrusted, but nothing fed hostile arguments to the tool table: xtask fuzz mutates PDFs only. And tools::check_args validates shape alone (object, no unknown keys, required present), not types or ranges, so a string where an integer belongs or i64::MIN where a page number belongs reaches the handler. Two tests. The first calls every tool with a zoo of wrong types, numeric extremes, empty and 100,000-character strings, NUL and right-to-left-override names, mistyped and deeply nested containers: 19,299 calls over 132 tools, each required to return Ok or a readable ToolError rather than panic. The second is the D3 audit mechanized: every path argument is tried with .. traversal, absolute, drive and UNC paths, and a planted symbolic link, and must be refused; 451 attempts, all refused. Reading the code says every handler resolves its own paths and that no tool dispatches another internally, so the PhotoCraft hole does not exist here. A test keeps that true as tools are added; a written finding would not. Both were proved to fail before passing. An out-of-bounds index in page_rotate is reported as: page_rotate panicked on {"degrees":-1,"doc":1}. Disabling resolve() is reported as 82 leaks naming the tool and path, including doc_save writing through ../ and through the symbolic link. The first draft of the confinement test did not catch a fully disabled resolve(): almost every tool failed on a non-existent document id before it ever looked at a path, and a file-listing diff cannot see a read. It now opens a real document first so handlers reach their path arguments, and asserts each call is refused rather than only diffing the filesystem. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Preserve alternate appearances and indirect push-button values Keep rollover, down and custom appearance entries when normal appearances are regenerated, without modifying shared dictionaries. Resolve indirect button caption and layout values. Cover full and incremental saved output, and wait for raster readiness in the existing Linux scroll test. Co-authored-by: Codex <codex@openai.com> * M13: common MCP tools through the confined automation path Preserve task tools and compact mode, add complete hints and strict argument-key errors, and catch escaped tool panics. Co-Authored-By: OpenAI Codex <noreply@openai.com> * M13: session resources and version-aware MCP result hints Keep per-document resources and legacy response shapes; add document/command resources and private cache hints for modern clients. Co-Authored-By: OpenAI Codex <noreply@openai.com> * M7: select and edit grouped Form XObject figures Co-Authored-By: Codex <noreply@openai.com> * M14: let neutral Windows locales fall through to display language Co-authored-by: Codex <codex@openai.com> * pdfcraft-cli --help lists `mcp` only when the build has it Follow-up to #540: a build without the `mcp` feature (`--no-default-features`) still listed `pdfcraft-cli mcp` in its usage, and `pdfcraft-cli mcp` then said "unknown command". USAGE is now put together with `concat!`, and its `mcp` lines come from a macro that is empty without the feature. In such a build `pdfcraft-cli mcp` says the MCP server was left out instead of calling it an unknown command. - tests/help.rs: usage_lists_mcp_only_when_built_with_it (runs in both feature sets) and mcp_without_the_feature_says_it_was_left_out (without the feature) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * engine README: the execution plan is local-only, so don't link a path that doesn't exist * Merge land/450 into land/414; XFA pictures are found again by a SHA-256 on their XObject The image cache was a /PCXfaImages map in the catalog, keyed by DefaultHasher (not stable across Rust releases) and read back from the file. Each image XObject now records the SHA-256 of its picture's bytes, and a re-layout finds the XObjects on the pages it replaces by that hash; nothing goes in the catalog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Merge land/414 into land/451; keep main's MCP protocol versions, cap command_batch at 1000 steps The 2026-07-28 revision and its resultType/ttlMs/cacheScope fields are dropped: main doesn't advertise them and they couldn't be checked against the spec. command_batch refuses more than 1000 steps. docs/mcp.md says that unknown argument keys are now refused with -32602 (a behaviour change for clients that sent extra keys). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Merge land/451 into land/351; escape targets stay in the test's temp dir, and out/image paths are tried Absolute escape targets are now paths under the test's own temp base instead of /etc/passwd, C:\Windows\… and a UNC share, so a confinement bug can only touch files the test made. String 'out' and 'image' properties (and any described as a file path) are tried too. The backslash traversal runs only on Windows: elsewhere it is an ordinary file name inside the root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Yuvraj Singh <singh.yuvraj31@icloud.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: sanyamsingh001 <314051937+sanyamsingh001@users.noreply.github.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: storybored379 <storybored379@users.noreply.github.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Flowershangfromthebranches <152056395+Flowershangfromthebranches@users.noreply.github.com> Co-authored-by: Streetblock <github@streetblock.de> Co-authored-by: lloyd <lloyd@stackworx.io> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
2d40d85450 |
D5: READMEs for cos and engine (#332)
* D5: README for cos and engine, and an honest cos status line Thirteen crates have no README, although the session protocol tells every session to read the README of the crate it is touching. These are the two biggest: cos is where every hostile file lands first, and engine is the whole API surface a frontend gets. Each states its layer and the rules that bind it, sketches the API against the real signatures, and ends with what is deliberately not done. The cos README spells out the clone-is-a-snapshot property and why both writers return bytes rather than taking a sink; the engine README spells out the four-step editing model that makes what you see what Save writes. Also corrects the module doc on cos/src/lib.rs, which still said encryption and object-stream output for full saves were not done. Both shipped: security.rs handles R2-R6 with crypt filters and /EFF, and SaveOptions::object_streams defaults to true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * M14: let neutral Windows locales fall through to display language Co-authored-by: Codex <codex@openai.com> * pdfcraft-cli --help lists `mcp` only when the build has it Follow-up to #540: a build without the `mcp` feature (`--no-default-features`) still listed `pdfcraft-cli mcp` in its usage, and `pdfcraft-cli mcp` then said "unknown command". USAGE is now put together with `concat!`, and its `mcp` lines come from a macro that is empty without the feature. In such a build `pdfcraft-cli mcp` says the MCP server was left out instead of calling it an unknown command. - tests/help.rs: usage_lists_mcp_only_when_built_with_it (runs in both feature sets) and mcp_without_the_feature_says_it_was_left_out (without the feature) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * engine README: the execution plan is local-only, so don't link a path that doesn't exist --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Streetblock <github@streetblock.de> Co-authored-by: Codex <codex@openai.com> Co-authored-by: lloyd <lloyd@stackworx.io> Co-authored-by: Brandon Thomas <echelon@gmail.com> |