A review found that a dense page came out as a stream PdfCraft's own
renderer refuses, so the page drew blank after optimizing: hayro-jbig2
decodes a text region of at most 10,000 symbol instances, and refuses
32,768 symbols or more. The stream was valid (Poppler decoded it), but
every mark went into one text region and up to 65,536 symbols were kept.
A 300 dpi letter page of the optimizer test's own text crossed it.
A page with more than 10,000 marks is now placed by several text
regions, each over a band of rows and all referring to the one symbol
dictionary, which each region but the last keeps retained (7.2.4). Past
32,767 distinct symbols the image is coded generically.
New tests: 25,000 copies placed by three regions, which decode; the
symbol caps decoded, not just reached; and in the optimizer, a whole
letter page of rendered text that looks the same as JBIG2 symbols (it
drew blank before). The READMEs say "still looked at" for the caps, and
that the scans figure is the six pages' total (20% on one page).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
A second review found two rules tested one way only: a mark 257 tall
(not just wide) is left to the generic region, which also bounds the
bitmap allocated for each mark, and runs joined at the lower-left corner
are one mark. The test now has a 1 x 257 column and an anti-diagonal.
The caps on distinct symbols (65,536) and their pixels (2^24) are tested
at and past them, as the run and row caps are.
The scans figure now names its noise: with 2% or more of the letters'
edge pixels flipped, symbols save at most 8% over the generic region.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Review found behaviour the tests let break unnoticed (each was a
surviving mutant, and each new test fails on it):
- black padding bits: a run reaching past the row would stop symbol
coding on many scans;
- Annex A's 32-bit range and PREV at exactly 256, where a decision would
go uncoded: coordinates past 4,435 on a 12,000-pixel-wide and a
10,050-row page, and the contexts of 4436's coding;
- marks joined at a corner are one mark; 256 pixels a side is a symbol,
257 is left to the generic region;
- the run and row caps, at and past them.
The roadmap's status line for this change now comes first, and the
"about a third smaller than Group 4 on scans" figure is replaced by what
was measured: 27-33% on the same pages with scan-like noise added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
pdfcraft_filters::encode_jbig2 stores each distinct mark of a 1-bit
image (8-connected black pixels, at most 256 a side) once, as a symbol
in a symbol dictionary, and places its exact copies with an immediate
lossless text region that refers to it. Larger marks, and in a second
plan marks that occur once, go to a generic region over their box. It
keeps whichever of that and the generic coding (encode_jbig2_generic,
unchanged) is smaller. Only exact copies share a symbol, so the stream
decodes to the same bits.
Written from the JBIG committee's ISO/IEC 14492 Final Committee Draft:
6.4 and 6.5 (text region and symbol dictionary decoding), 7.2, 7.4.2
and 7.4.3 (segment headers and syntax) and Annex A (the integer coders,
checked against its worked example and Table A.1).
On six pages of rendered 300 dpi text the output is 28% of the generic
coding's size and 17% of Group 4's, at about 100 ms a page; hayro-jbig2
and Poppler (pdfimages) both decode it to the original bits. Scans,
whose copies of a letter are seldom identical, gain little: that needs
refinement coding. A dictionary shared between pages (/JBIG2Globals)
is a further step.
The optimizer's Monochrome Images > JBIG2 (lossless) and doc_optimize
mono jbig2 use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
An image's /JBIG2Globals were decoded with the 1 GiB any stream may decode
to, before the JBIG2 page's size is known, so the page's pixel limit didn't
bound them: a few kilobytes of Flate could make export allocate up to 1 GiB.
They now decode within MAX_GLOBALS (32 MiB); globals past it skip the image,
with the limit in the reason. The test covers one byte over and exactly the
limit, and the image's /DecodeParms direct, by reference and in an array.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy Image and Copy Image in Original Format go through Export's
decoder, which reads JBIG2 now, so the view.copy-image parity note no
longer lists JBIG2 as missing. docs/target-app-parity.md's optimize row
lists JBIG2 symbol coding and lossy JBIG2 as missing instead of
monochrome JBIG2, with a status line and a revision row.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
With compatibility retain, doc_optimize reports the later of the
header's version and the catalog's /Version. It read /Version only when
it was direct, so a catalog saying 1.6 by reference was reported as the
header's 1.4. It is now resolved first, as the optimizer reads it
(a88c8afb). The bug was already on main; found in pre-push review as a
sibling of that fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Export ▸ Images skips an image whose JBIG2 page has another width or
more pixels than an export takes; a JBIG2 stencil mask refused the same
way is left out and the image exported opaque, as JPX and CCITT masks
are, which the export test now checks. The README and parity note said
only "is skipped". And redaction doesn't decode JBIG2 or Group 4 yet,
so a mark over part of an image the optimizer stored either way removes
the whole image; the optimizer README and parity note now say so (#4384
adds Group 4). Found in pre-push review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
When the optimizer stores an image as JBIG2 (PDF 1.4), it raises the
catalog's /Version for an older file. It read /Version only when it was
direct: a catalog saying 1.7 by reference looked like no version at all,
and was set to /1.4, lowering the document. It is now resolved first.
Found in pre-push review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The export checked an image's /Width × /Height against its 100-million-
pixel budget, and the JBIG2 page's width against /Width, but not the
page's height. The decoder decodes the whole page, rows past the
image's height too, and takes pages of up to about 2^28 pixels: a
20,000 × 4 image over a 20,000 × 10,000 JBIG2 page cost 200 million
pixels of decoding to export four rows. The page itself is now held to
the budget before it is decoded. Found by CodeRabbit on #4320.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The optimizer README's and optimize.mono-compression's "not yet" lists
said rows wider than 32,767 samples aren't stored. That is Group 4's
limit (MAX_G4_COLUMNS); JBIG2 takes rows of up to 1,048,576
(MAX_JBIG2_COLUMNS). Found by CodeRabbit on #4805.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Export ▸ Images and Save Image As decoded an image with
Stream::decoded, which reads /DecodeParms only when it is direct. An
image whose PNG predictor came by reference (the dictionary, an entry
of an array, or the numbers inside) kept its row tags in the samples,
and the picture came out shifted. The same was true of an indexed
image's lookup table given as a stream, and of a JBIG2 image's
/JBIG2Globals stream, which then couldn't be read, so the image was
skipped. All three now decode through Document::with_direct_parms
(#4304), as the optimizer does, and a test exports each. Found by
CodeRabbit on #4805; the image decode was already this way on main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
fa.tsv lacked the row: Persian came to main after it was written. It goes after "CCITT Group 4",
worded like the Arabic and Kurdish rows. id.tsv had it as an English copy beside rows it
translates; it now says "tanpa kehilangan", as de, tr and vi translate "lossless".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
JBIG2 images, and JBIG2 stencil masks, were reported as "can't be exported
yet", so an image the Optimizer stores as JBIG2 would stop exporting. They
are now decoded with hayro-jbig2 (already in the tree through hayro), with
their /JBIG2Globals from a DecodeParms dictionary or array, as JBIG2Decode
gives them: black as 0 (ISO 32000-2 §7.4.7). Unreadable globals, a JBIG2
page of another width, or fewer rows than the image skip the image; rows
past its height are dropped, so the output is bounded by the checked size.
docs/pdf-spec-parity.md §7.4 no longer says there is no JBIG2 encoding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_jbig2_generic writes 1-bit rows as a JBIG2
stream for JBIG2Decode (embedded organisation: a page information
segment and an immediate lossless generic region, no globals): template
0 with its nominal AT pixels, typical prediction, and the MQ arithmetic
coder (mq.rs), written from the JBIG committee's ISO/IEC 14492 Final
Committee Draft. The coder reproduces its Annex H.2 test sequence byte
for byte; output round-trips through hayro-jbig2, and Poppler renders
it the same as the original.
The optimizer's Monochrome Images gets JBIG2 (lossless) beside CCITT
Group 4 (still the default), doc_optimize mono jbig2; a document below
PDF 1.4 gets /Version /1.4 in its catalog. On 300 dpi text JBIG2 is
about a third smaller than Group 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
docs/pdf-spec-parity.md's §7.4 row still said there was no CCITT
encoding; it names the Optimizer's Group 4 encoder now, with a status
line and a revision row.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- Not only Export ▸ Images refuses a CCITT image until #4335: so does
everything that exports an image through it. Save Image As refuses
it, Copy Image fails, and Word, PowerPoint and HTML export leave the
picture out, with the page not reported. The optimize README and the
mono-compression parity note name them.
- optimize.mono-compression is partial now, so docs/target-app-parity.md
counts it on the table's own measurement: K Optimize and standards 15
shipped and 11 partial of 58 (35.3%), breadth 57.0% (423 + 75 of 808),
mainstream 63.3% (292 + 45 of 497), with a status line and a revision
row. ROADMAP.md's breadth rows and its K row say the same.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The optimizer's images decode no further than twice their samples plus
a byte a row, meant for ASCIIHex text under Flate; but hex text broken
into lines, as it usually is, is a little more than twice, so such an
image was left alone as if it were a decompression bomb (a 300 dpi
letter scan stayed Flate). It is three bytes for each now, a separator
for every two digits, and a test stores both images as hex text in
lines of 64 digits.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- Redaction doesn't decode CCITT, so a mark over part of an image the
optimizer stored as Group 4 removes the whole image (nothing leaks),
and Export ▸ Images refuses it. The optimize README and the
mono-compression parity note say so. Later PRs of this stack add both
(#4335 export, #4384 redaction).
- doc_reduce's description, Engine::reduced_bytes and docs/cli-batch.md
now list the black-and-white step; docs/target-app-parity.md's
optimize row gives the measured counts and no longer lists CCITT as
missing; the M11 milestone row lists it as done.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
A test now covers what qualifies and what doesn't: an ICC profile of
one component and a DeviceN of one colourant convert, three components
and two colourants (more than a bit a pixel) don't; an image mask
converts with /BitsPerComponent 1 given, not with 8; an image of exactly
80 million pixels converts, one row more doesn't. The byte totals are
checked exactly. Each of those rules could be broken before with no
test failing (cargo mutants). one_component's comment now says /G
counts with DeviceGray, as the code does.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
A 1-bit image was decoded for Group 4 up to MAX_DECODED (1 GiB), so a
few kilobytes of Flate could cost a gigabyte, and the same held for the
samples of a colour or grey image read to resample it (pixels). Both now
stop at twice the samples plus a byte a row and 4 KiB, as redaction
does; data past that can only be a decompression bomb, and the image is
left alone.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The decoders leave /Crypt out wherever it stands (Stream::filters), so
a chain of /Crypt and eight other filters decodes, but with_direct_parms
counted nine and left its parameters as they were: a predictor given by
reference was then skipped. Only the other filters count now, and a
/Crypt entry is null in the copy, as no decoder reads it. Tests cover
that chain and one exactly as long as a decoder takes.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Document::with_direct_parms resolved an indirect /DecodeParms and its
array entries but not the numbers inside (/Predictor 9 0 R), which
Dict::int doesn't follow. A 1-bit image with such a predictor was
decoded without it, its rows shifted by their tag bytes, and Group 4 of
that stored in its place; resampling read images the same way. It now
keeps, of each parameters dictionary, the numbers and flags the decoders
read, resolved, and /JBIG2Globals as a reference. Only those small
values are copied (an entry that is no dictionary is null), so a file
naming one big object from every key and filter can't have it copied a
hundred times.
The change is #4384's, moved down so that every PR stacked on this one
reads such parameters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Persian came to main after these rows were written, so fa.tsv had neither "Monochrome Images"
nor "CCITT Group 4". They go beside "Grayscale Images", worded like the Arabic and Kurdish rows,
with the Persian digit the catalogue uses elsewhere ("AES ۱۲۸-بیتی").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Stream::decoded reads only direct parameter dictionaries, so an image
whose /DecodeParms is a reference (whole, or as array entries) was
decoded without its predictor. For a 1-bit image with a PNG predictor
the rows came out shifted by their tag bytes, Group 4 of that was still
smaller, and the picture was replaced by a skewed one; resampling and
merging image strips read the same way.
pdfcraft_cos::Document::with_direct_parms gives the stream with the
parameters its filters use made direct: only the entries at the
filters' places, and none for a chain longer than any decoder takes,
so at most MAX_CHAIN objects are copied. The optimizer's pixels() and
the Group 4 step decode through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
A bare /Indexed, /I or /CalGray is no colour space on its own, so it can
only name a resource, which might be RGB: such an image is now left
alone instead of being read as one bit per pixel. doc_optimize refuses a
mono that isn't an object or a compression that isn't a known name,
instead of reading it as the default (from CodeRabbit's review).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_g4 encodes packed 1-bit rows as Group 4 (ITU-T
T.6, /K -1) through the fax crate (MIT, already in the tree via tiff),
with rows capped at 32,767 samples so its 16-bit positions can't
overflow; round trips are checked against hayro-ccitt.
The optimizer (Settings::mono, CCITT G4 by default, so Reduce File Size
too) re-encodes a 1-bit image of one colour component, or an image
mask, stored with no filter or general-purpose ones, when that is
smaller as stored. The samples decode to the same bits, so colour
space, decode array and masks stay. PDF Optimizer gets a Monochrome
Images row and doc_optimize a mono option. optimize.mono-compression
goes from planned to partial (JBIG2 and monochrome downsampling not
yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
* Make PdfCraft's app colour red
The icon tile, the Windows PDF file icon's badge and lines, the DMG background and the README
badges move from green #12a58a to crimson #b70228 (darker shade #8a011e, fold #f3cfd6).
Every PNG, .ico and .icns is re-rendered from the edited master SVGs with packaging/icons.sh
and resvg 0.47; the .icns is written with Pillow because iconutil is macOS-only, with
iconutil's size set. The DMG background is re-rendered with generate.py; its .DS_Store is
unchanged. ATTRIBUTION.toml hashes updated; cargo xtask assets passes.
* Fix OCR model attribution hashes
From the review of #5228. The finished CMYK group was drawn through
whatever paint transform a preceding shading pattern left on the
context, so its cyan half showed magenta. Hayro patch (27) now resets
the paint transform before drawing the group's pixels.
Two regression tests, both reported in that review:
- a pattern painted before the group (fails without the reset);
- the Color blend mode over an empty CMYK group, swept over all 256 K
levels (failed on nine levels before #5199's vello_cpu fix was merged
in; passes with it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9
Resolves test-file conflicts where main and this branch both appended tests:
main's file plus this branch's appended tests, unchanged.
fmt, clippy -p pdfcraft-forms -p pdfcraft-automation, and both crates' tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9
Resolves test-file conflicts where main and this branch both appended tests:
main's file plus this branch's appended tests, unchanged.
fmt, clippy -p pdfcraft-forms -p pdfcraft-automation, and both crates' tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9
Resolves test-file conflicts where main and this branch both appended tests:
main's file plus this branch's appended tests, unchanged.
fmt, clippy -p pdfcraft-forms -p pdfcraft-automation, and both crates' tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9
