diff --git a/crates/filters/Cargo.toml b/crates/filters/Cargo.toml index 91b5d3d578..fd63ac2b0b 100644 --- a/crates/filters/Cargo.toml +++ b/crates/filters/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "pdfcraft-filters" -description = "PDF stream filters (ISO 32000-2 §7.4): Flate, LZW, predictors, ASCII85, ASCIIHex, RunLength, CCITT Group 4 encoding" +description = "PDF stream filters (ISO 32000-2 §7.4): Flate, LZW, predictors, ASCII85, ASCIIHex, RunLength, CCITT Group 4 encoding and Group 3 and 4 decoding" version.workspace = true edition.workspace = true license.workspace = true diff --git a/crates/filters/README.md b/crates/filters/README.md index b8a4c53727..b8c1d22815 100644 --- a/crates/filters/README.md +++ b/crates/filters/README.md @@ -43,14 +43,14 @@ decode_ccitt(data, ¶ms, max_rows) -> Result, FilterError> // CCITT | ASCIIHexDecode | white space ignored, odd digit padded with 0, `>` EOD (optional), other characters → Corrupt | upper case, newline every 64 digits, `>` | | ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` | | RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 | -| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says (`pdfcraft_cos::Document::decoded_image` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt | +| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::Document::decoded_image` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt | ## Tests - Unit tests per filter with hand-computed vectors (the ISO 32000-2 LZW example, RFC 1951 stored blocks, PNG/TIFF rows). - LZW is cross-checked against `weezl` (dev-dependency) in both directions for both early-change settings. - `tests/proptests.rs`: encode→decode round trips for every encodable filter × predictor × colors × bpc × columns, two-filter chains, output-limit checks, and fuzz-style no-panic tests (random bytes, random/nonsense params, bit-flipped valid streams). -- `ccitt.rs`: `decode_ccitt` gives back the rows `encode_g4` was given (with `/BlackIs1`, `/Rows` and a row limit), refuses bad widths, and bounds its output for any input. Group 4 output decoded by `hayro-ccitt` (an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics. +- `ccitt.rs`: `decode_ccitt` gives back the rows `encode_g4` was given (with `/BlackIs1`, `/Rows` and a row limit), refuses bad widths, and bounds its output for any input; byte-aligned Group 3 in each layout, over 32 seeded pictures, decodes to its picture, and a byte short to fewer rows. Group 4 output decoded by `hayro-ccitt` (an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics. - `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`). ## Status diff --git a/crates/filters/src/ccitt.rs b/crates/filters/src/ccitt.rs index 7a2a310775..e87a82fe16 100644 --- a/crates/filters/src/ccitt.rs +++ b/crates/filters/src/ccitt.rs @@ -84,7 +84,9 @@ impl Default for CcittParams { /// Group 3 rows on bytes (`k` from 0, `encoded_byte_align`) are read by their end-of-line codes, /// whether the fill starts each code on a byte or ends it on one, when `data` starts with one (T.4 /// sends one before the first row when it sends them at all), and on bytes otherwise, whatever -/// `end_of_line` says. Group 4 is read as its parameters say. +/// `end_of_line` says. Group 4 is read as its parameters say. Data with codes but none before the +/// first row is read on bytes, as PdfCraft's renderer draws it; with `k` 0 and the fill ending each +/// code on a byte, that may be other rows than were coded. /// /// Errors: `columns` that aren't from 1 to [`MAX_CCITT_COLUMNS`] ([`FilterError::Unsupported`]). pub fn decode_ccitt(data: &[u8], p: &CcittParams, max_rows: u32) -> Result, FilterError> { diff --git a/crates/optimize/README.md b/crates/optimize/README.md index 4709111278..2c90981770 100644 --- a/crates/optimize/README.md +++ b/crates/optimize/README.md @@ -41,7 +41,10 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going 450 ppi, CCITT Group 4). Black-and-white images are a 1-bit image with one colour component (grey, indexed, a separation) or an image mask, stored with no filter, with general-purpose ones or with CCITT fax after any of those (decoded by `Document::decoded_image_within`, no further - than its rows can need), and an eight-bit grey image of + than its rows can need; byte-aligned Group 3 with end-of-line codes but none before the first + row, not as T.4 sends them, is read as PdfCraft draws it, which at `/K 0` may be other rows than + were coded, and an image resampled, or stored again as Group 4 from behind another filter, + keeps them), and an eight-bit grey image of only 0 and 255, packed to one bit (not an indexed image, whose samples are palette indices, nor one with a colour-key `/Mask`) unless the grey settings make it smaller still, its soft mask counted (resampled with it there, kept as it is here), as they can a dithered scan by downsampling it to @@ -64,8 +67,8 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going settings. Every change is kept only when it is smaller as stored. - Apart from the downsampling it is lossless: the samples keep their meaning, so the colour space, decode array and `/Mask` stay. Group 4 wins on scans, whose ragged edges defeat Flate; - on clean computer-drawn 1-bit art Flate can be smaller. Not yet: JBIG2, Group 3 to Group 4 - without resampling, inline images, an image's own `/SMask` or `/Mask` image (only images drawn + on clean computer-drawn 1-bit art Flate can be smaller. Not yet: JBIG2, a lone Group 3 image to + Group 4 without resampling (one behind another filter is stored again as Group 4), inline images, an image's own `/SMask` or `/Mask` image (only images drawn on a page are visited), and Group 4 for rows wider than 32,767 samples. - **Discard objects:** thumbnails, alternate images, document tags, print settings. - **Clean up:** Flate for streams with no filter; for streams with an outer ASCII85 or ASCIIHex diff --git a/crates/redact/README.md b/crates/redact/README.md index 6d273c2341..74509a5d06 100644 --- a/crates/redact/README.md +++ b/crates/redact/README.md @@ -20,7 +20,9 @@ clear_marks(&mut doc, None)?; // remove marks without app - **images**: fully covered → removed; partly covered → a copy with the covered pixels cleared (8/16-bit and 1/2/4-bit images, image masks; Flate/LZW/RL/A85/AHx, and CCITT fax as the last filter, decoded through `Document::decoded_image_within`; `/DecodeParms` given by reference, - numbers inside included, count). Codecs redaction doesn't decode (DCT, JPX, JBIG2) are removed + numbers inside included, count; byte-aligned Group 3 with end-of-line codes but none before the + first row, not as T.4 sends them, is cleared as PdfCraft draws it, which at `/K 0` may be other + rows than were coded). Codecs redaction doesn't decode (DCT, JPX, JBIG2) are removed whole (fail-closed); inline images under an area are removed. Images with separate `/SMask` or `/Mask` data, and images whose data decodes to more than three times their declared raster and a byte per row, plus 4 KiB (ASCIIHex text broken into lines under Flate is a little over diff --git a/docs/roadmap.md b/docs/roadmap.md index 6d12e87564..56b10853a2 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -3,6 +3,7 @@ > **Last reviewed:** 2026-10-27 | **Last updated:** 2026-10-27 | **Change:** minor (M10: OCR model downloads use verified ONNX pairs with legacy discovery fallback; estimates unchanged) > **Last reviewed:** 2026-10-25 | **Last updated:** 2026-10-25 | **Change:** minor (M14 localization current state: Arabic logical catalog, explicit RTL label direction and mirrored stock chrome; native/browser RTL workflows unqualified. Native release Hans face via LXGW WenKai Lite; estimates unchanged) > **Last reviewed:** 2026-10-24 · **Last updated:** 2026-10-24 · **Change:** minor (M11: black-and-white images above 450 ppi are averaged down to 300 ppi by the optimizer; estimates unchanged) +> **Last reviewed:** 2026-10-24 · **Last updated:** 2026-10-24 · **Change:** minor (M11: black-and-white pictures stay one bit when PDFs are created and optimized; CCITT images export; estimates unchanged) > **Last reviewed:** 2026-10-23 · **Last updated:** 2026-10-23 · **Change:** minor (M11: the PDF Optimizer and Reduce File Size store black-and-white images as CCITT Group 4 when smaller; estimates unchanged) > **Last reviewed:** 2026-10-23 · **Last updated:** 2026-10-23 · **Change:** minor (M7, #2636 partial: pasting added text boxes and images in the engine and through `content_copy`; estimates unchanged) > **Last reviewed:** 2026-10-23 · **Last updated:** 2026-10-23 · **Change:** minor (M11: image optimization accounts for page-local UserUnit; estimates unchanged) diff --git a/parity/acrobat-features.toml b/parity/acrobat-features.toml index d656b51a22..8f22714769 100644 --- a/parity/acrobat-features.toml +++ b/parity/acrobat-features.toml @@ -130,8 +130,8 @@ milestone = "M1" status = "partial" commands = [] tools = [] -evidence = ["crates/filters/src/ccitt.rs::decoding_gives_the_rows_back", "crates/cos/src/document/tests.rs::fax_images_decode_with_their_parameters"] -notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction" +evidence = ["crates/filters/src/ccitt.rs::decoding_gives_the_rows_back", "crates/filters/src/ccitt.rs::group_3_rows_on_bytes_read_by_the_codes_in_the_data", "crates/cos/src/document/tests.rs::fax_images_decode_with_their_parameters"] +notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction. Byte-aligned Group 3 is read by its end-of-line codes when the data starts with one and on bytes otherwise; still limited: data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at /K 0 may come out as other rows, which the optimizer and redaction then write back" [[feature]] id = "core.decompression-limits" @@ -7385,7 +7385,7 @@ status = "partial" commands = ["optimize.advanced", "optimize.reduce"] tools = ["doc_optimize", "doc_reduce"] evidence = ["crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"] -notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. An eight-bit grey image of only black and white is packed to one bit first, then stored the same way (indexed images and colour-key masks aside), unless the grey settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Not yet: JBIG2 (lossless or lossy, which Adobe's help says beats CCITT for monochrome), Group 3 to Group 4 without resampling, inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples." +notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. An eight-bit grey image of only black and white is packed to one bit first, then stored the same way (indexed images and colour-key masks aside), unless the grey settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Not yet: JBIG2 (lossless or lossy, which Adobe's help says beats CCITT for monochrome), a lone Group 3 image to Group 4 without resampling (one behind another filter is stored again as Group 4; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded), inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples." [[feature]] id = "optimize.only-if-smaller"