diff --git a/crates/create/README.md b/crates/create/README.md index 05131d9038..a9aef83493 100644 --- a/crates/create/README.md +++ b/crates/create/README.md @@ -8,9 +8,7 @@ Layer L4. Create a PDF (Acrobat's Create a PDF tool), execution plan M10.2: (`/DCTDecode`, grey, RGB or Adobe-inverted CMYK); PNG is decoded and stored with Flate, with transparency as a soft mask. A picture of only pure black and white (a 1-bit PNG or TIFF, eight-bit grey, a two-colour palette, RGB without a profile) is stored at one bit per pixel, as - CCITT Group 4 or Flate, whichever is smaller as stored (`pdfcraft_filters::encode_g4`). - Redaction doesn't decode Group 4 yet: a mark over part of such a picture removes the whole - picture (nothing leaks). An ICC profile in the file (JPEG `APP2 ICC_PROFILE`, PNG + CCITT Group 4 or Flate, whichever is smaller as stored (`pdfcraft_filters::encode_g4`). An ICC profile in the file (JPEG `APP2 ICC_PROFILE`, PNG `iCCP`) that matches the image's colour space is kept as an `/ICCBased` colour space with the device space as `/Alternate`; images with the same profile share one profile object. JPEG metadata scanning skips marker fill bytes and reads application segments after the diff --git a/crates/optimize/README.md b/crates/optimize/README.md index ae6c749ef3..d0a412a311 100644 --- a/crates/optimize/README.md +++ b/crates/optimize/README.md @@ -58,9 +58,7 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going - Apart from the downsampling it is lossless: the samples keep their meaning, so the colour space, decode array and `/Mask` stay. Group 4 wins on scans, whose ragged edges defeat Flate; on clean computer-drawn 1-bit art Flate can be smaller. Not yet: JBIG2, Group 3 to Group 4 - without resampling, inline images, and Group 4 for rows wider than 32,767 samples. Redaction - doesn't decode Group 4 yet: a mark over part of an image stored this way removes the whole - image (nothing leaks). + without resampling, inline images, and Group 4 for rows wider than 32,767 samples. - **Discard objects:** thumbnails, alternate images, document tags, print settings. - **Clean up:** Flate for streams with no filter; for streams with an outer ASCII85 or ASCIIHex layer, that layer is removed when the result is smaller (the inner filters keep their encoded diff --git a/crates/redact/README.md b/crates/redact/README.md index 34c9f4fe1f..2b540927d1 100644 --- a/crates/redact/README.md +++ b/crates/redact/README.md @@ -18,11 +18,12 @@ clear_marks(&mut doc, None)?; // remove marks without app more than a fifth of its size (or a point) both ways; zero-size text goes when its origin is inside; - **images**: fully covered → removed; partly covered → a copy with the covered pixels cleared - (8/16-bit and 1/2/4-bit images, image masks; Flate/LZW/RL/A85/AHx). Codecs PdfCraft can't - re-encode (DCT, JPX, JBIG2, CCITT) are removed whole (fail-closed); inline images under an - area are removed. Images with separate `/SMask` or `/Mask` data, and images whose data decodes - to more than twice their declared raster (plus a byte per row and 4 KiB), are removed whole; - cleared copies drop `/Alternates` so alternate pixels cannot survive; + (8/16-bit and 1/2/4-bit images, image masks; Flate/LZW/RL/A85/AHx, and CCITT fax as the last + filter, decoded through `Document::decoded_image`; `/DecodeParms` given by reference count). + Codecs PdfCraft can't decode (DCT, JPX, JBIG2) are removed whole (fail-closed); inline images + under an area are removed. Images with separate `/SMask` or `/Mask` data, and images whose + data decodes to more than twice their declared raster (plus a byte per row and 4 KiB), are + removed whole; cleared copies drop `/Alternates` so alternate pixels cannot survive; - **vectors**: covered paths are removed; partly covered paths and shadings are clipped so nothing paints inside the areas; - **form XObjects**: rewritten recursively into new objects (pages sharing the original keep it); diff --git a/crates/redact/src/image.rs b/crates/redact/src/image.rs index 4e26fbd1fc..2d22cc8174 100644 --- a/crates/redact/src/image.rs +++ b/crates/redact/src/image.rs @@ -1,8 +1,9 @@ //! Clearing the pixels of an image under redaction regions. The image is decoded, every pixel //! whose footprint intersects a region is set to zero (no ink for image masks), and the result is //! written as a new Flate image; the original object is untouched (other pages may use it). -//! Images whose codec PdfCraft can't decode (DCT, JPX, JBIG2, CCITT) return `None`, and the -//! caller removes the whole image instead; that is fail-closed. +//! CCITT fax images decode through `Document::decoded_image`. Images whose codec PdfCraft can't +//! decode (DCT, JPX, JBIG2) return `None`, and the caller removes the whole image instead; that +//! is fail-closed. use pdfcraft_content::{Matrix, overlaps}; use pdfcraft_cos::{Document, Object, Stream}; @@ -54,11 +55,18 @@ fn decode_and_clear(doc: &Document, s: &Stream, ctm: &Matrix, rects: &[[f64; 4]] b"FlateDecode" | b"Fl" | b"LZWDecode" | b"LZW" | b"ASCII85Decode" | b"A85" | b"ASCIIHexDecode" | b"AHx" | b"RunLengthDecode" | b"RL" ) }; - let ok = match s.dict.get(b"Filter") { - None => true, - Some(Object::Name(n)) => supported(n), - Some(Object::Array(a)) => a.iter().all(|f| f.as_name().is_some_and(supported)), - _ => false, + // A CCITT fax image (its last filter) decodes through `Document::decoded_image`, after the + // filters before it, so a scan stored as Group 4 is cleared like any other image. + let fax = |f: &Object| f.as_name().is_some_and(|n| matches!(n, b"CCITTFaxDecode" | b"CCF")); + let readable = |f: &Object| f.as_name().is_some_and(supported); + let (ok, is_fax) = match s.dict.get(b"Filter") { + None => (true, false), + Some(f @ Object::Name(_)) => (readable(f) || fax(f), fax(f)), + Some(Object::Array(a)) => match a.split_last() { + Some((last, before)) if fax(last) => (before.iter().all(readable), true), + _ => (a.iter().all(readable), false), + }, + _ => (false, false), }; if !ok { return None; @@ -82,9 +90,10 @@ fn decode_and_clear(doc: &Document, s: &Stream, ctm: &Matrix, rects: &[[f64; 4]] bytes.checked_mul(8)?; // Decode no further than the raster can need: a filter stage may legitimately produce up to // twice it (ASCIIHex text under Flate) plus a PNG predictor's byte per row. Data past that - // can only be a decompression bomb; the image is then removed whole. + // can only be a decompression bomb; the image is then removed whole. A fax image decodes to + // at most its /Height rows (the filters before it within `MAX_DECODED`). let allowance = bytes.checked_add(h)?.checked_mul(2)?.checked_add(4096)?; - let mut data = s.decoded_within(allowance).ok()?; + let mut data = if is_fax { doc.decoded_image(s).ok()? } else { s.decoded_within(allowance).ok()? }; if data.len() < bytes { return None; } diff --git a/crates/redact/src/tests.rs b/crates/redact/src/tests.rs index 6db9cec8c7..760d0727cf 100644 --- a/crates/redact/src/tests.rs +++ b/crates/redact/src/tests.rs @@ -575,6 +575,40 @@ fn an_image_whose_predictor_is_given_by_reference_has_the_covered_pixel_cleared( assert_eq!(named_image(&saved, 0, b"PCRedacted1"), Some(vec![10, 0, 30, 40, 50, 60, 70, 80])); } +/// A CCITT fax image (a scan the optimizer or Create PDF stored as Group 4) has its covered +/// pixels cleared in a copy, like any image PdfCraft can decode, instead of being removed whole: +/// whether it sits behind another filter, with its parameters given by reference, or is a mask. +#[test] +fn a_fax_image_has_its_covered_pixels_cleared_instead_of_being_removed() { + // 16 × 2, all 1 bits (white), or all 0 bits (a mask painting everywhere); the left half of + // the top row is covered. + let g4 = pdfcraft_filters::encode_g4(&[0xFF; 4], 16, 2).unwrap(); + let painted = pdfcraft_filters::encode_g4(&[0x00; 4], 16, 2).unwrap(); + let hex: Vec = g4.iter().map(|b| format!("{b:02x}")).collect::().bytes().chain(*b">").collect(); + let image = "/Type /XObject /Subtype /Image /Width 16 /Height 2 /BitsPerComponent 1"; + let parms = "<< /K -1 /Columns 16 /Rows 2 >>"; + for (dict, data, extra, cleared) in [ + (format!("{image} /ColorSpace /DeviceGray /Filter /CCITTFaxDecode /DecodeParms {parms}"), g4.clone(), vec![], [0x00, 0xFF, 0xFF, 0xFF]), + ( + format!("{image} /ColorSpace /DeviceGray /Filter [/ASCIIHexDecode /CCITTFaxDecode] /DecodeParms [null 8 0 R]"), + hex, + vec![parms.as_bytes().to_vec()], + [0x00, 0xFF, 0xFF, 0xFF], + ), + // A mask's 0 bits paint: cleared bits are set to 1, which paints nothing. + (format!("{image} /ImageMask true /Filter /CCITTFaxDecode /DecodeParms {parms}"), painted, vec![], [0xFF, 0x00, 0x00, 0x00]), + ] { + let mut objs = vec![stream(&dict, data)]; + objs.extend(extra); + let mut doc = one_page(b"q 160 0 0 20 0 0 cm /Im Do Q", "/XObject << /Im 7 0 R >>", objs); + mark(&mut doc, 0, &[[0.0, 12.0, 75.0, 18.0]], ""); + let report = apply(&mut doc, None).unwrap(); + assert_eq!((report.images_cleared, report.images_removed), (1, 0), "{dict}"); + let saved = Document::open(Arc::new(write_full(&doc, &SaveOptions::default()).unwrap())).unwrap(); + assert_eq!(named_image(&saved, 0, b"PCRedacted1").as_deref(), Some(&cleared[..]), "{dict}"); + } +} + /// Apply Redactions gives the page its own pruned /Resources, but the dictionary it drew through /// before (inherited from the /Pages node, or shared with another page) still named the removed /// image and form, so a full-rewrite save kept them. diff --git a/parity/acrobat-features.toml b/parity/acrobat-features.toml index d2ef8f4f75..c9ca864d45 100644 --- a/parity/acrobat-features.toml +++ b/parity/acrobat-features.toml @@ -131,7 +131,7 @@ status = "partial" commands = [] tools = [] evidence = ["crates/filters/src/ccitt.rs::decoding_gives_the_rows_back", "crates/cos/src/document/tests.rs::fax_images_decode_with_their_parameters"] -notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images and the optimizer" +notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction" [[feature]] id = "core.decompression-limits" @@ -5792,8 +5792,8 @@ milestone = "M8" status = "shipped" commands = ["redact.apply"] tools = ["redact_apply"] -evidence = ["crates/redact/src/tests.rs::images_are_removed_or_have_their_pixels_cleared", "crates/redact/src/tests.rs::redaction_clears_pixels_intersected_without_covering_their_centres"] -notes = "intersected pixel footprints are cleared conservatively in a copy; DCT/JPX/JBIG2/CCITT images removed whole" +evidence = ["crates/redact/src/tests.rs::images_are_removed_or_have_their_pixels_cleared", "crates/redact/src/tests.rs::redaction_clears_pixels_intersected_without_covering_their_centres", "crates/redact/src/tests.rs::a_fax_image_has_its_covered_pixels_cleared_instead_of_being_removed", "crates/redact/src/tests.rs::an_image_whose_predictor_is_given_by_reference_has_the_covered_pixel_cleared"] +notes = "intersected pixel footprints are cleared conservatively in a copy, CCITT fax images (scans stored as Group 4) included; DCT/JPX/JBIG2 images removed whole" [[feature]] id = "protect.redact-vectors" @@ -6885,7 +6885,7 @@ status = "shipped" commands = ["create.images"] tools = ["doc_create"] evidence = ["crates/create/src/tests.rs::black_and_white_images_are_stored_at_one_bit", "crates/create/src/tests.rs::jpeg_2000_images_are_embedded_as_is", "crates/create/src/tests.rs::images_export_as_jpeg_unchanged_and_others_as_png", "crates/create/src/tests.rs::embedded_icc_profiles_tag_the_image_colour_space"] -notes = "PNG, JPEG, TIFF (multi-page), GIF (first frame), BMP, and JPEG 2000 (JP2 or codestream, embedded as JPXDecode with its capture resolution). JPEG and PNG ICC profiles are kept as /ICCBased; TIFF profiles are not read yet. A picture of only pure black and white (a 1-bit PNG or TIFF, eight-bit grey, a two-colour palette, RGB without a profile) is stored at one bit per pixel, as CCITT Group 4 or Flate, whichever is smaller. Redaction doesn't decode Group 4 yet: a mark over part of such a picture removes it whole (nothing leaks)" +notes = "PNG, JPEG, TIFF (multi-page), GIF (first frame), BMP, and JPEG 2000 (JP2 or codestream, embedded as JPXDecode with its capture resolution). JPEG and PNG ICC profiles are kept as /ICCBased; TIFF profiles are not read yet. A picture of only pure black and white (a 1-bit PNG or TIFF, eight-bit grey, a two-colour palette, RGB without a profile) is stored at one bit per pixel, as CCITT Group 4 or Flate, whichever is smaller" [[feature]] id = "create.from-multipage-tiff" @@ -7385,7 +7385,7 @@ status = "partial" commands = ["optimize.advanced", "optimize.reduce"] tools = ["doc_optimize", "doc_reduce"] evidence = ["crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"] -notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. An eight-bit grey image of only black and white is packed to one bit first, then stored the same way (indexed images and colour-key masks aside), unless the grey settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Not yet: JBIG2 (lossless or lossy, which Adobe's help says beats CCITT for monochrome), Group 3 to Group 4 without resampling, inline images, soft masks and /Mask images, rows wider than 32,767 samples. Redaction doesn't decode Group 4 yet (a mark over part of such an image removes it whole; nothing leaks)." +notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. An eight-bit grey image of only black and white is packed to one bit first, then stored the same way (indexed images and colour-key masks aside), unless the grey settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Not yet: JBIG2 (lossless or lossy, which Adobe's help says beats CCITT for monochrome), Group 3 to Group 4 without resampling, inline images, soft masks and /Mask images, rows wider than 32,767 samples." [[feature]] id = "optimize.only-if-smaller"