Merge pull request #5472 from Piohrun/feat/mono-jbig2

M11: Monochrome Images reads JBIG2 scans: averaged like any, and stored again as JBIG2 (optimize.mono-compression)
This commit is contained in:
FredApps
2026-11-01 22:44:16 +01:00
committed by GitHub
7 changed files with 174 additions and 67 deletions
+1 -1
View File
@@ -1451,7 +1451,7 @@ pub fn tools() -> Vec<ToolDef> {
.with(schema(json!({ "doc": doc() }), &["doc"])),
t("doc_open_revision", "Open a revision", "Open saved revision `revision` (1 = the oldest) of a document as a new, unsaved document, to see the file as it was then.")
.with(schema(json!({ "doc": doc(), "revision": { "type": "integer", "minimum": 1 } }), &["doc", "revision"])),
t("doc_optimize", "Optimize PDF", "Write an optimized copy to `path` (Acrobat's PDF Optimizer). preset: standard (default; Reduce File Size's choices) or mobile (images to 96 ppi above 144, JPEG high quality); the options below change it. compatibility: retain (default; keeps the document's version, or 1.5 when object streams are written), 1.4, 1.5, 1.6, 1.7 or 2.0. merge_fragments (default true) first puts a page picture drawn as many stacked image strips (as some PDF printers write it) back together into one image, so it can be resampled (merged_images, fragments_merged). color / gray: { downsample, ppi, above_ppi, compression: jpeg|zip|retain, quality 1–100 } (defaults: downsample to 150 ppi above 225, JPEG 60). mono: { downsample, ppi, above_ppi, compression: ccitt_g4|jbig2|retain } for black-and-white images and image masks (default: averaged down to 300 ppi above 450 ppi, counted in images_resampled; stored losslessly as CCITT Group 4 or JBIG2 (or 1-bit Flate, once resampled or packed from more bits) when that is smaller, Group 3 fax images, black-and-white RGB and indexed images using only palette entries 0 and 1 among them, counted in images_recompressed; retain resamples only, a CCITT image staying CCITT and any other becoming Flate). Images are measured where pages draw them and replaced only when smaller. discard_*: thumbnails (default true), alternate_images (true), tags, print_settings; flate_unencoded (true); object_streams (default true; always off for 1.4); lzw_to_flate (true) decodes LZW streams and stores them with Flate instead, leaving any it cannot decode unchanged (lzw_streams counts the streams converted); remove_invalid_links and remove_unreferenced_dests (true); images and forms the pages list but never draw are always dropped (unused_xobjects). discard: Remove Hidden Information categories (metadata, attachments, comments, form-fields, hidden-text, hidden-layers, bookmarks, links-actions-scripts, private-data). Signed documents are refused. The open document is unchanged.")
t("doc_optimize", "Optimize PDF", "Write an optimized copy to `path` (Acrobat's PDF Optimizer). preset: standard (default; Reduce File Size's choices) or mobile (images to 96 ppi above 144, JPEG high quality); the options below change it. compatibility: retain (default; keeps the document's version, or 1.5 when object streams are written), 1.4, 1.5, 1.6, 1.7 or 2.0. merge_fragments (default true) first puts a page picture drawn as many stacked image strips (as some PDF printers write it) back together into one image, so it can be resampled (merged_images, fragments_merged). color / gray: { downsample, ppi, above_ppi, compression: jpeg|zip|retain, quality 1–100 } (defaults: downsample to 150 ppi above 225, JPEG 60). mono: { downsample, ppi, above_ppi, compression: ccitt_g4|jbig2|retain } for black-and-white images and image masks (default: averaged down to 300 ppi above 450 ppi, counted in images_resampled; stored losslessly as CCITT Group 4 or JBIG2 (or 1-bit Flate, once resampled or packed from more bits) when that is smaller, Group 3 fax images, JBIG2 images, black-and-white RGB and indexed images using only palette entries 0 and 1 among them, counted in images_recompressed; retain resamples only, a CCITT image staying CCITT, a JBIG2 image JBIG2 and any other becoming Flate). Images are measured where pages draw them and replaced only when smaller. discard_*: thumbnails (default true), alternate_images (true), tags, print_settings; flate_unencoded (true); object_streams (default true; always off for 1.4); lzw_to_flate (true) decodes LZW streams and stores them with Flate instead, leaving any it cannot decode unchanged (lzw_streams counts the streams converted); remove_invalid_links and remove_unreferenced_dests (true); images and forms the pages list but never draw are always dropped (unused_xobjects). discard: Remove Hidden Information categories (metadata, attachments, comments, form-fields, hidden-text, hidden-layers, bookmarks, links-actions-scripts, private-data). Signed documents are refused. The open document is unchanged.")
.cmd("optimize.advanced")
.with(schema(
json!({
+15 -11
View File
@@ -31,7 +31,7 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
stored as JPEG (Acrobat's quality steps) or ZIP; soft masks are resampled with their image.
An image is replaced only if the result is smaller. Left alone by these colour and grey
settings: CMYK and other colour spaces, image masks, decode arrays, more than 8 bits, JPEG 2000,
JBIG2 and CCITT (black-and-white images, CCITT among them, are below). A JPEG is read with
JBIG2 and CCITT (black-and-white images, CCITT and JBIG2 among them, are below). A JPEG is read with
its colour transform (an Adobe marker's, else `/ColorTransform`, ISO 32000-2 §7.4.8), so one
stored untransformed keeps its colours; one whose marker and `/ColorTransform` disagree is left
alone, since readers differ on which wins. Samples are read with `/DecodeParms` given by
@@ -40,7 +40,7 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
- **Monochrome images** (`Settings::mono`, `MonoSettings::REDUCE` by default: to 300 ppi above
450 ppi, CCITT Group 4). Black-and-white images are a 1-bit image with one colour component
(grey, indexed, a separation) or an image mask, stored with no filter, with general-purpose ones
or with CCITT fax after any of those (decoded by `Document::decoded_image_within`, no further
or with CCITT fax or JBIG2 after any of those (decoded by `Document::decoded_image_within`, no further
than its rows can need; byte-aligned Group 3 with end-of-line codes but none before the first
row, not as T.4 sends them, is read as PdfCraft draws it, which at `/K 0` may be other rows than
were coded, and an image stored again keeps them). Packed to one bit first, unless a colour-key `/Mask` is in the image's
@@ -78,25 +78,29 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
600 ppi page of text that is averaged, Group 4 at 300 ppi is about half the size.
- **Compression:** CCITT Group 4 (`pdfcraft_filters::encode_g4`), or 1-bit Flate when that is
smaller, which only a resampled image or one packed from more bits may be: a 1-bit image that
is only stored again becomes Group 4 or stays. Group 4 alone is only decoded to be resampled;
is only stored again becomes Group 4 or stays. Group 4 alone is only decoded to be resampled, and so is JBIG2 alone when Group 4 is asked for;
Group 3, or Group 4 inside another filter such as `ASCII85Decode`, is stored again as Group 4
alone when that is smaller. Damaged Group 3 with end-of-line codes is stored as the decoder
recovers it, a bad row picked up again at the next code, as it reports no damage there; without
them, damage stops the decoding and the image is left alone. With Retain, a resampled image
keeps its kind of compression (CCITT, or Flate) and nothing else changes; an image of more bits
is then left to the grey or colour settings. Every change is kept only when it is smaller as
stored.
them, damage stops the decoding and the image is left alone. JBIG2 inside another filter, or
alone when JBIG2 is asked for, is stored again as the compression asks when that is smaller (as JBIG2 symbols where
its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact
repeats and often comes out larger, and then stays as it is. A JBIG2 image stored again refers to no
`/JBIG2Globals`, and one whose page isn't its size (PdfCraft's
renderer draws the whole page into the image) is left alone. With Retain, a resampled image
keeps its kind of compression (CCITT, JBIG2, or Flate) and nothing else changes; an image of
more bits is then left to the grey or colour settings. Every change is kept only when it is
smaller as stored.
- Apart from the downsampling it is lossless: the samples keep their meaning, so the colour
space, decode array and `/Mask` stay, but for packed RGB, which becomes grey with one decode
pair. Group 4 wins on scans, whose ragged edges defeat Flate; on clean computer-drawn 1-bit
art Flate can be smaller. Not yet: JBIG2 images themselves (left as they are, neither
downsampled nor stored again, though `Document::decoded_image` reads them), inline images, an image's own `/SMask` or `/Mask`
art Flate can be smaller. Not yet: inline images, an image's own `/SMask` or `/Mask`
image (only images drawn on a page are visited), and Group 4 for rows wider than 32,767
samples.
- Lossless JBIG2 uses bounded generic-region encoding or exact-copy symbol dictionaries when
smaller. No lossy symbol substitution is used. Export reads embedded images and bounded
shared globals.
smaller. No lossy symbol substitution is used. Export, redaction and this step read JBIG2
images, with bounded shared globals.
- **Discard objects:** thumbnails, alternate images, document tags, print settings.
- **Clean up:** Flate for streams with no filter; for streams with an outer ASCII85 or ASCIIHex
layer, that layer is removed when the result is smaller (the inner filters keep their encoded
+52 -18
View File
@@ -463,7 +463,7 @@ fn is_dct(s: &Stream) -> Option<bool> {
#[cfg(test)]
thread_local! {
/// Pixels decoded from JPEG data, for the test that keeps a mismatched JPEG from decoding.
/// Pixels decoded, for the tests that keep an image from being decoded for nothing.
static DECODED_PIXELS: std::cell::Cell<u64> = const { std::cell::Cell::new(0) };
}
@@ -801,7 +801,7 @@ type Pixel = fn(&[u8]) -> Option<bool>;
/// with its width and height, and, when they were packed from more bits, its dictionary as a 1-bit
/// image. These count:
/// - one bit per pixel in one colour component, or an image mask, stored with no filter, with
/// general-purpose ones, or with CCITT fax last ([`Document::decoded_image_within`], no further
/// general-purpose ones, or with CCITT fax or JBIG2 last ([`Document::decoded_image_within`], no further
/// than its rows can need);
/// - with `pack`, and with no colour-key `/Mask` (its ranges are in the image's own values):
/// - eight-bit grey (one component, not indexed) whose every sample is 0 or 255. A decode array
@@ -832,11 +832,10 @@ fn one_bit(doc: &Document, s: &Stream, pack: bool, defaults: bool) -> Option<(Ve
let len = stride.checked_mul(hu)?;
let max = match s.filters().last() {
Some((pdfcraft_filters::Filter::CcittFax, _)) => wu.checked_mul(hu)?.checked_add(hu.checked_mul(4)?)?.checked_add(4096)?,
// `decoded_image_within` reads JBIG2 too, but not for this yet: a resampled JBIG2 image
// would come out Flate under Retain, and nothing weighs JBIG2 again against it.
Some((pdfcraft_filters::Filter::Jbig2, _)) => return None,
// A JBIG2 page, which has to be the image's size, is `len` bytes as rows of a bit.
_ => allowance(len, h)?,
};
count_decoded(u64::from(w) * u64::from(h));
let mut data = doc.decoded_image_within(s, max).ok()?;
if data.len() < len {
return None;
@@ -1094,12 +1093,13 @@ fn downsample_bits_checked(data: &[u8], w: u32, h: u32, nw: u32, nh: u32, dark_i
/// (a 50% screen black, a 25% one white). So would a square of four new pixels that each cover
/// both, as a small screened picture on a white page has (and ragged scanned text, which is then
/// left at its resolution too).
/// - **Stored** as CCITT Group 4 with `CcittG4`, or as 1-bit Flate when that is smaller, which only
/// a resampled image or one packed from more bits may be: a 1-bit image that is only stored again
/// becomes Group 4 or stays. Group 4 alone is only decoded to be resampled; Group 3 is stored
/// again like any other. With `Retain`, a resampled image keeps its kind of compression
/// (CCITT, or Flate), and nothing else changes: an image of more bits stays as it is, for the
/// grey or colour settings.
/// - **Stored** as CCITT Group 4 with `CcittG4`, or JBIG2 with `Jbig2`, or as 1-bit Flate when that
/// is smaller, which only a resampled image or one packed from more bits may be: a 1-bit image
/// that is only stored again becomes Group 4 or JBIG2, or stays. Group 4 alone is only decoded
/// to be resampled, and so is JBIG2 alone when Group 4 is asked for; Group 3, Group 4 inside
/// another filter, and JBIG2 otherwise are stored again like any other. With `Retain`, a resampled image keeps its kind of compression (CCITT, JBIG2, or
/// Flate), and nothing else changes: an image of more bits stays as it is, for the grey or
/// colour settings. A JBIG2 image stored again refers to no `/JBIG2Globals`.
///
/// The samples keep their meaning, so the colour space, decode array and `/Mask` stay, but for
/// black-and-white RGB, which becomes grey with one decode pair, and an indexed image's default
@@ -1111,10 +1111,14 @@ fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSet
let fax = doc.fax_params(s);
// Its kind of compression, which Retain keeps, behind any other filters.
let ccitt = fax.is_some();
// Group 4 alone is only worth decoding to resample it. Group 3, or Group 4 behind
// ASCII85Decode, say, can be smaller stored again as Group 4 alone.
let g4_alone = fax.is_some_and(|p| p.k < 0) && s.filters().len() == 1;
if (retain || (g4_alone && settings.compression == MonoCompression::CcittG4)) && !downsample {
let jbig2 = matches!(s.filters().last(), Some((pdfcraft_filters::Filter::Jbig2, _)));
// Group 4 alone is only worth decoding to resample it, and so is JBIG2 alone when Group 4 is what
// it would be stored as: that output is rarely smaller, and decoding an ordinary scan costs a
// quarter of a second a page. Group 3, Group 4 behind ASCII85Decode, say, and JBIG2 stored as
// JBIG2 (whose symbols may be smaller than its own coding) can be smaller stored again alone.
let alone_to_g4 = s.filters().len() == 1 && settings.compression == MonoCompression::CcittG4;
let only_to_resample = retain || (alone_to_g4 && (jbig2 || fax.is_some_and(|p| p.k < 0)));
if only_to_resample && !downsample {
return None;
}
let (data, w, h, packed) = one_bit(doc, s, !retain, defaults)?;
@@ -1146,7 +1150,7 @@ fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSet
Some(out) => (out, nw, nh),
None => (data, w, h),
};
if (retain || (g4_alone && settings.compression == MonoCompression::CcittG4)) && !resampled {
if only_to_resample && !resampled {
return None;
}
let mut dict = packed.unwrap_or_else(|| d.clone());
@@ -1169,7 +1173,7 @@ fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSet
} else {
None
};
let jbig = if settings.compression == MonoCompression::Jbig2 {
let jbig = if settings.compression == MonoCompression::Jbig2 || (retain && jbig2) {
pdfcraft_filters::encode_jbig2(&data, nw, nh).ok().map(|coded| {
let mut dict = dict.clone();
dict.set(b"Filter".to_vec(), Object::name("JBIG2Decode"));
@@ -1178,7 +1182,7 @@ fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSet
} else {
None
};
let flate = ((resampled || packed_from_more) && !(retain && ccitt)).then(|| Stream::flate(dict, &data));
let flate = ((resampled || packed_from_more) && !(retain && (ccitt || jbig2))).then(|| Stream::flate(dict, &data));
// Smaller as stored, counting the dictionary: a tiny image can lose by Group 4's parameters.
let best = [g4, jbig, flate].into_iter().flatten().filter_map(|c| Some((crate::stored_len(&c)?, c))).min_by_key(|(len, _)| *len)?;
(best.0 < crate::stored_len(s)?).then_some((best.1, resampled))
@@ -1321,6 +1325,36 @@ mod dimension_tests {
assert_eq!(pixels(&Document::new_empty(), &stream, 1), Some(vec![1, 2, 3, 4]));
}
/// JBIG2 alone, with Group 4 asked for, is decoded only to be resampled. A cost guard: dropped,
/// no output changes, and an ordinary scan pays a quarter of a second a page for nothing. Behind
/// another filter it can still be stored again, as Group 4 can.
#[test]
fn a_jbig2_image_alone_is_not_decoded_for_group_4_unless_it_is_resampled() {
let coded = pdfcraft_filters::encode_jbig2_generic(&[0xFF; 4], 16, 2).unwrap();
let hex: Vec<u8> = coded.iter().flat_map(|b| format!("{b:02x}").into_bytes()).chain(*b">").collect();
let image = |filter: Object, data: Vec<u8>| {
let mut dict = Dict::new();
dict.set(b"Width".to_vec(), Object::Int(16));
dict.set(b"Height".to_vec(), Object::Int(2));
dict.set(b"BitsPerComponent".to_vec(), Object::Int(1));
dict.set(b"ColorSpace".to_vec(), Object::name("DeviceGray"));
dict.set(b"Filter".to_vec(), filter);
Stream::from_raw(dict, data)
};
let settings = MonoSettings { compression: MonoCompression::CcittG4, ..MonoSettings::REDUCE };
let decoded = |s: &Stream, settings: &MonoSettings| {
DECODED_PIXELS.with(|c| c.set(0));
mono(&Document::new_empty(), s, 300.0, false, settings);
DECODED_PIXELS.with(|c| c.get())
};
let alone = image(Object::name("JBIG2Decode"), coded.clone());
assert_eq!(decoded(&alone, &settings), 0, "alone, for Group 4, below the threshold");
let behind = image(Object::Array(vec![Object::name("ASCIIHexDecode"), Object::name("JBIG2Decode")]), hex);
assert!(decoded(&behind, &settings) > 0, "behind another filter it can be stored again");
let jbig2 = MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE };
assert!(decoded(&alone, &jbig2) > 0, "stored as JBIG2 its symbols may come out smaller");
}
/// A JPEG larger than its dictionary says is refused from its header, before its pixels are decoded.
#[test]
fn a_jpeg_of_another_size_than_its_dictionary_is_not_decoded() {
+1 -1
View File
@@ -10,7 +10,7 @@
//! A new image replaces the old one only if it is smaller. The colour and grey settings leave
//! alone images they can't decode faithfully (CMYK and other colour spaces, masks, decode arrays
//! other than the default, JPEG 2000, JBIG2, CCITT, other than 8 bits, JPEGs whose Adobe marker
//! and `/ColorTransform` disagree). Black-and-white images (1-bit, image masks and CCITT fax,
//! and `/ColorTransform` disagree). Black-and-white images (1-bit, image masks, CCITT fax and JBIG2,
//! Group 3 among them; and eight-bit grey and RGB of only black and white, and indexed images
//! that use only palette entries 0 and 1, packed to one bit) above a threshold are averaged down to a
//! target resolution when grey, image masks or indexed, halftones and dithers aside (see the README), and
+98 -29
View File
@@ -2045,12 +2045,16 @@ fn rendered_text_becomes_jbig2_symbols() {
assert!(render(&doc) == before, "the page looks the same");
}
/// A JBIG2 image is left as it is above 450 ppi, whatever the compression asked for, and so is one
/// behind another filter: `Document::decoded_image` reads JBIG2, but the monochrome step doesn't
/// take it yet (a resampled one would come out Flate under Retain). The same picture stored with
/// Flate is averaged down under each, so it is the JBIG2 that stops it.
/// A JBIG2 scan goes through Monochrome Images like any black-and-white image. Above 450 ppi it
/// is averaged as its Flate copy is, to the same samples, and stored as each compression asks:
/// Group 4 or JBIG2, or Flate when that is smaller, or under Retain JBIG2 again; alone or behind another
/// filter, and with `/JBIG2Globals`, which the new image doesn't refer to. Below 450 ppi it is
/// stored again only when that is smaller: a generic region of text stays under Group 4, which
/// is larger, and under Retain, and becomes symbols with JBIG2 asked for; behind ASCIIHex it is
/// stored again alone, as Group 4 where the hex is spaced out. A JBIG2 page taller than the image
/// is left alone.
#[test]
fn jbig2_images_are_left_as_they_are() {
fn jbig2_scans_are_averaged_and_stored_again() {
// Black squares 16 samples a side on even rows and columns: at 600 ppi, halving leaves no new
// pixel covering both samples, so nothing keeps the picture at its resolution.
let (w, h) = (1200u32, 480u32);
@@ -2063,38 +2067,103 @@ fn jbig2_images_are_left_as_they_are() {
}
}
}
let coded = pdfcraft_filters::encode_jbig2_generic(&squares, w, h).unwrap();
let hex: Vec<u8> = coded.iter().flat_map(|b| format!("{b:02x}").into_bytes()).chain(*b">").collect();
let grey = [("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))];
let hex = |coded: &[u8]| -> Vec<u8> { coded.iter().flat_map(|b| format!("{b:02x}").into_bytes()).chain(*b">").collect() };
// An image of `coded` behind `filters`, with globals a decoder skips when `globals`, drawn at
// `width` × `height` pt.
let jbig2_page = |coded: &[u8], filters: &[&str], globals: bool, (iw, ih): (u32, u32), (width, height): (f64, f64)| {
let mut doc = Document::new_empty();
let mut d = Dict::new();
d.set(b"Subtype".to_vec(), Object::name("Image"));
d.set(b"Width".to_vec(), Object::Int(iw.into()));
d.set(b"Height".to_vec(), Object::Int(ih.into()));
for (k, v) in &grey {
d.set(k.as_bytes().to_vec(), v.clone());
}
d.set(b"Filter".to_vec(), Object::Array(filters.iter().map(|f| Object::name(f)).collect()));
if globals {
let g = doc.add(Object::Stream(Stream::from_raw(Dict::new(), vec![0, 0, 0, 0, 0x3E, 0, 0, 0, 0, 0, 4, 0, 0, 0, 1])));
let mut parms = Dict::new();
parms.set(b"JBIG2Globals".to_vec(), Object::Ref(g));
let entries = filters.iter().map(|f| if *f == "JBIG2Decode" { Object::Dict(parms.clone()) } else { Object::Null }).collect();
d.set(b"DecodeParms".to_vec(), Object::Array(entries));
}
let data = if filters.len() > 1 { hex(coded) } else { coded.to_vec() };
let im = doc.add(Object::Stream(Stream::from_raw(d, data)));
page(&mut doc, &[("Im", im)], &format!("q {width} 0 0 {height} 20 600 cm /Im Do Q"));
(doc, im)
};
let coded = pdfcraft_filters::encode_jbig2_generic(&squares, w, h).unwrap();
// 144 × 57.6 pt is 600 ppi.
let at_600 = (144.0, 57.6);
for compression in [MonoCompression::CcittG4, MonoCompression::Jbig2, MonoCompression::Retain] {
let settings = Settings { mono: MonoSettings { compression, ..MonoSettings::REDUCE }, ..Settings::default() };
// 144 × 57.6 pt is 600 ppi.
let content = "q 144 0 0 57.6 20 600 cm /Im Do Q";
let mut doc = Document::new_empty();
let im = mono_image(&mut doc, w, h, &squares, None, &grey);
page(&mut doc, &[("Im", im)], content);
let r = optimize(&mut doc, &settings).unwrap();
assert_eq!(r.images_resampled, 1, "Flate, {compression:?}: {r:?}");
for (filters, data) in
[(vec![Object::name("JBIG2Decode")], &coded), (vec![Object::name("ASCIIHexDecode"), Object::name("JBIG2Decode")], &hex)]
{
let mut doc = Document::new_empty();
let mut d = Dict::new();
d.set(b"Subtype".to_vec(), Object::name("Image"));
d.set(b"Width".to_vec(), Object::Int(w.into()));
d.set(b"Height".to_vec(), Object::Int(h.into()));
for (k, v) in &grey {
d.set(k.as_bytes().to_vec(), v.clone());
}
d.set(b"Filter".to_vec(), Object::Array(filters.clone()));
let im = doc.add(Object::Stream(Stream::from_raw(d, data.clone())));
page(&mut doc, &[("Im", im)], content);
let before = stream(&doc, im);
page(&mut doc, &[("Im", im)], "q 144 0 0 57.6 20 600 cm /Im Do Q");
assert_eq!(optimize(&mut doc, &settings).unwrap().images_resampled, 1, "Flate, {compression:?}");
let averaged = doc.decoded_image(&stream(&doc, im)).unwrap();
for (filters, globals) in [(&["JBIG2Decode"][..], false), (&["ASCIIHexDecode", "JBIG2Decode"][..], false), (&["JBIG2Decode"][..], true)] {
let (mut doc, im) = jbig2_page(&coded, filters, globals, (w, h), at_600);
let r = optimize(&mut doc, &settings).unwrap();
assert_eq!((r.images_resampled, r.images_recompressed), (0, 0), "{filters:?}, {compression:?}: {r:?}");
assert_eq!(stream(&doc, im), before, "{filters:?}, {compression:?}");
let s = stream(&doc, im);
let what = format!("{filters:?}, globals {globals}, {compression:?}: {r:?}");
assert_eq!((r.images_resampled, s.dict.int(b"Width"), s.dict.int(b"Height")), (1, Some(600), Some(240)), "{what}");
assert_eq!(doc.decoded_image(&s).unwrap(), averaged, "{what}");
assert!(!s.dict.contains(b"DecodeParms") || s.dict.name(b"Filter") == Some(&b"CCITTFaxDecode"[..]), "{what}");
let filter = s.dict.name(b"Filter").map(<[u8]>::to_vec);
// The smallest of what each compression may store; under Retain, JBIG2 again.
let allowed: &[&[u8]] = match compression {
MonoCompression::CcittG4 => &[b"CCITTFaxDecode", b"FlateDecode"],
MonoCompression::Jbig2 => &[b"JBIG2Decode", b"FlateDecode"],
MonoCompression::Retain => &[b"JBIG2Decode"],
};
assert!(filter.as_deref().is_some_and(|f| allowed.contains(&f)), "{what}: {:?}", filter.map(String::from_utf8));
}
}
// At 300 ppi, below 450: rendered text, whose letters repeat, coded as one generic region.
let (tw, th) = (600u32, 480u32);
let text = rendered_text(tw, th);
let generic = pdfcraft_filters::encode_jbig2_generic(&text, tw, th).unwrap();
let at_300 = (144.0, 115.2);
for compression in [MonoCompression::CcittG4, MonoCompression::Retain] {
let settings = Settings { mono: MonoSettings { compression, ..MonoSettings::REDUCE }, ..Settings::default() };
let (mut doc, im) = jbig2_page(&generic, &["JBIG2Decode"], false, (tw, th), at_300);
let before = stream(&doc, im);
optimize(&mut doc, &settings).unwrap();
assert_eq!(stream(&doc, im), before, "JBIG2 below 450 ppi, {compression:?}");
}
let jbig2_asked = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
let (mut doc, im) = jbig2_page(&generic, &["JBIG2Decode"], false, (tw, th), at_300);
let r = optimize(&mut doc, &jbig2_asked).unwrap();
let s = stream(&doc, im);
assert_eq!((r.images_resampled, r.images_recompressed, s.dict.name(b"Filter")), (0, 1, Some(&b"JBIG2Decode"[..])), "{r:?}");
assert!(s.raw.len() * 2 < generic.len(), "symbols {} bytes, the generic region {}", s.raw.len(), generic.len());
assert_eq!(doc.decoded_image(&s).unwrap(), text);
// Behind ASCIIHex it is stored again alone, being smaller so.
let (mut doc, im) = jbig2_page(&generic, &["ASCIIHexDecode", "JBIG2Decode"], false, (tw, th), at_300);
optimize(&mut doc, &jbig2_asked).unwrap();
let s = stream(&doc, im);
assert_eq!((s.filters().len(), s.dict.name(b"Filter")), (1, Some(&b"JBIG2Decode"[..])));
assert_eq!(doc.decoded_image(&s).unwrap(), text);
// So it is as Group 4 when that is smaller: here the hex digits are spaced out, as ASCIIHex
// allows, so that the stream is many times the code.
let (mut doc, im) = jbig2_page(&generic, &["ASCIIHexDecode", "JBIG2Decode"], false, (tw, th), at_300);
let mut padded = stream(&doc, im);
padded.raw = generic.iter().flat_map(|b| format!("{b:02x}{}", " ".repeat(30)).into_bytes()).chain(*b">").collect::<Vec<u8>>().into();
doc.set(im, Object::Stream(padded));
optimize(&mut doc, &Settings::default()).unwrap();
let s = stream(&doc, im);
assert_eq!((s.filters().len(), s.dict.name(b"Filter")), (1, Some(&b"CCITTFaxDecode"[..])), "Group 4 alone");
assert_eq!(doc.decoded_image(&s).unwrap(), text);
// A JBIG2 page taller than the image: the renderer draws the whole page into it, so it isn't
// read, and stays as it is.
let (mut doc, im) = jbig2_page(&coded, &["JBIG2Decode"], false, (w, h - 8), at_600);
let before = stream(&doc, im);
optimize(&mut doc, &Settings::default()).unwrap();
assert_eq!(stream(&doc, im), before, "a taller page");
}
/// A whole 300 dpi letter page of rendered text, more marks than one JBIG2 text region of the
+2 -2
View File
@@ -73,8 +73,8 @@ The PDF object graph *is* the model (`crates/engine/README.md` §The editing mod
- PDF in: any revision chain, xref tables or streams, hybrid files, damaged files (repaired and
logged), all standard-security revisions. Image codecs (DCT, JPX) decode through the bootstrap
renderer; CCITT and JBIG2 decode through `pdfcraft-filters` (`decode_ccitt`, `decode_jbig2`,
behind `Document::decoded_image`) for export and redaction, CCITT for the optimizer too, and
through the renderer for display. Export ▸ Images decodes JPEG itself.
behind `Document::decoded_image`) for export, redaction and the optimizer, and through the
renderer for display. Export ▸ Images decodes JPEG itself.
- PDF out: incremental or full saves, object streams, re-encryption, resource de-duplication when
combining.
- Other formats: images and text in (`create`); PNG/JPEG/TIFF, text, RTF, Word and HTML out
+5 -5
View File
@@ -131,7 +131,7 @@ status = "partial"
commands = []
tools = []
evidence = ["crates/filters/src/ccitt.rs::decoding_gives_the_rows_back", "crates/filters/src/ccitt.rs::group_3_rows_on_bytes_read_by_the_codes_in_the_data", "crates/cos/src/document/tests.rs::fax_images_decode_with_their_parameters", "crates/filters/src/jbig2/decode.rs::both_codings_decode_to_their_samples", "crates/filters/src/jbig2/decode.rs::what_it_cannot_decode_is_an_error", "crates/cos/src/document/tests.rs::jbig2_images_decode_with_their_globals", "crates/cos/src/document/tests.rs::shared_jbig2_globals_are_read_once"]
notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction; so is JBIG2 decoding (decode_jbig2, through hayro-jbig2, the page bounded before it is decoded), which it uses with the image's /JBIG2Globals (within 32 MiB) for Export ▸ Images, Copy Image and redaction. Byte-aligned Group 3 is read by its end-of-line codes when the data starts with one and on bytes otherwise; still limited: data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at /K 0 may come out as other rows, which the optimizer then writes back. The bootstrap renderer draws byte-aligned Group 3 whose fill ends each end-of-line code on a byte as other rows (black box, at every /K), which decode_ccitt reads right; redaction removes Group 3 on bytes whole"
notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction; so is JBIG2 decoding (decode_jbig2, through hayro-jbig2, the page bounded before it is decoded), which it uses with the image's /JBIG2Globals (within 32 MiB) for Export ▸ Images, Copy Image, redaction and the optimizer. Byte-aligned Group 3 is read by its end-of-line codes when the data starts with one and on bytes otherwise; still limited: data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at /K 0 may come out as other rows, which the optimizer then writes back. The bootstrap renderer draws byte-aligned Group 3 whose fill ends each end-of-line code on a byte as other rows (black box, at every /K), which decode_ccitt reads right; redaction removes Group 3 on bytes whole"
[[feature]]
id = "core.decompression-limits"
@@ -7404,8 +7404,8 @@ milestone = "M11"
status = "partial"
commands = ["optimize.advanced"]
tools = ["doc_optimize"]
evidence = ["crates/optimize/tests/optimize.rs::images_are_measured_where_drawn_and_downsampled", "crates/optimize/tests/optimize.rs::settings_choose_what_happens", "crates/automation/tests/it/automation.rs::optimizing_through_tools", "crates/optimize/tests/optimize.rs::a_page_picture_printed_as_one_pixel_strips_is_merged_into_one_image", "crates/optimize/tests/optimize.rs::images_that_do_not_tile_a_picture_are_left_alone", "crates/optimize/tests/optimize.rs::a_deeply_nested_form_is_measured_at_its_largest_use", "crates/optimize/tests/optimize.rs::images_are_not_resampled_when_some_uses_could_not_be_measured", "crates/optimize/tests/optimize.rs::image_resolution_uses_the_pages_physical_unit", "crates/optimize/tests/optimize.rs::optimization_retains_an_image_below_threshold_on_a_scaled_page", "crates/optimize/tests/optimize.rs::shared_forms_are_measured_in_each_pages_physical_units", "crates/optimize/tests/optimize.rs::ambiguous_monochrome_text_is_recompressed_without_lossy_reduction", "crates/optimize/tests/optimize.rs::indexed_thin_rules_survive_averaging_for_both_palette_orders_and_decodes", "crates/optimize/src/images.rs::an_indexed_images_palette_says_which_sample_is_dark", "crates/optimize/src/images.rs::it_matches_the_slow_way", "crates/automation/tests/it/automation.rs::doc_optimize_averages_black_and_white_scans_down", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"]
notes = "colour and grayscale, bicubic, effective resolution measured where drawn (every use, with page-local UserUnit scaling, through nested forms up to 12 deep; when a use can't be measured, nothing is resampled); image fragments stacked edge to edge are merged first (side-by-side tiles not yet); monochrome grey, image masks and indexed images (1-bit, CCITT fax, and eight-bit black-and-white grey and RGB packed to one bit) averaged by area to 300 ppi above 450 ppi by default, a tie to the dark sample (black in grey, painted in a mask, the darker palette entry over DeviceGray or DeviceRGB, after a decode array of two different numbers), then stored as Group 4 or 1-bit Flate (Monochrome Images ▸ Downsample to; doc_optimize mono); not separations or DeviceN (their tint transforms aren't read), calibrated or ICC grey, indexed images over another base or with a table over 4 KiB, images drawn or listed where /DefaultGray or /DefaultRGB is or may be in force (an indexed image also used beyond the page and form resources read, as an annotation's appearance), images with a colour key, which can hide the dark sample, or a soft mask, halftones and dithers (most of their new pixels would cover both black and white, and their tones turn solid), images with a square of four such new pixels (a small screened picture on a white page; ragged scanned text has them along its strokes too, and keeps its resolution), or images wider than a fax row (1,048,576 samples), and Acrobat's bicubic and subsampling choices for monochrome aren't offered"
evidence = ["crates/optimize/tests/optimize.rs::images_are_measured_where_drawn_and_downsampled", "crates/optimize/tests/optimize.rs::settings_choose_what_happens", "crates/automation/tests/it/automation.rs::optimizing_through_tools", "crates/optimize/tests/optimize.rs::a_page_picture_printed_as_one_pixel_strips_is_merged_into_one_image", "crates/optimize/tests/optimize.rs::images_that_do_not_tile_a_picture_are_left_alone", "crates/optimize/tests/optimize.rs::a_deeply_nested_form_is_measured_at_its_largest_use", "crates/optimize/tests/optimize.rs::images_are_not_resampled_when_some_uses_could_not_be_measured", "crates/optimize/tests/optimize.rs::image_resolution_uses_the_pages_physical_unit", "crates/optimize/tests/optimize.rs::optimization_retains_an_image_below_threshold_on_a_scaled_page", "crates/optimize/tests/optimize.rs::shared_forms_are_measured_in_each_pages_physical_units", "crates/optimize/tests/optimize.rs::ambiguous_monochrome_text_is_recompressed_without_lossy_reduction", "crates/optimize/tests/optimize.rs::indexed_thin_rules_survive_averaging_for_both_palette_orders_and_decodes", "crates/optimize/src/images.rs::an_indexed_images_palette_says_which_sample_is_dark", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::it_matches_the_slow_way", "crates/automation/tests/it/automation.rs::doc_optimize_averages_black_and_white_scans_down", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"]
notes = "colour and grayscale, bicubic, effective resolution measured where drawn (every use, with page-local UserUnit scaling, through nested forms up to 12 deep; when a use can't be measured, nothing is resampled); image fragments stacked edge to edge are merged first (side-by-side tiles not yet); monochrome grey, image masks and indexed images (1-bit, CCITT fax, JBIG2, and eight-bit black-and-white grey and RGB packed to one bit) averaged by area to 300 ppi above 450 ppi by default, a tie to the dark sample (black in grey, painted in a mask, the darker palette entry over DeviceGray or DeviceRGB, after a decode array of two different numbers), then stored as Group 4 or 1-bit Flate (Monochrome Images ▸ Downsample to; doc_optimize mono); not separations or DeviceN (their tint transforms aren't read), calibrated or ICC grey, indexed images over another base or with a table over 4 KiB, images drawn or listed where /DefaultGray or /DefaultRGB is or may be in force (an indexed image also used beyond the page and form resources read, as an annotation's appearance), images with a colour key, which can hide the dark sample, or a soft mask, halftones and dithers (most of their new pixels would cover both black and white, and their tones turn solid), images with a square of four such new pixels (a small screened picture on a white page; ragged scanned text has them along its strokes too, and keeps its resolution), or images wider than a fax row (1,048,576 samples), and Acrobat's bicubic and subsampling choices for monochrome aren't offered"
[[feature]]
id = "optimize.recompress-images"
@@ -7428,8 +7428,8 @@ milestone = "M11"
status = "partial"
commands = ["optimize.advanced", "optimize.reduce"]
tools = ["doc_optimize", "doc_reduce"]
evidence = ["crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_images_are_left_as_they_are"]
notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. Not yet: inline images, soft masks and /Mask images, JBIG2 images themselves (left as they are: neither downsampled nor stored again), Group 4 for rows wider than 32,767 samples."
evidence = ["crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again"]
notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples."
[[feature]]
id = "optimize.only-if-smaller"