mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-11 16:36:13 -06:00
M11: JBIG2 images share a symbol dictionary, up to 16 at a time (optimize.mono-compression) (#5910)
* M11: JBIG2 images share a symbol dictionary, up to 16 at a time With Monochrome Images set to JBIG2, Optimize stored each image's letters in a symbol dictionary of its own, so a document of N pages carried its font's shapes N times. Now up to 16 images the run stores as JBIG2 share one dictionary in a /JBIG2Globals stream, holding the marks found in more than one of them; each image's text regions refer to it first, then to a dictionary of its own for the marks only it repeats. An image is stored that way when it comes out smaller, and a group only when what its images save is more than the globals take. - filters: encode_jbig2_shared codes a group: the globals are a symbol dictionary segment numbered 0 and associated with no page (7.2.6); each page numbers its segments from 1, as a segment refers only to lower numbers (7.2.5), and the shared symbols' IDs come first, as the dictionaries are referred to in that order (Table 31, SBSYMS). At most 16,383 shared marks, those in the most images, so a page keeps room for its own under the decoder's 32,767. - optimize: the run holds the 1-bit samples of the images it stores as JBIG2 (16 at most, 64 MiB at most) and stores them again sharing, weighing each with the longest reference the globals could have. Eight pages of 10 pt text rendered at 300 dpi take 41% less in Helvetica and 42% in Times than coded alone, 10% in Courier, and 46% in Helvetica at 600 dpi; groups of 2, 4, 16 and 32 save 19%, 33%, 45% and 47%. Redacting one image stores the others of its group again without the dictionary, so 16 keeps that to 15 images. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: test what the mutants found untested in shared JBIG2 dictionaries - filters: the group goes through encode_jbig2_shared, which refuses it for an image the encoder refuses; two marks in common are enough to share; past the 65,536 distinct marks a group holds, a mark first seen later isn't shared while those held count every image. - optimize: optimizing again keeps both the shared pages and the pages coded alone as they are; under Retain, averaged JBIG2 pages share nothing; a last group of two shares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: only images redaction can store again share a JBIG2 dictionary Redacting one image of a group stores the others again without the dictionary, which can hold the shapes of marks under the area, and refuses the redaction for one it can't: it takes image masks and images of one component as its own reading counts them (no DeviceN, the family named directly), and at most 64 Mi pixels. Optimize took DeviceN of one colourant and up to 80 million pixels, so such an image in a group made any redaction of the others fail, where before it was coded alone and the same redaction worked (the Consumers review). Only an image mask or a DeviceGray image named directly, of at most 2^26 pixels, now joins a group; others are coded alone. An engine test optimizes three pages, two DeviceGray and one DeviceN, and redacts the first: the DeviceN page stays alone, the second is stored again, and no image names the dictionary after a save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: optimize and redaction share one limit on the images redaction stores again The second Consumers review: Optimize's limit on images that share a JBIG2 dictionary copied redaction's private 64 Mi, with nothing tying the two, so a lower one in redaction would bring back refused redactions with no test going red. Both now read pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS. The engine test adds an image mask, the other kind that shares, so a redaction that stopped storing either kind again fails it. doc_optimize's description, the README, the parity note and ROADMAP say which images share (named directly). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: a JBIG2 group is coded before its samples would pass the limit CodeRabbit on #5910: hold added an image to the group and then checked SHARED_BYTES, so a group could pass the 64 MiB it is documented to stay within by its last image, which can itself be up to 64 MiB (an image that shares has at most 2^26 pixels, and as its rows are whole bytes, at most that many bytes; a 1-pixel-wide one takes them all). hold_in codes the group first when an image would take it past the limit, and after when it has 16 images; with room for 10 bytes, images of 4 go two to a group, where the old order made one of 12. CodeRabbit's follow-up too: the 18-image test decodes every image and compares it with its samples, across the boundary between the groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * optimize: count the globals' object framing; filters: test the shared coder's size limit - A JBIG2 group shares a dictionary only when it saves more than the globals' dictionary and data plus NEW_OBJECT (96 bytes: the object's framing, its xref entry, and an encrypted file's IV and padding), so a group that saves less than that no longer makes the file larger. - encode_jbig2_shared is Unsupported for an image taller than the decoder takes and codes one at the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filters: drop the unused rows binding in encode_shared page() no longer takes the rows since #5511. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filters: the shared coder's at-bound test says what it returns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
This commit is contained in:
Piohrun
co-authored by
Claude Opus 5.5
FredApps
Claude Opus 5.5
FredApps
parent
e989e48e63
commit
d579f26de3
@@ -89,7 +89,15 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
alone when JBIG2 is asked for, is stored again as the compression asks when that is smaller (as JBIG2 symbols where
|
||||
its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact
|
||||
repeats and often comes out larger, and then stays as it is. A JBIG2 image stored again refers to no
|
||||
`/JBIG2Globals`, and one whose page isn't its size (PdfCraft's
|
||||
`/JBIG2Globals` of its own. With JBIG2 asked for, the images stored as JBIG2, up to 16 at a
|
||||
time, then share one symbol dictionary in a new `/JBIG2Globals` stream, holding the marks found
|
||||
in more than one of them: those that come out smaller that way, when together they save more
|
||||
than it takes. Only images that redaction can store again without it share one: image masks
|
||||
and DeviceGray images named directly, of at most 64 Mi pixels (redaction's limit,
|
||||
`pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS`); others are coded alone.
|
||||
Redacting one of them stores the others of its group again without it, so the group keeps that
|
||||
to 15 images, and each image's decode reads the dictionary of 16 at most. A
|
||||
JBIG2 image whose page isn't its size (PdfCraft's
|
||||
renderer draws the whole page into the image) is left alone. With Retain, a resampled image
|
||||
keeps its kind of compression (CCITT, JBIG2, or Flate) and nothing else changes; an image of
|
||||
more bits is then left to the grey or colour settings. Every change is kept only when it is
|
||||
@@ -103,7 +111,7 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
is past the 80 million pixels above).
|
||||
|
||||
- Lossless JBIG2 uses bounded generic-region encoding or exact-copy symbol dictionaries when
|
||||
smaller. No lossy symbol substitution is used. Export, redaction and this step read JBIG2
|
||||
smaller, a dictionary shared by up to 16 images when that is smaller still. No lossy symbol substitution is used. Export, redaction and this step read JBIG2
|
||||
images, with bounded shared globals. All of a run's images share one allowance of decoding
|
||||
(`pdfcraft_cos::ImageBudget`), filters, decoders and pixels alike, JPEG samples and merged
|
||||
pictures too: about
|
||||
|
||||
@@ -1117,12 +1117,23 @@ fn downsample_bits_checked(data: &[u8], w: u32, h: u32, nw: u32, nh: u32, dark_i
|
||||
/// to be resampled, and so is JBIG2 alone when Group 4 is asked for; Group 3, Group 4 inside
|
||||
/// another filter, and JBIG2 otherwise are stored again like any other. With `Retain`, a resampled image keeps its kind of compression (CCITT, JBIG2, or
|
||||
/// Flate), and nothing else changes: an image of more bits stays as it is, for the grey or
|
||||
/// colour settings. A JBIG2 image stored again refers to no `/JBIG2Globals`.
|
||||
/// colour settings. A JBIG2 image stored again refers to no `/JBIG2Globals`; with `Jbig2`, [`share`]
|
||||
/// may then have a group of them share one.
|
||||
///
|
||||
/// The samples keep their meaning, so the colour space, decode array and `/Mask` stay, but for
|
||||
/// black-and-white RGB, which becomes grey with one decode pair, and an indexed image's default
|
||||
/// decode array, which goes as the default at one bit.
|
||||
fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSettings, budget: &mut ImageBudget) -> Option<(Stream, bool)> {
|
||||
///
|
||||
/// Stored as JBIG2 with `Jbig2`, it comes with the 1-bit samples it was coded from, for [`share`]
|
||||
/// to code again with the run's other JBIG2 images.
|
||||
fn mono(
|
||||
doc: &Document,
|
||||
s: &Stream,
|
||||
ppi: f64,
|
||||
defaults: bool,
|
||||
settings: &MonoSettings,
|
||||
budget: &mut ImageBudget,
|
||||
) -> Option<(Stream, bool, Option<Vec<u8>>)> {
|
||||
let d = &s.dict;
|
||||
let downsample = settings.downsample && ppi.is_finite() && ppi > settings.above_ppi && settings.target_ppi > 0.0 && !d.contains(b"SMask");
|
||||
let retain = settings.compression == MonoCompression::Retain;
|
||||
@@ -1206,7 +1217,106 @@ fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSet
|
||||
let flate = ((resampled || packed_from_more) && !(retain && (ccitt || jbig2))).then(|| Stream::flate(dict, &data));
|
||||
// Smaller as stored, counting the dictionary: a tiny image can lose by Group 4's parameters.
|
||||
let best = [g4, jbig, flate].into_iter().flatten().filter_map(|c| Some((crate::stored_len(&c)?, c))).min_by_key(|(len, _)| *len)?;
|
||||
(best.0 < crate::stored_len(s)?).then_some((best.1, resampled))
|
||||
let shareable = settings.compression == MonoCompression::Jbig2 && best.1.dict.name(b"Filter") == Some(b"JBIG2Decode");
|
||||
(best.0 < crate::stored_len(s)?).then(|| (best.1, resampled, shareable.then_some(data)))
|
||||
}
|
||||
|
||||
/// The most images that share one symbol dictionary ([`share`]). Redaction stores again every
|
||||
/// image that shares a redacted one's dictionary, and each image's decode reads all of it, so a
|
||||
/// group keeps both to that many; most of what sharing saves comes with the first few images.
|
||||
const SHARED_IMAGES: usize = 16;
|
||||
|
||||
/// The most bytes of 1-bit samples a group holds: a group is coded before it would pass them
|
||||
/// ([`hold_in`]). An image that shares has at most [`SHARED_PIXELS`], and as its rows are whole
|
||||
/// bytes, at most that many bytes, which is this: one always fits a group with nothing in it.
|
||||
const SHARED_BYTES: usize = 64 << 20;
|
||||
|
||||
/// What the globals cost as an object of their own besides their dictionary and data: `N 0 obj`,
|
||||
/// `stream`, `endstream` and `endobj`, a cross-reference entry, and an encrypted file's IV and
|
||||
/// padding. A group is shared only when it saves more than that as well.
|
||||
const NEW_OBJECT: usize = 96;
|
||||
|
||||
/// Adds `image` to `group`, coding the group (`share`, which empties it) first when the image's
|
||||
/// samples would take it past `cap` bytes, and after when it has [`SHARED_IMAGES`].
|
||||
fn hold_in(group: &mut Vec<Held>, image: Held, cap: usize, mut share: impl FnMut(&mut Vec<Held>)) {
|
||||
if group.iter().map(|h| h.bits.len()).fold(image.bits.len(), usize::saturating_add) > cap {
|
||||
share(group);
|
||||
}
|
||||
group.push(image);
|
||||
if group.len() == SHARED_IMAGES {
|
||||
share(group);
|
||||
}
|
||||
}
|
||||
|
||||
/// The most pixels an image sharing a dictionary has: redaction's own limit on the images it
|
||||
/// stores again (see [`shares`]).
|
||||
const SHARED_PIXELS: u64 = pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS;
|
||||
|
||||
/// Whether an image stored as JBIG2 (its dictionary `d`) may share a dictionary: only one that
|
||||
/// redaction can store again without it, an image mask or a DeviceGray image, named directly, of
|
||||
/// at most [`SHARED_PIXELS`]. Redacting one image of a group stores the others so, as the
|
||||
/// dictionary can hold the shapes of marks under the area, and refuses the redaction for one it
|
||||
/// can't; an image of another colour space, though redaction may take it, is coded alone.
|
||||
fn shares(d: &Dict) -> bool {
|
||||
let side = |key: &[u8]| d.int(key).and_then(|v| u64::try_from(v).ok());
|
||||
let within = side(b"Width").zip(side(b"Height")).is_some_and(|(w, h)| w.saturating_mul(h) <= SHARED_PIXELS);
|
||||
within && (matches!(d.get(b"ImageMask"), Some(Object::Bool(true))) || d.name(b"ColorSpace") == Some(b"DeviceGray"))
|
||||
}
|
||||
|
||||
/// An image the run stored as JBIG2, held to be stored again sharing a dictionary ([`share`]): its
|
||||
/// 1-bit samples, `width` × `height`.
|
||||
struct Held {
|
||||
image: ObjRef,
|
||||
bits: Vec<u8>,
|
||||
width: u32,
|
||||
height: u32,
|
||||
}
|
||||
|
||||
/// Stores the images of `group`, each JBIG2 already, again sharing one symbol dictionary in a
|
||||
/// `/JBIG2Globals` stream ([`pdfcraft_filters::encode_jbig2_shared`]): those that come out smaller
|
||||
/// that way, if together they save more than the globals take. The others stay as they are, and
|
||||
/// so do all of them when that saves nothing. `report` counts the bytes as they end up, the globals
|
||||
/// included.
|
||||
fn share(doc: &mut Document, group: &mut Vec<Held>, report: &mut Report) {
|
||||
let held = std::mem::take(group);
|
||||
if held.len() < 2 {
|
||||
return;
|
||||
}
|
||||
let images: Vec<(&[u8], u32, u32)> = held.iter().map(|h| (&h.bits[..], h.width, h.height)).collect();
|
||||
let Ok(Some(shared)) = pdfcraft_filters::encode_jbig2_shared(&images) else { return };
|
||||
let with_globals = |s: &Stream, coded: Vec<u8>, globals: ObjRef| {
|
||||
let mut dict = s.dict.clone();
|
||||
let mut parms = Dict::new();
|
||||
parms.set(b"JBIG2Globals".to_vec(), Object::Ref(globals));
|
||||
dict.set(b"DecodeParms".to_vec(), Object::Dict(parms));
|
||||
Stream::from_raw(dict, coded)
|
||||
};
|
||||
// Weighed naming the globals with the longest reference there is: the real one is no longer.
|
||||
let longest = ObjRef { num: u32::MAX, generation: u16::MAX };
|
||||
let (mut chosen, mut saved) = (Vec::new(), 0usize);
|
||||
for (h, coded) in held.iter().zip(shared.images) {
|
||||
let Some(coded) = coded else { continue };
|
||||
let object = doc.get(h.image);
|
||||
let Object::Stream(alone) = &*object else { continue };
|
||||
let shorter = crate::stored_len(alone).zip(crate::stored_len(&with_globals(alone, coded.clone(), longest))).filter(|(a, b)| b < a);
|
||||
if let Some((before, after)) = shorter {
|
||||
saved = saved.saturating_add(before - after);
|
||||
chosen.push((h.image, coded));
|
||||
}
|
||||
}
|
||||
let globals = Stream::from_raw(Dict::new(), shared.globals);
|
||||
if chosen.is_empty() || crate::stored_len(&globals).is_none_or(|takes| saved <= takes.saturating_add(NEW_OBJECT)) {
|
||||
return;
|
||||
}
|
||||
report.image_bytes_after = report.image_bytes_after.saturating_add(globals.raw.len());
|
||||
let globals = doc.add(Object::Stream(globals));
|
||||
for (image, coded) in chosen {
|
||||
let object = doc.get(image);
|
||||
let Object::Stream(alone) = &*object else { continue };
|
||||
report.image_bytes_after = report.image_bytes_after.saturating_sub(alone.raw.len()).saturating_add(coded.len());
|
||||
let new = with_globals(alone, coded, globals);
|
||||
doc.set(image, Object::Stream(new));
|
||||
}
|
||||
}
|
||||
|
||||
/// One pass over the document's images, within `budget`. An image past it stays as it is, and is
|
||||
@@ -1239,6 +1349,13 @@ pub(crate) fn run(
|
||||
refs.sort_by_key(|(r, _)| r.num);
|
||||
let total = refs.len();
|
||||
let mut jbig2 = false;
|
||||
let mut group: Vec<Held> = Vec::new();
|
||||
// Holds a JBIG2 image for `share`, which codes the group once it is full.
|
||||
let mut hold = |doc: &mut Document, image: ObjRef, size: (Option<i64>, Option<i64>), bits: Option<Vec<u8>>, report: &mut Report| {
|
||||
let side = |v: Option<i64>| v.and_then(|v| u32::try_from(v).ok());
|
||||
let (Some(bits), Some(width), Some(height)) = (bits, side(size.0), side(size.1)) else { return };
|
||||
hold_in(&mut group, Held { image, bits, width, height }, SHARED_BYTES, |group| share(doc, group, report));
|
||||
};
|
||||
for (done, (r, ppi)) in refs.into_iter().enumerate() {
|
||||
if !progress(Stage::Images { done, total }) {
|
||||
return Err(OptimizeError::Cancelled);
|
||||
@@ -1254,7 +1371,7 @@ pub(crate) fn run(
|
||||
Some(m) if packed_from_more => (Some(m), None),
|
||||
m => (None, m),
|
||||
};
|
||||
if let Some((new, resampled)) = mono {
|
||||
if let Some((new, resampled, bits)) = mono {
|
||||
report.image_bytes_before += s.raw.len();
|
||||
report.image_bytes_after += new.raw.len();
|
||||
if resampled {
|
||||
@@ -1263,7 +1380,10 @@ pub(crate) fn run(
|
||||
report.images_recompressed += 1;
|
||||
}
|
||||
jbig2 |= new.dict.name(b"Filter") == Some(b"JBIG2Decode");
|
||||
let size = (new.dict.int(b"Width"), new.dict.int(b"Height"));
|
||||
let bits = bits.filter(|_| shares(&new.dict));
|
||||
doc.set(r, Object::Stream(new));
|
||||
hold(doc, r, size, bits, report);
|
||||
continue;
|
||||
}
|
||||
let n = s.dict.get(b"ColorSpace").and_then(|cs| components(doc, cs));
|
||||
@@ -1278,10 +1398,10 @@ pub(crate) fn run(
|
||||
}),
|
||||
};
|
||||
let total = |new: &Stream, smask: Option<&(ObjRef, Stream)>| Some(crate::stored_len(new)?.saturating_add(smask_len(smask).unwrap_or(0)));
|
||||
let packed_wins = packed.as_ref().is_some_and(|(p, _)| {
|
||||
let packed_wins = packed.as_ref().is_some_and(|(p, _, _)| {
|
||||
processed.as_ref().is_none_or(|(new, smask)| total(p, None).zip(total(new, smask.as_ref())).is_some_and(|(a, b)| a <= b))
|
||||
});
|
||||
if let Some((new, resampled)) = packed.filter(|_| packed_wins) {
|
||||
if let Some((new, resampled, bits)) = packed.filter(|_| packed_wins) {
|
||||
jbig2 |= new.dict.name(b"Filter") == Some(b"JBIG2Decode");
|
||||
report.image_bytes_before += s.raw.len();
|
||||
report.image_bytes_after += new.raw.len();
|
||||
@@ -1290,7 +1410,10 @@ pub(crate) fn run(
|
||||
} else {
|
||||
report.images_recompressed += 1;
|
||||
}
|
||||
let size = (new.dict.int(b"Width"), new.dict.int(b"Height"));
|
||||
let bits = bits.filter(|_| shares(&new.dict));
|
||||
doc.set(r, Object::Stream(new));
|
||||
hold(doc, r, size, bits, report);
|
||||
continue;
|
||||
}
|
||||
let Some((new, smask)) = processed else {
|
||||
@@ -1311,6 +1434,7 @@ pub(crate) fn run(
|
||||
doc.set(mr, Object::Stream(m));
|
||||
}
|
||||
}
|
||||
share(doc, &mut group, report);
|
||||
if !progress(Stage::Images { done: total, total }) {
|
||||
return Err(OptimizeError::Cancelled);
|
||||
}
|
||||
@@ -1338,6 +1462,58 @@ fn at_least_pdf_1_4(doc: &mut Document) -> Result<(), OptimizeError> {
|
||||
mod dimension_tests {
|
||||
use super::*;
|
||||
|
||||
/// A group is coded before an image's samples would take it past its limit, and once it has
|
||||
/// [`SHARED_IMAGES`]: with room for 10 bytes, images of 4 go two to a group, never three, and
|
||||
/// with room for 8, two of 4 fill one exactly; one of exactly the limit joins a group with
|
||||
/// nothing in it.
|
||||
#[test]
|
||||
fn a_group_is_coded_before_its_samples_would_pass_the_limit() {
|
||||
let held = |bytes: usize| Held { image: ObjRef { num: 1, generation: 0 }, bits: vec![0; bytes], width: 8, height: 8 };
|
||||
let mut coded: Vec<Vec<usize>> = Vec::new();
|
||||
let mut group = Vec::new();
|
||||
let mut code = |g: &mut Vec<Held>| coded.push(std::mem::take(g).iter().map(|h| h.bits.len()).collect());
|
||||
for _ in 0..5 {
|
||||
hold_in(&mut group, held(4), 10, &mut code);
|
||||
}
|
||||
hold_in(&mut group, held(10), 10, &mut code);
|
||||
assert_eq!(coded, [vec![4, 4], vec![4, 4], vec![4]]);
|
||||
assert_eq!(group.iter().map(|h| h.bits.len()).collect::<Vec<_>>(), [10]);
|
||||
// Exactly the limit is within it: with room for 8, two images of 4 stay together.
|
||||
let (mut count, mut group) = (0, Vec::new());
|
||||
let mut code = |g: &mut Vec<Held>| count += std::mem::take(g).len();
|
||||
hold_in(&mut group, held(4), 8, &mut code);
|
||||
hold_in(&mut group, held(4), 8, &mut code);
|
||||
assert_eq!((count, group.len()), (0, 2));
|
||||
let (mut count, mut group) = (0, Vec::new());
|
||||
for _ in 0..SHARED_IMAGES {
|
||||
hold_in(&mut group, held(1), usize::MAX, |g| count += std::mem::take(g).len());
|
||||
}
|
||||
assert_eq!((count, group.len()), (SHARED_IMAGES, 0), "coded on the 16th");
|
||||
}
|
||||
|
||||
/// Only what redaction can store again without the dictionary shares one: an image mask, or
|
||||
/// DeviceGray named directly, of at most 2^26 pixels.
|
||||
#[test]
|
||||
fn only_what_redaction_can_store_again_shares_a_dictionary() {
|
||||
let dict = |entries: Vec<(&str, Object)>| {
|
||||
let mut d = Dict::new();
|
||||
for (k, v) in entries {
|
||||
d.set(k.as_bytes().to_vec(), v);
|
||||
}
|
||||
d
|
||||
};
|
||||
let sized = |w: i64, h: i64, cs: Object| dict(vec![("Width", Object::Int(w)), ("Height", Object::Int(h)), ("ColorSpace", cs)]);
|
||||
assert!(shares(&sized(8192, 8192, Object::name("DeviceGray"))));
|
||||
assert!(!shares(&sized(8192, 8193, Object::name("DeviceGray"))), "a row past 2^26 pixels");
|
||||
assert!(shares(&dict(vec![("Width", Object::Int(8)), ("Height", Object::Int(8)), ("ImageMask", Object::Bool(true))])));
|
||||
let device_n =
|
||||
Object::Array(vec![Object::name("DeviceN"), Object::Array(vec![Object::name("Black")]), Object::name("DeviceGray"), Object::Null]);
|
||||
assert!(!shares(&sized(8, 8, device_n)));
|
||||
assert!(!shares(&sized(8, 8, Object::name("CalGray"))));
|
||||
assert!(!shares(&sized(8, 8, Object::Ref(ObjRef { num: 3, generation: 0 }))), "a colour space by reference");
|
||||
assert!(!shares(&dict(vec![("ColorSpace", Object::name("DeviceGray"))])), "no size");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pixel_reader_rejects_hostile_dimensions_before_decoding() {
|
||||
for (width, height, channels) in [(-1, 1, 1), (0, 1, 1), (i64::from(u32::MAX) + 101, 1, 1), (80_000_001, 1, 1), (2, 2, usize::MAX)] {
|
||||
|
||||
@@ -1951,8 +1951,17 @@ fn black_and_white_images_become_jbig2() {
|
||||
assert!(r.image_bytes_after < as_g4.image_bytes_after, "JBIG2 {} bytes, Group 4 {}", r.image_bytes_after, as_g4.image_bytes_after);
|
||||
let images: Vec<Stream> = image_refs(&jbig2).into_iter().map(|i| stream(&jbig2, i)).collect();
|
||||
assert_eq!(images.len(), 3);
|
||||
// The same text three times: the grey image and the mask share one symbol dictionary, and the
|
||||
// indexed image, which only those share, is coded alone.
|
||||
let globals_of = |s: &Stream| s.dict.get(b"DecodeParms").and_then(Object::as_dict).map(|d| (d.len(), d.reference(b"JBIG2Globals")));
|
||||
let is_indexed = |s: &Stream| s.dict.get(b"ColorSpace").and_then(Object::as_array).is_some();
|
||||
let shared: Vec<_> = images.iter().filter(|s| !is_indexed(s)).map(globals_of).collect();
|
||||
assert!(shared.len() == 2 && shared[0].is_some_and(|(n, g)| n == 1 && g.is_some()) && shared[0] == shared[1], "{shared:?}");
|
||||
for s in &images {
|
||||
assert_eq!((s.dict.name(b"Filter"), s.dict.contains(b"DecodeParms")), (Some(&b"JBIG2Decode"[..]), false));
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(&b"JBIG2Decode"[..]));
|
||||
if is_indexed(s) {
|
||||
assert!(!s.dict.contains(b"DecodeParms"));
|
||||
}
|
||||
}
|
||||
assert!(images.iter().any(|s| s.dict.get(b"Decode") == Some(&decode)), "the decode array stays");
|
||||
let indexed_kept =
|
||||
@@ -1999,6 +2008,11 @@ fn a_catalog_version_given_by_reference_is_read() {
|
||||
/// `#` a 3 × 3 block, three columns apart with a wider gap between words, every copy of a letter
|
||||
/// the same to the pixel.
|
||||
fn rendered_text(w: u32, h: u32) -> Vec<u8> {
|
||||
rendered_text_from(w, h, 0x2545_F491_4F6C_DD1D)
|
||||
}
|
||||
|
||||
/// [`rendered_text`], its letters picked from `seed`, which isn't 0.
|
||||
fn rendered_text_from(w: u32, h: u32, mut seed: u64) -> Vec<u8> {
|
||||
const LETTERS: [&[&str]; 5] = [
|
||||
&["##", "##", "##", "##", "##", "##", "##", "##", "##", "##"],
|
||||
&["......", ".####.", "##..##", "##..##", "##..##", "##..##", ".####."],
|
||||
@@ -2008,7 +2022,6 @@ fn rendered_text(w: u32, h: u32) -> Vec<u8> {
|
||||
];
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut data = vec![0xFF; stride * h as usize];
|
||||
let mut seed = 0x2545_F491_4F6C_DD1Du64;
|
||||
let mut next = move || {
|
||||
seed ^= seed << 13;
|
||||
seed ^= seed >> 7;
|
||||
@@ -2053,9 +2066,194 @@ fn rendered_text_becomes_jbig2_symbols() {
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(&b"JBIG2Decode"[..]));
|
||||
let generic = pdfcraft_filters::encode_jbig2_generic(&text, w, h).unwrap();
|
||||
assert!(s.raw.len() * 2 < generic.len(), "symbols {} bytes, generic {}", s.raw.len(), generic.len());
|
||||
assert!(!s.dict.contains(b"DecodeParms"), "alone, it shares no dictionary");
|
||||
assert!(render(&doc) == before, "the page looks the same");
|
||||
}
|
||||
|
||||
/// Page `index` of `doc`, saved and opened again, rendered at scale 1.
|
||||
fn render_page(doc: &Document, index: usize) -> Vec<u8> {
|
||||
let bytes = write_full(doc, &SaveOptions::default()).unwrap();
|
||||
let mut r = pdfcraft_render::PageRenderer::new(Arc::new(bytes), pdfcraft_render::RenderConfig::default());
|
||||
let out = r.render(pdfcraft_render::RenderRequest { page: index, scale: 1.0, ..Default::default() });
|
||||
assert!(out.error.is_none(), "{:?}", out.error);
|
||||
out.rgba.to_vec()
|
||||
}
|
||||
|
||||
/// Text in a font of 26 letters, about 10 pt at 300 dpi: each letter 32 rows of a run from a stem
|
||||
/// down its left side, of 2 to 24 pixels, every copy the same to the pixel; lines 44 rows apart,
|
||||
/// the letters picked from `seed`, which isn't 0.
|
||||
fn font_text(w: u32, h: u32, mut seed: u64) -> Vec<u8> {
|
||||
let mut font_seed = 0x2545_F491_4F6C_DD1Du64;
|
||||
let next = |seed: &mut u64| {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
*seed
|
||||
};
|
||||
let letters: Vec<Vec<usize>> = (0..26).map(|_| (0..16).map(|_| 1 + (next(&mut font_seed) % 12) as usize).collect()).collect();
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut data = vec![0xFF; stride * h as usize];
|
||||
for top in (8..(h as usize).saturating_sub(40)).step_by(44) {
|
||||
let mut x = 8;
|
||||
while x + 32 < w as usize {
|
||||
let letter = &letters[(next(&mut seed) % 26) as usize];
|
||||
for (row, run) in letter.iter().enumerate() {
|
||||
for y in [top + 2 * row, top + 2 * row + 1] {
|
||||
for px in x..x + 2 * run {
|
||||
data[y * stride + px / 8] &= !(0x80 >> (px % 8));
|
||||
}
|
||||
}
|
||||
}
|
||||
x += 2 * letter.iter().max().unwrap() + 4 + if next(&mut seed) % 6 == 0 { 14 } else { 0 };
|
||||
}
|
||||
}
|
||||
data
|
||||
}
|
||||
|
||||
/// The `/JBIG2Globals` an image names in its `/DecodeParms`.
|
||||
fn globals_of(doc: &Document, image: ObjRef) -> Option<ObjRef> {
|
||||
stream(doc, image).dict.get(b"DecodeParms")?.as_dict()?.reference(b"JBIG2Globals")
|
||||
}
|
||||
|
||||
/// Pages of text in one font, each letter the same to the pixel on every page, stored as JBIG2
|
||||
/// share one symbol dictionary (`/JBIG2Globals`): together they take less than each page coded
|
||||
/// alone, and every page looks the same, the last one, whose crosses no other page has, placing
|
||||
/// them from a dictionary of its own beside the shared one. The report counts the bytes they
|
||||
/// take, the globals once.
|
||||
#[test]
|
||||
fn jbig2_pages_share_one_symbol_dictionary() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (600u32, 480u32);
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let pages: Vec<Vec<u8>> = (1..=4u64)
|
||||
.map(|k| {
|
||||
let mut text = font_text(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15));
|
||||
if k == 4 {
|
||||
// Crosses down the right margin, three pixels a stroke.
|
||||
for top in (40..440).step_by(40) {
|
||||
for (dx, dy) in (0..9).map(|i| (i, 4)).chain((0..9).map(|i| (4, i))) {
|
||||
let (x, y) = (580 + dx, top + dy);
|
||||
text[y * stride + x / 8] &= !(0x80 >> (x % 8));
|
||||
}
|
||||
}
|
||||
}
|
||||
text
|
||||
})
|
||||
.collect();
|
||||
let images: Vec<ObjRef> = pages
|
||||
.iter()
|
||||
.map(|text| {
|
||||
let im = mono_image(&mut doc, w, h, text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", im)], "q 600 0 0 480 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let before: Vec<Vec<u8>> = (0..pages.len()).map(|p| render_page(&doc, p)).collect();
|
||||
let alone: usize = pages.iter().map(|text| pdfcraft_filters::encode_jbig2(text, w, h).unwrap().len()).sum();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
let report = optimize(&mut doc, &settings).unwrap();
|
||||
let globals = globals_of(&doc, images[0]).expect("the first page names the globals");
|
||||
assert!(images.iter().all(|im| globals_of(&doc, *im) == Some(globals)), "every page names the same globals");
|
||||
let shared = stream(&doc, globals).raw.len() + images.iter().map(|im| stream(&doc, *im).raw.len()).sum::<usize>();
|
||||
assert!(shared * 4 < alone * 3, "shared {shared} bytes, alone {alone}");
|
||||
assert_eq!(report.image_bytes_after, shared);
|
||||
for (p, before) in before.iter().enumerate() {
|
||||
assert!(render_page(&doc, p) == *before, "page {p} looks the same");
|
||||
let s = stream(&doc, images[p]);
|
||||
assert_eq!(decoded_image(&doc, &s).unwrap(), pages[p], "page {p} decodes to its samples");
|
||||
}
|
||||
// Optimized again, each page would come out larger alone than it is beside the globals: the
|
||||
// sharing stays.
|
||||
let streams: Vec<Stream> = images.iter().map(|im| stream(&doc, *im)).collect();
|
||||
let again = optimize(&mut doc, &settings).unwrap();
|
||||
assert_eq!((again.images_recompressed, again.images_resampled), (0, 0));
|
||||
assert!(images.iter().zip(&streams).all(|(im, s)| stream(&doc, *im) == *s));
|
||||
}
|
||||
|
||||
/// Images share a dictionary only where that saves more than it takes: two pages of five small
|
||||
/// letters save less than their globals would take, and stay as they are coded alone.
|
||||
#[test]
|
||||
fn a_dictionary_that_saves_less_than_it_takes_isnt_shared() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (600u32, 480u32);
|
||||
let images: Vec<ObjRef> = (1..=2u64)
|
||||
.map(|k| {
|
||||
let text = rendered_text_from(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15));
|
||||
let im = mono_image(&mut doc, w, h, &text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", im)], "q 600 0 0 480 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
let report = optimize(&mut doc, &settings).unwrap();
|
||||
assert_eq!(report.images_recompressed, 2);
|
||||
for im in &images {
|
||||
let s = stream(&doc, *im);
|
||||
assert_eq!((s.dict.name(b"Filter"), s.dict.contains(b"DecodeParms")), (Some(&b"JBIG2Decode"[..]), false));
|
||||
}
|
||||
// Optimized again, they code to what they are, no smaller: they stay.
|
||||
let streams: Vec<Stream> = images.iter().map(|im| stream(&doc, *im)).collect();
|
||||
assert_eq!(optimize(&mut doc, &settings).unwrap().images_recompressed, 0);
|
||||
assert!(images.iter().zip(&streams).all(|(im, s)| stream(&doc, *im) == *s));
|
||||
}
|
||||
|
||||
/// Only JBIG2 asked for shares: under Retain, JBIG2 pages drawn at 600 ppi are averaged and stored
|
||||
/// as JBIG2 again, each alone, as Retain changes nothing but their resolution.
|
||||
#[test]
|
||||
fn retained_jbig2_shares_nothing() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (1200u32, 960u32);
|
||||
let images: Vec<ObjRef> = (1..=4u64)
|
||||
.map(|k| {
|
||||
let text = font_text(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15));
|
||||
let mut d = Dict::new();
|
||||
for (key, value) in [("Type", Object::name("XObject")), ("Subtype", Object::name("Image")), ("ColorSpace", Object::name("DeviceGray"))] {
|
||||
d.set(key.as_bytes().to_vec(), value);
|
||||
}
|
||||
for (key, value) in [("Width", i64::from(w)), ("Height", i64::from(h)), ("BitsPerComponent", 1)] {
|
||||
d.set(key.as_bytes().to_vec(), Object::Int(value));
|
||||
}
|
||||
d.set(b"Filter".to_vec(), Object::name("JBIG2Decode"));
|
||||
let im = doc.add(Object::Stream(Stream::from_raw(d, pdfcraft_filters::encode_jbig2(&text, w, h).unwrap())));
|
||||
// 1200 samples over 144 pt: 600 ppi.
|
||||
page(&mut doc, &[("Im", im)], "q 144 0 0 115.2 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Retain, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
assert_eq!(optimize(&mut doc, &settings).unwrap().images_resampled, 4);
|
||||
for im in images {
|
||||
let s = stream(&doc, im);
|
||||
assert_eq!((s.dict.name(b"Filter"), s.dict.contains(b"DecodeParms")), (Some(&b"JBIG2Decode"[..]), false));
|
||||
}
|
||||
}
|
||||
|
||||
/// Sixteen images at most share one dictionary, and two are enough: of eighteen, the first
|
||||
/// sixteen share one and the other two another.
|
||||
#[test]
|
||||
fn sixteen_images_at_most_share_a_dictionary() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (600u32, 480u32);
|
||||
let texts: Vec<Vec<u8>> = (1..=18u64).map(|k| font_text(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15))).collect();
|
||||
let images: Vec<ObjRef> = texts
|
||||
.iter()
|
||||
.map(|text| {
|
||||
let im = mono_image(&mut doc, w, h, text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", im)], "q 600 0 0 480 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
optimize(&mut doc, &settings).unwrap();
|
||||
let globals: Vec<Option<ObjRef>> = images.iter().map(|im| globals_of(&doc, *im)).collect();
|
||||
assert!(globals.iter().all(Option::is_some), "{globals:?}");
|
||||
assert!(globals[..16].iter().all(|g| *g == globals[0]) && globals[16..].iter().all(|g| *g == globals[16]), "{globals:?}");
|
||||
assert_ne!(globals[0], globals[16]);
|
||||
for (k, (im, text)) in images.iter().zip(&texts).enumerate() {
|
||||
assert!(decoded_image(&doc, &stream(&doc, *im)).unwrap() == *text, "image {k} decodes to its samples");
|
||||
}
|
||||
}
|
||||
|
||||
/// A JBIG2 scan goes through Monochrome Images like any black-and-white image. Above 450 ppi it
|
||||
/// is averaged as its Flate copy is, to the same samples, and stored as each compression asks:
|
||||
/// Group 4 or JBIG2, or Flate when that is smaller, or under Retain JBIG2 again; alone or behind another
|
||||
|
||||
Reference in New Issue
Block a user