mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-02 22:41:47 -06:00
M2.6: JBIG2 decoding in pdfcraft-filters; Document::decoded_image reads JBIG2 images
decode_jbig2 sits next to decode_ccitt, through hayro-jbig2 as Export ▸ Images used it, and shares CCITT's row sink. The JBIG2 page has to be the image's size: PdfCraft's renderer draws the whole page into the image (black box: a 4-row page behind /Height 2 fills the image box with all four rows), so the rows of a taller page aren't the image's. The decoder decodes a page whole, so the page is checked before it is decoded: that size, within the caller's bound as rows of one bit, and at most 2^28 pixels (MAX_JBIG2_PIXELS; the decoder keeps a bit a pixel, about 34 MB there). Document::decoded_image_within decodes a JBIG2 image as it does a fax image: the filters before it undone first, then the decoder with the /JBIG2Globals of the image's own /DecodeParms entry (/Crypt counts in the position; fax_params and the globals now share last_filter_parms). Globals have a limit of their own, MAX_JBIG2_GLOBALS (32 MiB, moved from Export), and globals that can't be read are an error rather than ignored. Export ▸ Images reads JBIG2 through it, so its own decoder, row sink and DecodeParms reader go, and hayro-jbig2 leaves pdfcraft-create. JBIG2 behind another filter now exports, and so copies (Copy Image uses the same function); a page taller than the image, which Export cut to the image's rows, now skips the image, as one of another width did. doc_export_all_images' description no longer lists JBIG2 inside other filters as skipped. The optimizer's monochrome step read images through decoded_image_within and relied on JBIG2 failing there: it now leaves JBIG2 images alone explicitly, as before (a resampled one would come out Flate under Retain), and the README and parity note list them as not yet handled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Grzegorz Martens
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
c0e3c6bec5
commit
ea6c329731
Generated
-1
@@ -4336,7 +4336,6 @@ version = "0.15.0"
|
||||
dependencies = [
|
||||
"encoding_rs",
|
||||
"hayro-ccitt",
|
||||
"hayro-jbig2",
|
||||
"hayro-syntax",
|
||||
"image",
|
||||
"pdfcraft-cos",
|
||||
|
||||
@@ -1281,7 +1281,7 @@ pub fn tools() -> Vec<ToolDef> {
|
||||
.ro()
|
||||
.cmd("ocr.recognize")
|
||||
.with(schema(json!({}), &[])),
|
||||
t("doc_export_all_images", "Export all images", "Write the images that pages use into a folder (`<name>_Page_<n>_Image_<k>.jpg|png`, n zero-padded to the page count's digits), each once: JPEG images unchanged, others as PNG with their soft mask as alpha. min_size skips images with fewer pixels on their shorter side. Images that can't be decoded yet (JPEG 2000, JBIG2 inside other filters, separations) are listed under skipped. Includes unsaved edits.")
|
||||
t("doc_export_all_images", "Export all images", "Write the images that pages use into a folder (`<name>_Page_<n>_Image_<k>.jpg|png`, n zero-padded to the page count's digits), each once: JPEG images unchanged, others as PNG with their soft mask as alpha. min_size skips images with fewer pixels on their shorter side. Images that can't be decoded yet (JPEG 2000, separations) are listed under skipped. Includes unsaved edits.")
|
||||
.cmd("export.all_images")
|
||||
.with(schema(
|
||||
json!({
|
||||
|
||||
@@ -90,6 +90,6 @@ recognisable in a diff and byte-stable when re-serialized.
|
||||
- Linearized output, Fast Web View (`core.linearization`, M11).
|
||||
- Structural validation against the Arlington model (`core.arlington-validation`), which will live
|
||||
in the reserved `arlington` crate.
|
||||
- Image codecs. `pdfcraft-filters` carries the non-image filters and CCITT fax decoding
|
||||
(`Document::decoded_image`); DCT, JPX and JBIG2 are still decoded through the bootstrap renderer
|
||||
(`core.image-filters`, M2.6).
|
||||
- Image codecs. `pdfcraft-filters` carries the non-image filters and CCITT fax and JBIG2 decoding
|
||||
(`Document::decoded_image`, a JBIG2 image's `/JBIG2Globals` within `MAX_JBIG2_GLOBALS`, 32 MiB);
|
||||
DCT and JPX are still decoded through the bootstrap renderer (`core.image-filters`, M2.6).
|
||||
|
||||
@@ -10,7 +10,7 @@ use std::collections::{BTreeMap, HashMap, HashSet};
|
||||
use std::sync::{Arc, Mutex, Weak};
|
||||
|
||||
use crate::CosError;
|
||||
use crate::object::{Dict, MAX_DECODED, ObjRef, Object, Stream};
|
||||
use crate::object::{Dict, MAX_DECODED, MAX_JBIG2_GLOBALS, ObjRef, Object, Stream};
|
||||
use crate::parser::{EndstreamIndex, Lexer, is_whitespace, parse_indirect, parse_indirect_shared};
|
||||
|
||||
mod objects;
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
use super::reader::find;
|
||||
use super::state::{LENGTHS, LoadGuard, ObjStm, ObjectReader, ReadState, Slot, SourceIdentity, objstm_member, owned_bytes};
|
||||
use super::xref::{Revision, XrefEntry};
|
||||
use super::{Arc, CosError, Cow, Dict, Document, Lexer, MAX_DECODED, ObjRef, Object, Stream, is_whitespace, parse_indirect_shared};
|
||||
use super::{
|
||||
Arc, CosError, Cow, Dict, Document, Lexer, MAX_DECODED, MAX_JBIG2_GLOBALS, ObjRef, Object, Stream, is_whitespace, parse_indirect_shared,
|
||||
};
|
||||
|
||||
impl Document {
|
||||
pub fn bytes(&self) -> &Arc<Vec<u8>> {
|
||||
@@ -233,18 +235,7 @@ impl Document {
|
||||
if !matches!(chain.last(), Some((pdfcraft_filters::Filter::CcittFax, _))) {
|
||||
return None;
|
||||
}
|
||||
// Its place in /Filter, /Crypt included, pairs it with its /DecodeParms entry (§7.4.1): the
|
||||
// last filter but a /Crypt, which `filters` leaves out wherever it stands.
|
||||
let index = match s.dict.get(b"Filter") {
|
||||
Some(Object::Array(a)) => a.iter().rposition(|f| f.as_name() != Some(b"Crypt".as_slice())).unwrap_or(0),
|
||||
_ => 0,
|
||||
};
|
||||
let parms = match s.dict.get(b"DecodeParms").map(|p| self.resolve(p)).as_deref() {
|
||||
Some(Object::Array(a)) => a.get(index).and_then(|p| self.dict(p)),
|
||||
Some(p) if index == 0 => p.as_dict().cloned(),
|
||||
_ => None,
|
||||
}
|
||||
.unwrap_or_default();
|
||||
let parms = self.last_filter_parms(s);
|
||||
let int = |key: &[u8], default: i64| parms.get(key).and_then(|v| self.resolve(v).as_int()).unwrap_or(default);
|
||||
let flag = |key: &[u8], default: bool| match parms.get(key).map(|v| self.resolve(v)).as_deref() {
|
||||
Some(Object::Bool(b)) => *b,
|
||||
@@ -262,23 +253,62 @@ impl Document {
|
||||
})
|
||||
}
|
||||
|
||||
/// The `/DecodeParms` dictionary of stream `s`'s last filter, resolved: that of a lone filter,
|
||||
/// or its entry in an array. Empty when there is none.
|
||||
fn last_filter_parms(&self, s: &Stream) -> Dict {
|
||||
// Its place in /Filter, /Crypt included, pairs it with its /DecodeParms entry (§7.4.1): the
|
||||
// last filter but a /Crypt, which `filters` leaves out wherever it stands.
|
||||
let index = match s.dict.get(b"Filter") {
|
||||
Some(Object::Array(a)) => a.iter().rposition(|f| f.as_name() != Some(b"Crypt".as_slice())).unwrap_or(0),
|
||||
_ => 0,
|
||||
};
|
||||
match s.dict.get(b"DecodeParms").map(|p| self.resolve(p)).as_deref() {
|
||||
Some(Object::Array(a)) => a.get(index).and_then(|p| self.dict(p)),
|
||||
Some(p) if index == 0 => p.as_dict().cloned(),
|
||||
_ => None,
|
||||
}
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// The samples of image `s`, decoded: through its filters ([`Stream::decoded`]), or, for a
|
||||
/// CCITT fax image (its last filter), through the filters before it and then
|
||||
/// [`pdfcraft_filters::decode_ccitt`] with its [`Document::fax_params`], as `CCITTFaxDecode`
|
||||
/// gives them (ISO 32000-2 §7.4.6). Parameters given by reference count, for every filter
|
||||
/// ([`Document::with_direct_parms`]). A fax image's `/Columns` has to be its `/Width`. Up to
|
||||
/// its `/Height` rows are decoded, whatever `/Rows` says, within [`MAX_DECODED`] bytes; damaged
|
||||
/// data gives fewer, so a caller checks it has the rows it needs.
|
||||
/// CCITT fax or JBIG2 image (its last filter), through the filters before it and then
|
||||
/// [`pdfcraft_filters::decode_ccitt`] with its [`Document::fax_params`], or
|
||||
/// [`pdfcraft_filters::decode_jbig2`] with its `/JBIG2Globals`, as `CCITTFaxDecode` and
|
||||
/// `JBIG2Decode` give them (ISO 32000-2 §7.4.6, §7.4.7). Parameters given by reference count,
|
||||
/// for every filter ([`Document::with_direct_parms`]). A fax image's `/Columns` has to be its
|
||||
/// `/Width`, and a JBIG2 image's page its `/Width` × `/Height`: PdfCraft's renderer draws the
|
||||
/// whole page into the image. Up to its `/Height` rows are decoded, whatever a fax image's `/Rows` says,
|
||||
/// within [`MAX_DECODED`] bytes; damaged fax data gives fewer, so a caller checks it has the
|
||||
/// rows it needs. Damaged JBIG2 data, and globals that can't be read (within
|
||||
/// [`MAX_JBIG2_GLOBALS`]), are an error.
|
||||
pub fn decoded_image(&self, s: &Stream) -> Result<Vec<u8>, CosError> {
|
||||
self.decoded_image_within(s, MAX_DECODED)
|
||||
}
|
||||
|
||||
/// [`Document::decoded_image`] for an image whose decoding has a tighter bound than
|
||||
/// [`MAX_DECODED`]: no stage, the filters before a fax decoder included, may produce more than
|
||||
/// `max` bytes.
|
||||
/// [`MAX_DECODED`]: no stage, the filters before a fax or JBIG2 decoder included, may produce
|
||||
/// more than `max` bytes. A JBIG2 image's `/JBIG2Globals` are another stream, shared between
|
||||
/// images, and are decoded within [`MAX_JBIG2_GLOBALS`] whatever `max` is.
|
||||
pub fn decoded_image_within(&self, s: &Stream, max: usize) -> Result<Vec<u8>, CosError> {
|
||||
let max = max.min(MAX_DECODED);
|
||||
let s = &*self.with_direct_parms(s);
|
||||
let filter = |e: pdfcraft_filters::FilterError| CosError::Filter(e.to_string());
|
||||
let chain = s.readable_filters().map_err(filter)?;
|
||||
// The filters before the image codec, such as ASCII85Decode, are undone first.
|
||||
let undo = |before: &[(pdfcraft_filters::Filter, pdfcraft_filters::Params)]| match before {
|
||||
[] => Ok(Cow::Borrowed(&s.raw[..])),
|
||||
_ => pdfcraft_filters::decode_tolerant(before, &s.raw, max).map(|(data, _)| Cow::Owned(data)).map_err(filter),
|
||||
};
|
||||
// `/Width` and `/Height` are read as given directly, as every caller reads them first: one
|
||||
// given by reference has the image refused there, before it gets here.
|
||||
let height = s.dict.int(b"Height").and_then(|h| u32::try_from(h).ok()).unwrap_or(0);
|
||||
if let Some(((pdfcraft_filters::Filter::Jbig2, _), before)) = chain.split_last() {
|
||||
let Some(columns) = s.dict.int(b"Width").and_then(|w| u32::try_from(w).ok()) else {
|
||||
return Err(CosError::Filter("a JBIG2 image's /Width isn't a size".into()));
|
||||
};
|
||||
let globals = self.jbig2_globals(s)?;
|
||||
return pdfcraft_filters::decode_jbig2(&undo(before)?, globals.as_deref(), columns, height, max).map_err(filter);
|
||||
}
|
||||
let Some(p) = self.fax_params(s) else { return s.decoded_within(max) };
|
||||
// As many rows as the image has: a /Rows below /Height would cut the picture short, and
|
||||
// rows past /Height aren't drawn.
|
||||
@@ -286,18 +316,25 @@ impl Document {
|
||||
if s.dict.int(b"Width") != Some(p.columns) {
|
||||
return Err(CosError::Filter("a fax image's /Columns aren't its /Width".into()));
|
||||
}
|
||||
let height = s.dict.int(b"Height").and_then(|h| u32::try_from(h).ok()).unwrap_or(0);
|
||||
let stride = usize::try_from(p.columns).unwrap_or(usize::MAX).div_ceil(8).max(1);
|
||||
let rows = u32::try_from(max / stride).unwrap_or(u32::MAX).min(height);
|
||||
// The filters before it, such as ASCII85Decode, are undone first.
|
||||
let chain = s.readable_filters().map_err(|e| CosError::Filter(e.to_string()))?;
|
||||
let data = match chain.split_last() {
|
||||
Some((_, before)) if !before.is_empty() => {
|
||||
Cow::Owned(pdfcraft_filters::decode_tolerant(before, &s.raw, max).map_err(|e| CosError::Filter(e.to_string()))?.0)
|
||||
}
|
||||
_ => Cow::Borrowed(&s.raw[..]),
|
||||
};
|
||||
pdfcraft_filters::decode_ccitt(&data, &p, rows).map_err(|e| CosError::Filter(e.to_string()))
|
||||
let data = undo(chain.split_last().map_or(&[][..], |(_, before)| before))?;
|
||||
pdfcraft_filters::decode_ccitt(&data, &p, rows).map_err(filter)
|
||||
}
|
||||
|
||||
/// What the `/JBIG2Globals` stream of JBIG2 image `s` (in its last filter's `/DecodeParms`)
|
||||
/// decodes to, within [`MAX_JBIG2_GLOBALS`]; `None` without one. Globals that can't be read
|
||||
/// are an error, rather than the image decoded without them.
|
||||
fn jbig2_globals(&self, s: &Stream) -> Result<Option<Vec<u8>>, CosError> {
|
||||
let globals = self.last_filter_parms(s).get(b"JBIG2Globals").map(|g| self.resolve(g));
|
||||
let Some(Object::Stream(g)) = globals.as_deref() else { return Ok(None) };
|
||||
self.with_direct_parms(g).decoded_within(MAX_JBIG2_GLOBALS).map(Some).map_err(|e| {
|
||||
let detail = match e {
|
||||
CosError::Filter(detail) => detail,
|
||||
e => e.to_string(),
|
||||
};
|
||||
CosError::Filter(format!("the JBIG2 globals can't be read: {detail}"))
|
||||
})
|
||||
}
|
||||
|
||||
pub fn try_get(&self, num: u32) -> Result<Arc<Object>, CosError> {
|
||||
|
||||
@@ -173,6 +173,64 @@ fn fax_images_decode_with_their_parameters() {
|
||||
assert!(behind(&[Object::Int(5), fax], &[null, own], coded.clone()).is_err(), "a /Filter entry that isn't a name");
|
||||
}
|
||||
|
||||
/// A JBIG2 image reads through its decoder, behind other filters too, with the `/JBIG2Globals` of
|
||||
/// its own `/DecodeParms` (a dictionary or its place in an array, /Crypt counting, by reference
|
||||
/// too): globals that can't be read are an error, and ones at another filter's place aren't its
|
||||
/// own, nor are those of a lone dictionary when other filters come first. Its page has to be its
|
||||
/// /Width × /Height, and fit the bound.
|
||||
#[test]
|
||||
fn jbig2_images_decode_with_their_globals() {
|
||||
// 10 × 3: the first five samples of each row white, the rest black.
|
||||
let bits = [0b1111_1000, 0, 0b1111_1000, 0, 0b1111_1000, 0];
|
||||
let coded = pdfcraft_filters::encode_jbig2_generic(&bits, 10, 3).unwrap();
|
||||
let hex = coded.iter().map(|b| format!("{b:02x}")).chain([">".to_string()]).collect::<String>().into_bytes();
|
||||
let mut doc = Document::new_empty();
|
||||
// An extension segment (ISO/IEC 14492 7.4.14), which a decoder skips, Flate-encoded; and data
|
||||
// that isn't JBIG2.
|
||||
let globals = |doc: &mut Document, data: &[u8]| {
|
||||
let mut d = Dict::new();
|
||||
d.set(b"Filter".to_vec(), Object::name("FlateDecode"));
|
||||
let globals = doc.add(Object::Stream(Stream::from_raw(d, pdfcraft_filters::encode_flate(data))));
|
||||
let mut parms = Dict::new();
|
||||
parms.set(b"JBIG2Globals".to_vec(), Object::Ref(globals));
|
||||
Object::Dict(parms)
|
||||
};
|
||||
let (good, bad) = (globals(&mut doc, &[0, 0, 0, 0, 0x3E, 0, 0, 0, 0, 0, 4, 0, 0, 0, 1]), globals(&mut doc, b"not JBIG2"));
|
||||
let indirect = Object::Ref(doc.add(bad.clone()));
|
||||
let image = |filters: &[&str], parms: Option<Object>, data: &[u8], (width, height): (i64, i64), max: usize| {
|
||||
let mut d = Dict::new();
|
||||
d.set(b"Width".to_vec(), Object::Int(width));
|
||||
d.set(b"Height".to_vec(), Object::Int(height));
|
||||
d.set(b"Filter".to_vec(), Object::Array(filters.iter().map(|f| Object::name(f)).collect()));
|
||||
if let Some(p) = parms {
|
||||
d.set(b"DecodeParms".to_vec(), p);
|
||||
}
|
||||
doc.decoded_image_within(&Stream::from_raw(d, data.to_vec()), max)
|
||||
};
|
||||
let jbig2 = ["JBIG2Decode"];
|
||||
let full = (10, 3);
|
||||
assert_eq!(image(&jbig2, None, &coded, full, MAX_DECODED).unwrap(), bits);
|
||||
assert_eq!(image(&jbig2, Some(good.clone()), &coded, full, MAX_DECODED).unwrap(), bits, "globals it skips");
|
||||
for parms in [bad.clone(), Object::Array(vec![bad.clone()]), indirect.clone(), Object::Array(vec![indirect])] {
|
||||
let e = image(&jbig2, Some(parms.clone()), &coded, full, MAX_DECODED).unwrap_err();
|
||||
assert!(e.to_string().contains("JBIG2"), "{parms:?}: {e}");
|
||||
}
|
||||
let behind = ["ASCIIHexDecode", "JBIG2Decode"];
|
||||
assert_eq!(image(&behind, None, &hex, full, MAX_DECODED).unwrap(), bits, "behind ASCIIHexDecode");
|
||||
assert!(image(&behind, Some(Object::Array(vec![Object::Null, bad.clone()])), &hex, full, MAX_DECODED).is_err(), "its own globals");
|
||||
assert_eq!(image(&behind, Some(Object::Array(vec![bad.clone(), Object::Null])), &hex, full, MAX_DECODED).unwrap(), bits, "another's");
|
||||
assert_eq!(image(&behind, Some(bad.clone()), &hex, full, MAX_DECODED).unwrap(), bits, "a lone dictionary, the first filter's");
|
||||
let after_crypt = ["Crypt", "JBIG2Decode"];
|
||||
assert!(image(&after_crypt, Some(Object::Array(vec![Object::Null, bad.clone()])), &coded, full, MAX_DECODED).is_err(), "after /Crypt");
|
||||
assert_eq!(image(&after_crypt, Some(Object::Array(vec![bad, Object::Null])), &coded, full, MAX_DECODED).unwrap(), bits, "/Crypt's");
|
||||
|
||||
for size in [(11, 3), (9, 3), (10, 2), (10, 4)] {
|
||||
assert!(image(&jbig2, None, &coded, size, MAX_DECODED).is_err(), "a page other than {size:?}");
|
||||
}
|
||||
assert_eq!(image(&jbig2, None, &coded, full, 6).unwrap(), bits, "the page within the bound");
|
||||
assert!(image(&jbig2, None, &coded, full, 5).is_err(), "the page past the bound");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_file_id_is_derived_from_the_encryption_seed() {
|
||||
let mut doc = Document::new_empty();
|
||||
|
||||
@@ -25,7 +25,7 @@ mod writer;
|
||||
pub use budget::{DecodeBudget, MAX_PAGE_DECODED, MAX_PAGE_FORMS};
|
||||
pub use bytes::Bytes;
|
||||
pub use document::{Document, ReadState, Revision, SourceIdentity, XrefEntry};
|
||||
pub use object::{Dict, MAX_DECODED, Name, ObjRef, Object, PdfString, Stream};
|
||||
pub use object::{Dict, MAX_DECODED, MAX_JBIG2_GLOBALS, Name, ObjRef, Object, PdfString, Stream};
|
||||
pub use parser::{Lexer, is_whitespace, parse_indirect};
|
||||
pub use pdfcraft_crypt::{
|
||||
Algorithm, Auth, Method as CryptMethod, NewEncryption, Permissions, SecurityHandler, effective_password, pdfdoc_byte, pdfdoc_char,
|
||||
|
||||
@@ -213,6 +213,12 @@ pub struct Stream {
|
||||
/// Upper bound for a single decoded stream (decompression-bomb defence).
|
||||
pub const MAX_DECODED: usize = 1 << 30;
|
||||
|
||||
/// The most an image's `/JBIG2Globals` stream may decode to: the symbol and pattern dictionaries an
|
||||
/// encoder shares between pages, themselves coded, so far less than any stream may decode to
|
||||
/// ([`MAX_DECODED`]). They are decoded before the image's JBIG2 page is read, so the page's size
|
||||
/// can't bound them: they have a limit of their own.
|
||||
pub const MAX_JBIG2_GLOBALS: usize = 32 << 20;
|
||||
|
||||
impl Stream {
|
||||
/// A new stream from already-encoded bytes (`/Length` is set at write time).
|
||||
pub fn from_raw(dict: Dict, raw: Vec<u8>) -> Self {
|
||||
|
||||
@@ -15,8 +15,6 @@ pdfcraft-model = { workspace = true }
|
||||
pdfcraft-fonts = { workspace = true }
|
||||
# For `image::bmp_with_pixel_offset`, shared with XFA pictures.
|
||||
pdfcraft-xfa = { workspace = true }
|
||||
# Already in the graph through hayro: decodes JBIG2 images for Export ▸ Images.
|
||||
hayro-jbig2.workspace = true
|
||||
png = "0.18"
|
||||
tiff = "0.11"
|
||||
image = { version = "0.25", default-features = false, features = ["bmp", "gif", "jpeg"] }
|
||||
|
||||
@@ -27,11 +27,12 @@ Layer L4. Create a PDF (Acrobat's Create a PDF tool), execution plan M10.2:
|
||||
Every function returns a `pdfcraft_cos::Document`; the caller writes it.
|
||||
|
||||
Image export and Save Image As apply each lossless image's Decode pair before converting its color components.
|
||||
JBIG2 images and masks (JBIG2 as their only filter) are decoded with `hayro-jbig2`, their `/JBIG2Globals` included,
|
||||
as `JBIG2Decode` gives them: black as 0. An image whose JBIG2 page has another width than the image, or more
|
||||
pixels than an exported image may have, or whose globals decode to more than 32 MiB (they are decoded before the
|
||||
page's size is known, so they have a limit of their own), is skipped; such a mask is left out and the image
|
||||
exported opaque, as JPX and CCITT masks are.
|
||||
JBIG2 images and masks are decoded through `pdfcraft_cos::Document::decoded_image` (`pdfcraft_filters::decode_jbig2`), behind
|
||||
other filters too, with the `/JBIG2Globals` of their own `/DecodeParms` entry, as `JBIG2Decode` gives them: black as 0. An image
|
||||
whose JBIG2 page isn't its size (PdfCraft's renderer draws the whole page into the image; the page is refused before it is decoded),
|
||||
or whose globals can't be read or decode to more than 32 MiB (they are decoded before the page's size is known, so they have a
|
||||
limit of their own), is skipped; such a mask is left out and the image exported opaque, as JPX and CCITT
|
||||
masks are.
|
||||
Indexed Decode values select palette entries after rounding and clipping. Color-key masks compare the original samples.
|
||||
Missing, null or malformed Decode arrays keep the default mapping.
|
||||
Soft-mask Decode intervals map samples to opacity before the mask is resized, including reversed ranges and indirect entries.
|
||||
@@ -43,4 +44,4 @@ CCITT fax images (Group 3 and 4) are decoded for export through `pdfcraft_cos::D
|
||||
`ASCII85Decode` too. Byte-aligned Group 3 rows are read by their end-of-line codes, whether the fill starts each code on a byte or
|
||||
ends it on one, when the data starts with one (T.4 sends one before the first row), and on bytes otherwise, whatever `/EndOfLine`
|
||||
says; data that comes up short is reported. Data with codes but none before the first row is read on bytes, and is reported or,
|
||||
at `/K 0`, may come out wrong. JPEG 2000 and JBIG2 images are still reported, not exported.
|
||||
at `/K 0`, may come out wrong. JPEG 2000 images are still reported, not exported.
|
||||
|
||||
@@ -100,114 +100,14 @@ fn filters(s: &Stream) -> Vec<Vec<u8>> {
|
||||
}
|
||||
}
|
||||
|
||||
/// The samples of image `s`, decoded: through its filters, with a `/DecodeParms` dictionary given
|
||||
/// by reference, or as an array's entry, or with numbers inside by reference, read too (a predictor
|
||||
/// given so would otherwise be lost), or, for a JBIG2 image
|
||||
/// (its only filter), through a JBIG2 decoder with its `/JBIG2Globals`, as rows of one-bit samples
|
||||
/// each starting on a byte, with black as 0, as `JBIG2Decode` gives them (ISO 32000-2 §7.4.7).
|
||||
/// The samples of image `s`, decoded ([`Document::decoded_image`]): through its filters, with a
|
||||
/// `/DecodeParms` dictionary given by reference, or as an array's entry, or with numbers inside by
|
||||
/// reference, read too (a predictor given so would otherwise be lost); a CCITT fax or JBIG2 image
|
||||
/// through its decoder too, behind other filters as well, a JBIG2 image with its `/JBIG2Globals`.
|
||||
/// A JBIG2 page has to be the image's size, which the caller checked against the export's limit,
|
||||
/// before the page is decoded.
|
||||
fn decoded(doc: &Document, s: &Stream) -> Result<Vec<u8>, String> {
|
||||
match filters(s).as_slice() {
|
||||
[f] if f == b"JBIG2Decode" => jbig2(doc, s),
|
||||
_ => doc.decoded_image(s).map_err(|e| e.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// The `/DecodeParms` of a stream with one filter: a dictionary, or the one in an array (7.4.2).
|
||||
fn decode_parms(doc: &Document, s: &Stream) -> Dict {
|
||||
let Some(p) = s.dict.get(b"DecodeParms").map(|p| doc.resolve(p)) else { return Dict::default() };
|
||||
let p = match &*p {
|
||||
Object::Array(a) => match a.first() {
|
||||
Some(first) => doc.resolve(first),
|
||||
None => return Dict::default(),
|
||||
},
|
||||
_ => p,
|
||||
};
|
||||
p.as_dict().cloned().unwrap_or_default()
|
||||
}
|
||||
|
||||
/// The most an image's `/JBIG2Globals` decode to: the symbol and pattern dictionaries an encoder
|
||||
/// shares between pages, themselves coded, so far less than any stream may decode to
|
||||
/// ([`pdfcraft_cos::MAX_DECODED`], 1 GiB). Globals past it refuse the image.
|
||||
const MAX_GLOBALS: usize = 32 << 20;
|
||||
|
||||
/// [`decoded`] for a JBIG2 image. Its JBIG2 page has to be the image's width, which the caller
|
||||
/// checked against the export's size limit, and within that limit itself: the decoder decodes the
|
||||
/// whole page, rows past the image's height too, so a small `/Height` over a far taller page
|
||||
/// would cost more than the limit allows. Rows past the image's height are dropped, and too few
|
||||
/// rows are refused by the caller. Globals that can't be read refuse the image, rather than
|
||||
/// decoding it without them. They are decoded before the page's size is known, so the page's
|
||||
/// limit can't bound them; they have [`MAX_GLOBALS`] of their own.
|
||||
fn jbig2(doc: &Document, s: &Stream) -> Result<Vec<u8>, String> {
|
||||
let globals = match decode_parms(doc, s).get(b"JBIG2Globals").map(|g| doc.resolve(g)).as_deref() {
|
||||
Some(Object::Stream(g)) => {
|
||||
Some(doc.with_direct_parms(g).decoded_within(MAX_GLOBALS).map_err(|e| format!("the JBIG2 globals can't be read: {e}"))?)
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
let image = hayro_jbig2::Image::new_embedded(&s.raw, globals.as_deref()).map_err(|e| format!("the JBIG2 image can't be read: {e}"))?;
|
||||
if s.dict.int(b"Width") != Some(i64::from(image.width())) {
|
||||
return Err("the JBIG2 image's width isn't the image's width".into());
|
||||
}
|
||||
if u64::from(image.width()).saturating_mul(u64::from(image.height())) > MAX_PIXELS {
|
||||
return Err(format!("the JBIG2 page, {} × {}, is too large to export", image.width(), image.height()));
|
||||
}
|
||||
let height = s.dict.int(b"Height").and_then(|h| usize::try_from(h).ok()).unwrap_or(0);
|
||||
let mut rows = Rows::new(image.width() as usize, height);
|
||||
image.decode(&mut rows).map_err(|e| format!("the JBIG2 image can't be decoded: {e}"))?;
|
||||
Ok(rows.out)
|
||||
}
|
||||
|
||||
/// Rows of one-bit samples, each starting on a byte, filled from a decoder's pixels: `columns`
|
||||
/// a row and at most `rows` rows. Pixels past a row's end and rows past the last are dropped, so
|
||||
/// the output is bounded by the size the caller gave.
|
||||
struct Rows {
|
||||
out: Vec<u8>,
|
||||
row: Vec<u8>,
|
||||
x: usize,
|
||||
columns: usize,
|
||||
rows: usize,
|
||||
done: usize,
|
||||
}
|
||||
|
||||
impl Rows {
|
||||
fn new(columns: usize, rows: usize) -> Self {
|
||||
Rows { out: Vec::new(), row: vec![0; columns.div_ceil(8)], x: 0, columns, rows, done: 0 }
|
||||
}
|
||||
|
||||
/// `count` pixels, as 1 samples when `one`.
|
||||
fn push(&mut self, one: bool, count: usize) {
|
||||
let end = self.x.saturating_add(count);
|
||||
if one {
|
||||
for x in self.x..end.min(self.columns) {
|
||||
if let Some(b) = self.row.get_mut(x / 8) {
|
||||
*b |= 0x80 >> (x % 8);
|
||||
}
|
||||
}
|
||||
}
|
||||
self.x = end;
|
||||
}
|
||||
|
||||
fn end_row(&mut self) {
|
||||
if self.done < self.rows {
|
||||
self.out.extend_from_slice(&self.row);
|
||||
self.done += 1;
|
||||
}
|
||||
self.row.fill(0);
|
||||
self.x = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// JBIG2 codes black as 1, which `JBIG2Decode` gives as 0.
|
||||
impl hayro_jbig2::Decoder for Rows {
|
||||
fn push_pixel(&mut self, black: bool) {
|
||||
self.push(!black, 1);
|
||||
}
|
||||
fn push_pixel_chunk(&mut self, black: bool, chunks: u32) {
|
||||
self.push(!black, (chunks as usize).saturating_mul(8));
|
||||
}
|
||||
fn next_line(&mut self) {
|
||||
self.end_row();
|
||||
}
|
||||
doc.decoded_image(s).map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
/// The file for one image: its JPEG data as is, or a PNG. An image with transparency (a soft
|
||||
@@ -218,7 +118,6 @@ fn image(doc: &Document, s: &Stream) -> Result<(&'static str, Vec<u8>), String>
|
||||
let jpeg = matches!(f.last().map(Vec::as_slice), Some(b"DCTDecode" | b"DCT")) && f.len() == 1;
|
||||
match f.last().map(Vec::as_slice) {
|
||||
Some(b"JPXDecode") => return Err("JPEG 2000 images can't be exported yet".into()),
|
||||
Some(b"JBIG2Decode") if f.len() > 1 => return Err("JBIG2 images inside other filters can't be exported yet".into()),
|
||||
Some(b"DCTDecode" | b"DCT") if !jpeg => return Err("JPEG images inside other filters can't be exported yet".into()),
|
||||
_ => {}
|
||||
}
|
||||
|
||||
+45
-21
@@ -2054,10 +2054,10 @@ fn symbol_coded_jbig2_images_export_as_their_picture() {
|
||||
|
||||
/// Export ▸ Images reads JBIG2 images (ISO 32000-2 §7.4.7), JBIG2 black as 0: as the image, and as
|
||||
/// a stencil `/Mask` of another. Their `/JBIG2Globals` are read, from a dictionary or the one in
|
||||
/// an array, so globals that can't be read skip the image rather than being ignored. Rows past the
|
||||
/// image's height are left out; a JBIG2 page of another width, or with fewer rows than the image,
|
||||
/// or larger than an exported image may be (however small the image's `/Height`), or data that
|
||||
/// isn't JBIG2, skips the image.
|
||||
/// an array, so globals that can't be read skip the image rather than being ignored. A JBIG2 page
|
||||
/// of another size than the image (a taller one too: PdfCraft's renderer draws the whole page into
|
||||
/// the image), refused before it is decoded however large, or data that isn't JBIG2, skips the
|
||||
/// image.
|
||||
#[test]
|
||||
fn jbig2_images_export_as_their_picture() {
|
||||
let (w, h) = (203i64, 41i64);
|
||||
@@ -2118,33 +2118,24 @@ fn jbig2_images_export_as_their_picture() {
|
||||
for in_array in [false, true] {
|
||||
assert!(skipped(&page(w, h, &jbig2, Some((&b"not JBIG2"[..], in_array)), false)), "unreadable globals, in an array: {in_array}");
|
||||
}
|
||||
assert_eq!(exported_rgba(&page(w, h - 5, &jbig2, None, false)).1, picture(h - 5), "rows past the height");
|
||||
for (width, height, data) in [(w + 8, h, &jbig2[..]), (w - 1, h, &jbig2[..]), (w, h + 1, &jbig2[..]), (w, h, &b"not JBIG2"[..])] {
|
||||
for (width, height, data) in
|
||||
[(w + 8, h, &jbig2[..]), (w - 1, h, &jbig2[..]), (w, h + 1, &jbig2[..]), (w, h - 5, &jbig2[..]), (w, h, &b"not JBIG2"[..])]
|
||||
{
|
||||
assert!(skipped(&page(width, height, data, None, false)), "{width} × {height}, {} bytes", data.len());
|
||||
}
|
||||
// The same stream with its page information (the first segment's data, after its 11-byte
|
||||
// header: width, then height) claiming a 20,000 × 10,000 page: 200 million pixels, over the
|
||||
// export's 100 million, behind an image of 20,000 × 4. The decoder would decode the whole page
|
||||
// (it takes pages of up to about 2^28 pixels), so it is refused first.
|
||||
// header: width, then height) claiming a 20,000 × 10,000 page, behind an image of 20,000 × 4:
|
||||
// the decoder would decode the whole page, so it is refused first.
|
||||
let mut large = jbig2.clone();
|
||||
large[11..15].copy_from_slice(&20_000u32.to_be_bytes());
|
||||
large[15..19].copy_from_slice(&10_000u32.to_be_bytes());
|
||||
assert!(skipped(&page(20_000, 4, &large, None, false)), "a page over the export's size");
|
||||
assert!(skipped(&page(20_000, 4, &large, None, false)), "a page far taller than the image");
|
||||
// Refused the same ways as a stencil /Mask, it is left out: the image exports opaque.
|
||||
for (width, height, mask) in [(w + 8, h, &jbig2), (20_000, 4, &large)] {
|
||||
let rgba = exported_rgba(&page(width, height, mask, None, true)).1;
|
||||
assert_eq!(rgba.len() as i64, width * height * 4, "a {width}-wide mask");
|
||||
assert!(alphas(&rgba).iter().all(|a| *a == 255), "a refused {width}-wide mask is left out");
|
||||
}
|
||||
// A page of exactly the budget, 20,000 × 5,000, is decoded; one row more is refused.
|
||||
for (rows, refused) in [(5_000u32, false), (5_001, true)] {
|
||||
large[15..19].copy_from_slice(&rows.to_be_bytes());
|
||||
let doc = page(20_000, 4, &large, None, false);
|
||||
assert_eq!(skipped(&doc), refused, "a page of 20,000 × {rows}");
|
||||
if !refused {
|
||||
assert_eq!(exported_rgba(&doc).1.len(), 20_000 * 4 * 4);
|
||||
}
|
||||
}
|
||||
// Damaged data never panics.
|
||||
for end in [0, 1, 10, 20, 30, jbig2.len() / 2] {
|
||||
let _ = extract_images(&page(w, h, &jbig2[..end], None, false), &[0], 0);
|
||||
@@ -2155,6 +2146,38 @@ fn jbig2_images_export_as_their_picture() {
|
||||
assert_eq!(alphas(&exported_rgba(&page(w, h, &jbig2, None, true)).1), alpha);
|
||||
}
|
||||
|
||||
/// A JBIG2 image behind another filter (ASCIIHexDecode) exports as its picture, as a fax image does,
|
||||
/// with the `/JBIG2Globals` at its own place in a `/DecodeParms` array: globals that can't be read
|
||||
/// there skip the image, and ones at the other filter's place aren't the image's.
|
||||
#[test]
|
||||
fn jbig2_images_behind_another_filter_export_as_their_picture() {
|
||||
let bits = [0xF0u8, 0x0F, 0xAA, 0x55];
|
||||
let jbig2 = pdfcraft_filters::encode_jbig2_generic(&bits, 16, 2).unwrap();
|
||||
let hex: Vec<u8> = jbig2.iter().flat_map(|b| format!("{b:02x}").into_bytes()).chain(*b">").collect();
|
||||
let picture: Vec<u8> = (0..32).flat_map(|i| if bits[i / 8] & (0x80 >> (i % 8)) != 0 { [255; 4] } else { [0, 0, 0, 255] }).collect();
|
||||
// With unreadable globals at place `at` of its /DecodeParms.
|
||||
let page = |at: Option<usize>| {
|
||||
let mut doc = from_images(&[("flat.png".into(), png_bytes(false))]).unwrap();
|
||||
with_image(&mut doc, |doc| {
|
||||
let mut d = image_dict(16, 2, "DeviceGray", 1);
|
||||
d.set(b"Filter".to_vec(), Object::Array(vec![Object::name("ASCIIHexDecode"), Object::name("JBIG2Decode")]));
|
||||
if let Some(at) = at {
|
||||
let mut parms = Dict::new();
|
||||
parms.set(b"JBIG2Globals".to_vec(), Object::Ref(doc.add(Object::Stream(Stream::from_raw(Dict::new(), b"not JBIG2".to_vec())))));
|
||||
let mut entries = vec![Object::Null, Object::Null];
|
||||
entries[at] = Object::Dict(parms);
|
||||
d.set(b"DecodeParms".to_vec(), Object::Array(entries));
|
||||
}
|
||||
Stream::from_raw(d, hex.clone())
|
||||
});
|
||||
doc
|
||||
};
|
||||
assert_eq!(exported_rgba(&page(None)), ("png", picture.clone()));
|
||||
assert_eq!(exported_rgba(&page(Some(0))).1, picture, "globals at ASCIIHexDecode's place");
|
||||
let out = extract_images(&page(Some(1)), &[0], 0);
|
||||
assert!(out.images.is_empty() && out.skipped.len() == 1, "unreadable globals at its own place");
|
||||
}
|
||||
|
||||
/// A JBIG2 image's `/JBIG2Globals` are decoded before its JBIG2 page's size is known, so they have
|
||||
/// a limit of their own, 32 MiB, far below the 1 GiB of any stream: globals that Flate-decode to
|
||||
/// one byte more skip the image, with the limit in the reason, though they take a few kilobytes
|
||||
@@ -2162,7 +2185,8 @@ fn jbig2_images_export_as_their_picture() {
|
||||
/// indirect `/DecodeParms`, in an array or not.
|
||||
#[test]
|
||||
fn jbig2_globals_are_decoded_within_a_limit_of_their_own() {
|
||||
const LIMIT: usize = 32 << 20;
|
||||
const LIMIT: usize = pdfcraft_cos::MAX_JBIG2_GLOBALS;
|
||||
assert_eq!(LIMIT, 32 << 20);
|
||||
let bits = [0xF0u8, 0x0F, 0xAA, 0x55];
|
||||
let jbig2 = pdfcraft_filters::encode_jbig2_generic(&bits, 16, 2).unwrap();
|
||||
let picture: Vec<u8> = (0..32).flat_map(|i| if bits[i / 8] & (0x80 >> (i % 8)) != 0 { [255; 4] } else { [0, 0, 0, 255] }).collect();
|
||||
@@ -2206,6 +2230,6 @@ fn jbig2_globals_are_decoded_within_a_limit_of_their_own() {
|
||||
let out = extract_images(&page(&over, form), &[0], 0);
|
||||
assert!(out.images.is_empty(), "globals over the limit, form {form}");
|
||||
let why = &out.skipped.first().expect("the image is skipped").2;
|
||||
assert!(why.starts_with("the JBIG2 globals can't be read") && why.contains(&format!("limit of {LIMIT} bytes")), "{why}");
|
||||
assert!(why.contains("the JBIG2 globals can't be read") && why.contains(&format!("limit of {LIMIT} bytes")), "{why}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "pdfcraft-filters"
|
||||
description = "PDF stream filters (ISO 32000-2 §7.4): Flate, LZW, predictors, ASCII85, ASCIIHex, RunLength, CCITT Group 4 encoding and Group 3 and 4 decoding"
|
||||
description = "PDF stream filters (ISO 32000-2 §7.4): Flate, LZW, predictors, ASCII85, ASCIIHex, RunLength, CCITT Group 4 encoding and Group 3 and 4 decoding, JBIG2 encoding and decoding"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
@@ -12,13 +12,12 @@ workspace = true
|
||||
|
||||
[dependencies]
|
||||
fax.workspace = true
|
||||
# CCITT fax decoding (`decode_ccitt`); already in the tree through hayro.
|
||||
# CCITT fax and JBIG2 decoding (`decode_ccitt`, `decode_jbig2`); already in the tree through hayro.
|
||||
hayro-ccitt.workspace = true
|
||||
hayro-jbig2.workspace = true
|
||||
flate2.workspace = true
|
||||
thiserror.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
hayro-ccitt.workspace = true
|
||||
hayro-jbig2.workspace = true
|
||||
proptest.workspace = true
|
||||
weezl.workspace = true
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
PDF stream filters: the decoders and encoders behind `/Filter` and `/DecodeParms`.
|
||||
|
||||
- **Layer:** L0, standalone. No dependency on any other PdfCraft crate. `cos` depends on it (architecture §3 rule 3).
|
||||
- **Licence:** MIT OR Apache-2.0. Clean-room: written from ISO 32000-2 §7.4, RFC 1950/1951 (through `flate2` with the pure-Rust `miniz_oxide` backend), TIFF 6.0 §14 (predictor 2) and RFC 2083 §6 (PNG filters). CCITT Group 4 encoding (ITU-T T.6) is the `fax` crate's (MIT, already in the tree through `tiff`), checked here by round trips through an independent decoder; CCITT decoding (T.4 and T.6) is the `hayro-ccitt` crate's (MIT OR Apache-2.0, already in the tree through hayro), bounded here. JBIG2 generic region encoding is written from the public ISO/IEC 14492 Final Committee Draft (WG1 N1359, 1999); Annex H.2 checks the arithmetic coder byte for byte.
|
||||
- **Licence:** MIT OR Apache-2.0. Clean-room: written from ISO 32000-2 §7.4, RFC 1950/1951 (through `flate2` with the pure-Rust `miniz_oxide` backend), TIFF 6.0 §14 (predictor 2) and RFC 2083 §6 (PNG filters). CCITT Group 4 encoding (ITU-T T.6) is the `fax` crate's (MIT, already in the tree through `tiff`), checked here by round trips through an independent decoder; CCITT decoding (T.4 and T.6) is the `hayro-ccitt` crate's, and JBIG2 decoding the `hayro-jbig2` crate's (both MIT OR Apache-2.0, already in the tree through hayro), bounded here. JBIG2 generic region encoding is written from the public ISO/IEC 14492 Final Committee Draft (WG1 N1359, 1999); Annex H.2 checks the arithmetic coder byte for byte.
|
||||
|
||||
- `#![forbid(unsafe_code)]`. Builds for `wasm32-unknown-unknown`.
|
||||
|
||||
@@ -26,6 +26,7 @@ encode_flate(data) -> Vec<u8> // zlib, level 6
|
||||
encode_g4(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as CCITT Group 4; columns <= MAX_G4_COLUMNS
|
||||
pub struct CcittParams { k, columns, rows, end_of_line, encoded_byte_align, end_of_block, black_is_1 } // Table 11; Default = its defaults
|
||||
decode_ccitt(data, ¶ms, max_rows) -> Result<Vec<u8>, FilterError> // CCITT fax as CCITTFaxDecode gives it; columns <= MAX_CCITT_COLUMNS
|
||||
decode_jbig2(data, globals, columns, rows, max) -> Result<Vec<u8>, FilterError> // JBIG2 as JBIG2Decode gives it; a page of columns × rows, within max and MAX_JBIG2_PIXELS
|
||||
encode_jbig2_generic(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode; columns <= MAX_JBIG2_COLUMNS
|
||||
encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode, symbol coded when that is smaller; columns <= MAX_JBIG2_COLUMNS
|
||||
```
|
||||
@@ -47,7 +48,7 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
| ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` |
|
||||
| RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 |
|
||||
| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::Document::decoded_image` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt |
|
||||
| JBIG2Decode | not here (`Unsupported`; the renderer and image export decode it with `hayro-jbig2`) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, or more than 2^24 rows, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made |
|
||||
| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, about the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::Document::decoded_image` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, or more than 2^24 rows, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made |
|
||||
|
||||
## Tests
|
||||
|
||||
@@ -57,12 +58,13 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
- `ccitt.rs`: `decode_ccitt` gives back the rows `encode_g4` was given (with `/BlackIs1`, `/Rows` and a row limit), refuses bad widths, and bounds its output for any input; byte-aligned Group 3 in each layout, over 32 seeded pictures, decodes to its picture, and a byte short to fewer rows. Group 4 output decoded by `hayro-ccitt` (an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics.
|
||||
- `mq.rs`: the arithmetic coder reproduces Annex H.2's 30 bytes from its 256 decisions.
|
||||
- `jbig2.rs`: output decoded by `hayro-jbig2` (dev-dependency, an independent decoder) matches the input bits: a text-like page (smaller than Group 4), edges, padding, all-white and all-black regions (typical rows), and property tests over bitmaps of noise and of runs; any input returns rather than panics.
|
||||
- `jbig2/decode.rs`: `decode_jbig2` gives back the rows of both codings, padding bits 0; refuses a page of another width or height, past `max` or past `MAX_JBIG2_PIXELS` before decoding it (2^28 pixels take about 34 MB, a bit each), globals that aren't JBIG2 and damaged data, and returns rather than panics on every prefix of a stream.
|
||||
- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics.
|
||||
- `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`).
|
||||
|
||||
## Status
|
||||
|
||||
M1.1/M1.2 for the general-purpose filters are done. Not yet here: the image decoders (DCT via zune-jpeg, JPX via hayro-jpeg2000, JBIG2), DCT encoding, CCITT Group 3 encoding and Flate encode levels other than 6.
|
||||
M1.1/M1.2 for the general-purpose filters are done. Not yet here: the DCT and JPX decoders (DCT via zune-jpeg, JPX via hayro-jpeg2000), DCT encoding, CCITT Group 3 encoding and Flate encode levels other than 6.
|
||||
|
||||
|
||||
Lossless symbol coding is also implemented: exact-copy dictionaries/text regions, bounded by the encoder; no approximate or lossy symbol substitution.
|
||||
|
||||
+17
-11
@@ -132,9 +132,10 @@ fn starts_with_end_of_line(data: &[u8]) -> bool {
|
||||
}
|
||||
|
||||
/// Rows of 1-bit samples filled from a decoder's pixels: `columns` a row, each row on a new byte, at
|
||||
/// most `rows` rows. Pixels past a row's end and rows past the last are dropped.
|
||||
struct Rows {
|
||||
out: Vec<u8>,
|
||||
/// most `rows` rows. Pixels past a row's end and rows past the last are dropped. The JBIG2 decoder
|
||||
/// ([`crate::decode_jbig2`]) fills them too.
|
||||
pub(crate) struct Rows {
|
||||
pub(crate) out: Vec<u8>,
|
||||
row: Vec<u8>,
|
||||
x: usize,
|
||||
columns: usize,
|
||||
@@ -144,12 +145,12 @@ struct Rows {
|
||||
}
|
||||
|
||||
impl Rows {
|
||||
fn new(columns: usize, rows: usize, black_is_1: bool) -> Self {
|
||||
pub(crate) fn new(columns: usize, rows: usize, black_is_1: bool) -> Self {
|
||||
Rows { out: Vec::new(), row: vec![0; columns.div_ceil(8)], x: 0, columns, rows, done: 0, black_is_1 }
|
||||
}
|
||||
|
||||
/// `count` pixels, white or black.
|
||||
fn push(&mut self, white: bool, count: usize) {
|
||||
pub(crate) fn push(&mut self, white: bool, count: usize) {
|
||||
let end = self.x.saturating_add(count);
|
||||
if white != self.black_is_1 {
|
||||
for x in self.x..end.min(self.columns) {
|
||||
@@ -160,6 +161,16 @@ impl Rows {
|
||||
}
|
||||
self.x = end;
|
||||
}
|
||||
|
||||
/// The row filled so far kept, if fewer than `rows` are, and the next one begun.
|
||||
pub(crate) fn end_row(&mut self) {
|
||||
if self.done < self.rows {
|
||||
self.out.extend_from_slice(&self.row);
|
||||
self.done += 1;
|
||||
}
|
||||
self.row.fill(0);
|
||||
self.x = 0;
|
||||
}
|
||||
}
|
||||
|
||||
impl hayro_ccitt::Decoder for Rows {
|
||||
@@ -170,12 +181,7 @@ impl hayro_ccitt::Decoder for Rows {
|
||||
self.push(white, (chunks as usize).saturating_mul(8));
|
||||
}
|
||||
fn next_line(&mut self) {
|
||||
if self.done < self.rows {
|
||||
self.out.extend_from_slice(&self.row);
|
||||
self.done += 1;
|
||||
}
|
||||
self.row.fill(0);
|
||||
self.x = 0;
|
||||
self.end_row();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,10 +2,13 @@
|
||||
//! a generic region, template 0 with its nominal adaptive pixels, typical prediction on, MQ
|
||||
//! arithmetic coding ([`crate::mq`]); or the image's marks as symbols ([`symbols`]). Written from
|
||||
//! the standard's 6.2.5 (coding a generic region), 7.2 (segment headers), 7.4.1, 7.4.6 and 7.4.8
|
||||
//! (the segments) and D.3 (the embedded organisation PDF uses).
|
||||
//! (the segments) and D.3 (the embedded organisation PDF uses). Decoding is in [`decode`].
|
||||
|
||||
mod decode;
|
||||
mod symbols;
|
||||
|
||||
pub use decode::{MAX_JBIG2_PIXELS, decode_jbig2};
|
||||
|
||||
use crate::FilterError;
|
||||
use crate::mq::{Context, MqEncoder};
|
||||
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
//! JBIG2 decoding for `JBIG2Decode` (ISO 32000-2 §7.4.7), through `hayro-jbig2`.
|
||||
|
||||
use crate::FilterError;
|
||||
use crate::ccitt::Rows;
|
||||
|
||||
/// The largest JBIG2 page [`decode_jbig2`] decodes, in pixels. The decoder decodes a page whole,
|
||||
/// and takes pages of up to about this many (20,000 × 10,000 decodes).
|
||||
pub const MAX_JBIG2_PIXELS: u64 = 1 << 28;
|
||||
|
||||
/// A `JBIG2Decode` stream's `data`, with what its `/JBIG2Globals` stream decodes to, decoded as the
|
||||
/// filter gives it: `rows` rows of `columns` 1-bit samples, most significant bit first, each
|
||||
/// starting on a new byte, with black as 0 (JBIG2 codes black as 1).
|
||||
///
|
||||
/// The JBIG2 page has to be the image's size, `columns` × `rows`: PdfCraft's renderer draws the
|
||||
/// whole page into the image, so the rows of a page of another height wouldn't be the image's. The
|
||||
/// decoder decodes the page whole, so it is checked before it is decoded, and has to fit in `max`
|
||||
/// bytes as such rows and have at most [`MAX_JBIG2_PIXELS`]. Regions inside it are bounded by the
|
||||
/// decoder alone: a region may claim more pixels than its page, and is decoded before it is cut to
|
||||
/// the page. With no rows asked for, nothing is decoded.
|
||||
///
|
||||
/// Errors: `data` or `globals` that don't decode, or a page of another size
|
||||
/// ([`FilterError::Corrupt`]); a page past `max` ([`FilterError::LimitExceeded`]) or
|
||||
/// [`MAX_JBIG2_PIXELS`] ([`FilterError::Unsupported`]).
|
||||
pub fn decode_jbig2(data: &[u8], globals: Option<&[u8]>, columns: u32, rows: u32, max: usize) -> Result<Vec<u8>, FilterError> {
|
||||
if rows == 0 {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let corrupt = |detail: String| FilterError::Corrupt { filter: "JBIG2Decode", detail };
|
||||
let image = hayro_jbig2::Image::new_embedded(data, globals).map_err(|e| corrupt(e.to_string()))?;
|
||||
let (width, height) = (image.width(), image.height());
|
||||
if (width, height) != (columns, rows) {
|
||||
return Err(corrupt(format!("a page of {width} × {height} pixels for an image of {columns} × {rows}")));
|
||||
}
|
||||
if u64::from(width) * u64::from(height) > MAX_JBIG2_PIXELS {
|
||||
return Err(FilterError::Unsupported(format!("JBIG2 pages of {width} × {height} pixels")));
|
||||
}
|
||||
// u32 to usize: wasm32 and up have at least 32-bit usize.
|
||||
let (width, height) = (width as usize, height as usize);
|
||||
if width.div_ceil(8).checked_mul(height).is_none_or(|bytes| bytes > max) {
|
||||
return Err(FilterError::LimitExceeded(max));
|
||||
}
|
||||
let mut out = Rows::new(width, height, false);
|
||||
image.decode(&mut out).map_err(|e| corrupt(e.to_string()))?;
|
||||
Ok(out.out)
|
||||
}
|
||||
|
||||
/// JBIG2 codes black as 1, which `JBIG2Decode` gives as 0.
|
||||
impl hayro_jbig2::Decoder for Rows {
|
||||
fn push_pixel(&mut self, black: bool) {
|
||||
self.push(!black, 1);
|
||||
}
|
||||
fn push_pixel_chunk(&mut self, black: bool, chunks: u32) {
|
||||
self.push(!black, (chunks as usize).saturating_mul(8));
|
||||
}
|
||||
fn next_line(&mut self) {
|
||||
self.end_row();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::{encode_jbig2, encode_jbig2_generic};
|
||||
|
||||
/// `rows` rows of `columns` samples of noise, the bits that pad each row to a byte 0.
|
||||
fn noise(columns: u32, rows: u32) -> Vec<u8> {
|
||||
let stride = (columns as usize).div_ceil(8);
|
||||
let mut seed = 0x2545_F491_4F6C_DD1Du64;
|
||||
let mut data: Vec<u8> = (0..stride * rows as usize)
|
||||
.map(|_| {
|
||||
seed ^= seed << 13;
|
||||
seed ^= seed >> 7;
|
||||
seed ^= seed << 17;
|
||||
seed as u8
|
||||
})
|
||||
.collect();
|
||||
let pad = (stride * 8 - columns as usize) as u32;
|
||||
data.chunks_mut(stride).for_each(|row| *row.last_mut().unwrap() &= 0xFFu8.checked_shl(pad).unwrap_or(0));
|
||||
data
|
||||
}
|
||||
|
||||
/// The same stream with its page information (the first segment's data, after its 11-byte
|
||||
/// header: width, then height) claiming a page of `width` × `height`.
|
||||
fn page_of(data: &[u8], width: u32, height: u32) -> Vec<u8> {
|
||||
let mut data = data.to_vec();
|
||||
data[11..15].copy_from_slice(&width.to_be_bytes());
|
||||
data[15..19].copy_from_slice(&height.to_be_bytes());
|
||||
data
|
||||
}
|
||||
|
||||
/// Either coding decodes to its samples, black as 0, the bits that pad each row 0: a generic
|
||||
/// region of noise, and symbols for letters repeated across the page. With no rows asked for,
|
||||
/// nothing is decoded.
|
||||
#[test]
|
||||
fn both_codings_decode_to_their_samples() {
|
||||
let (w, h) = (203u32, 41u32);
|
||||
let bits = noise(w, h);
|
||||
let generic = encode_jbig2_generic(&bits, w, h).unwrap();
|
||||
assert_eq!(decode_jbig2(&generic, None, w, h, usize::MAX).unwrap(), bits);
|
||||
assert_eq!(decode_jbig2(b"not JBIG2", None, w, 0, usize::MAX).unwrap(), b"", "no rows asked for");
|
||||
|
||||
// Four lines of three letters, black on white.
|
||||
const LETTERS: [[&str; 7]; 3] = [
|
||||
["#####", "#....", "####.", "#....", "#....", "#....", "#####"],
|
||||
[".###.", "#...#", "#....", "#....", "#....", "#...#", ".###."],
|
||||
["#...#", "#...#", "#####", "#...#", "#...#", "#...#", "#...#"],
|
||||
];
|
||||
let (w, h) = (400u32, 40u32);
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut marks = vec![0xFFu8; stride * h as usize];
|
||||
for (line, top) in [2usize, 12, 22, 32].into_iter().enumerate() {
|
||||
for (k, x) in (3..w as usize - 6).step_by(7).enumerate() {
|
||||
for (dy, row) in LETTERS[(k + line) % 3].iter().enumerate() {
|
||||
for (dx, _) in row.bytes().enumerate().filter(|(_, c)| *c == b'#') {
|
||||
marks[(top + dy) * stride + (x + dx) / 8] &= !(0x80 >> ((x + dx) % 8));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let symbols = encode_jbig2(&marks, w, h).unwrap();
|
||||
assert!(symbols.len() < encode_jbig2_generic(&marks, w, h).unwrap().len(), "symbol coded");
|
||||
assert_eq!(decode_jbig2(&symbols, None, w, h, usize::MAX).unwrap(), marks);
|
||||
}
|
||||
|
||||
/// The page has to be the image's size, and is bounded before it is decoded, being decoded
|
||||
/// whole: within `max` as rows of one bit, and within [`MAX_JBIG2_PIXELS`]. Globals are given
|
||||
/// to the decoder: ones it skips decode, ones that aren't JBIG2 are an error, as is damaged
|
||||
/// data, which never panics.
|
||||
#[test]
|
||||
fn what_it_cannot_decode_is_an_error() {
|
||||
let (w, h) = (203u32, 41u32);
|
||||
let data = encode_jbig2_generic(&noise(w, h), w, h).unwrap();
|
||||
for (columns, rows) in [(w - 1, h), (w + 1, h), (w, h - 1), (w, h + 1)] {
|
||||
let e = decode_jbig2(&data, None, columns, rows, usize::MAX).unwrap_err();
|
||||
assert!(matches!(e, FilterError::Corrupt { .. }), "{columns} × {rows}: {e}");
|
||||
}
|
||||
let page = 26 * h as usize;
|
||||
assert!(matches!(decode_jbig2(&data, None, w, h, page - 1), Err(FilterError::LimitExceeded(m)) if m == page - 1), "a byte short");
|
||||
assert_eq!(decode_jbig2(&data, None, w, h, page).unwrap().len(), page, "the page within max");
|
||||
// 16,384 × 16,384 is 2^28 pixels, and fails only later; a row more is past the cap.
|
||||
let large = |rows| decode_jbig2(&page_of(&data, 16_384, rows), None, 16_384, rows, usize::MAX);
|
||||
assert!(matches!(large(16_385), Err(FilterError::Unsupported(_))), "{:?}", large(16_385));
|
||||
assert!(!matches!(large(16_384), Err(FilterError::Unsupported(_))), "{:?}", large(16_384));
|
||||
|
||||
// An extension segment (ISO/IEC 14492 7.4.14), which a decoder skips: its 11-byte header
|
||||
// gives the length of the data after it.
|
||||
let skipped = [0, 0, 0, 0, 0x3E, 0, 0, 0, 0, 0, 4, 0, 0, 0, 1];
|
||||
let generic = decode_jbig2(&data, None, w, h, usize::MAX).unwrap();
|
||||
assert_eq!(decode_jbig2(&data, Some(&skipped), w, h, usize::MAX).unwrap(), generic);
|
||||
assert!(matches!(decode_jbig2(&data, Some(b"not JBIG2"), w, h, usize::MAX), Err(FilterError::Corrupt { .. })));
|
||||
for end in 0..data.len() {
|
||||
let _ = decode_jbig2(&data[..end], None, w, h, usize::MAX);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,9 @@
|
||||
//! TIFF/PNG predictors, `ASCIIHexDecode`, `ASCII85Decode`, `RunLengthDecode`).
|
||||
//! Image codecs (`DCTDecode`, `JPXDecode`, `JBIG2Decode`, `CCITTFaxDecode`),
|
||||
//! `Crypt` and unknown filters are reported as [`FilterError::Unsupported`] so
|
||||
//! callers keep such data encoded. 1-bit samples can be encoded as CCITT Group 4
|
||||
//! callers keep such data encoded. CCITT fax and JBIG2 images decode on their own
|
||||
//! ([`decode_ccitt`], [`decode_jbig2`]), as `pdfcraft_cos::Document::decoded_image`
|
||||
//! does after the filters before them. 1-bit samples can be encoded as CCITT Group 4
|
||||
//! ([`encode_g4`]) or lossless JBIG2 ([`encode_jbig2`]).
|
||||
//!
|
||||
//! Every decoder is bounded by a caller-supplied `max_output` (decompression-bomb
|
||||
@@ -27,7 +29,7 @@ mod runlength;
|
||||
use std::borrow::Cow;
|
||||
|
||||
pub use ccitt::{CcittParams, MAX_CCITT_COLUMNS, MAX_G4_COLUMNS, decode_ccitt, encode_g4};
|
||||
pub use jbig2::{MAX_JBIG2_COLUMNS, encode_jbig2, encode_jbig2_generic};
|
||||
pub use jbig2::{MAX_JBIG2_COLUMNS, MAX_JBIG2_PIXELS, decode_jbig2, encode_jbig2, encode_jbig2_generic};
|
||||
|
||||
/// A PDF stream filter, identified by its `/Filter` name.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
|
||||
@@ -85,7 +85,8 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
- Apart from the downsampling it is lossless: the samples keep their meaning, so the colour
|
||||
space, decode array and `/Mask` stay, but for packed RGB, which becomes grey with one decode
|
||||
pair. Group 4 wins on scans, whose ragged edges defeat Flate; on clean computer-drawn 1-bit
|
||||
art Flate can be smaller. Not yet: inline images, an image's own `/SMask` or `/Mask`
|
||||
art Flate can be smaller. Not yet: JBIG2 images themselves (left as they are, neither
|
||||
downsampled nor stored again, though `Document::decoded_image` reads them), inline images, an image's own `/SMask` or `/Mask`
|
||||
image (only images drawn on a page are visited), and Group 4 for rows wider than 32,767
|
||||
samples.
|
||||
|
||||
|
||||
@@ -802,6 +802,9 @@ fn one_bit(doc: &Document, s: &Stream, pack: bool, defaults: bool) -> Option<(Ve
|
||||
let len = stride.checked_mul(hu)?;
|
||||
let max = match s.filters().last() {
|
||||
Some((pdfcraft_filters::Filter::CcittFax, _)) => wu.checked_mul(hu)?.checked_add(hu.checked_mul(4)?)?.checked_add(4096)?,
|
||||
// `decoded_image_within` reads JBIG2 too, but not for this yet: a resampled JBIG2 image
|
||||
// would come out Flate under Retain, and nothing weighs JBIG2 again against it.
|
||||
Some((pdfcraft_filters::Filter::Jbig2, _)) => return None,
|
||||
_ => allowance(len, h)?,
|
||||
};
|
||||
let mut data = doc.decoded_image_within(s, max).ok()?;
|
||||
|
||||
@@ -2044,6 +2044,58 @@ fn rendered_text_becomes_jbig2_symbols() {
|
||||
assert!(render(&doc) == before, "the page looks the same");
|
||||
}
|
||||
|
||||
/// A JBIG2 image is left as it is above 450 ppi, whatever the compression asked for, and so is one
|
||||
/// behind another filter: `Document::decoded_image` reads JBIG2, but the monochrome step doesn't
|
||||
/// take it yet (a resampled one would come out Flate under Retain). The same picture stored with
|
||||
/// Flate is averaged down under each, so it is the JBIG2 that stops it.
|
||||
#[test]
|
||||
fn jbig2_images_are_left_as_they_are() {
|
||||
// Black squares 16 samples a side on even rows and columns: at 600 ppi, halving leaves no new
|
||||
// pixel covering both samples, so nothing keeps the picture at its resolution.
|
||||
let (w, h) = (1200u32, 480u32);
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut squares = vec![0xFFu8; stride * h as usize];
|
||||
for (top, left) in (0..h as usize / 32).flat_map(|r| (0..w as usize / 32).map(move |c| (r * 32 + 8, c * 32 + 8))) {
|
||||
for y in top..top + 16 {
|
||||
for x in left..left + 16 {
|
||||
squares[y * stride + x / 8] &= !(0x80 >> (x % 8));
|
||||
}
|
||||
}
|
||||
}
|
||||
let coded = pdfcraft_filters::encode_jbig2_generic(&squares, w, h).unwrap();
|
||||
let hex: Vec<u8> = coded.iter().flat_map(|b| format!("{b:02x}").into_bytes()).chain(*b">").collect();
|
||||
let grey = [("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))];
|
||||
for compression in [MonoCompression::CcittG4, MonoCompression::Jbig2, MonoCompression::Retain] {
|
||||
let settings = Settings { mono: MonoSettings { compression, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
// 144 × 57.6 pt is 600 ppi.
|
||||
let content = "q 144 0 0 57.6 20 600 cm /Im Do Q";
|
||||
let mut doc = Document::new_empty();
|
||||
let im = mono_image(&mut doc, w, h, &squares, None, &grey);
|
||||
page(&mut doc, &[("Im", im)], content);
|
||||
let r = optimize(&mut doc, &settings).unwrap();
|
||||
assert_eq!(r.images_resampled, 1, "Flate, {compression:?}: {r:?}");
|
||||
for (filters, data) in
|
||||
[(vec![Object::name("JBIG2Decode")], &coded), (vec![Object::name("ASCIIHexDecode"), Object::name("JBIG2Decode")], &hex)]
|
||||
{
|
||||
let mut doc = Document::new_empty();
|
||||
let mut d = Dict::new();
|
||||
d.set(b"Subtype".to_vec(), Object::name("Image"));
|
||||
d.set(b"Width".to_vec(), Object::Int(w.into()));
|
||||
d.set(b"Height".to_vec(), Object::Int(h.into()));
|
||||
for (k, v) in &grey {
|
||||
d.set(k.as_bytes().to_vec(), v.clone());
|
||||
}
|
||||
d.set(b"Filter".to_vec(), Object::Array(filters.clone()));
|
||||
let im = doc.add(Object::Stream(Stream::from_raw(d, data.clone())));
|
||||
page(&mut doc, &[("Im", im)], content);
|
||||
let before = stream(&doc, im);
|
||||
let r = optimize(&mut doc, &settings).unwrap();
|
||||
assert_eq!((r.images_resampled, r.images_recompressed), (0, 0), "{filters:?}, {compression:?}: {r:?}");
|
||||
assert_eq!(stream(&doc, im), before, "{filters:?}, {compression:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A whole 300 dpi letter page of rendered text, more marks than one JBIG2 text region of the
|
||||
/// decoder PdfCraft renders with takes, still looks the same as JBIG2 symbols.
|
||||
#[test]
|
||||
|
||||
@@ -130,8 +130,8 @@ milestone = "M1"
|
||||
status = "partial"
|
||||
commands = []
|
||||
tools = []
|
||||
evidence = ["crates/filters/src/ccitt.rs::decoding_gives_the_rows_back", "crates/filters/src/ccitt.rs::group_3_rows_on_bytes_read_by_the_codes_in_the_data", "crates/cos/src/document/tests.rs::fax_images_decode_with_their_parameters"]
|
||||
notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction. Byte-aligned Group 3 is read by its end-of-line codes when the data starts with one and on bytes otherwise; still limited: data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at /K 0 may come out as other rows, which the optimizer then writes back. The bootstrap renderer draws byte-aligned Group 3 whose fill ends each end-of-line code on a byte as other rows (black box, at every /K), which decode_ccitt reads right; redaction removes Group 3 on bytes whole"
|
||||
evidence = ["crates/filters/src/ccitt.rs::decoding_gives_the_rows_back", "crates/filters/src/ccitt.rs::group_3_rows_on_bytes_read_by_the_codes_in_the_data", "crates/cos/src/document/tests.rs::fax_images_decode_with_their_parameters", "crates/filters/src/jbig2/decode.rs::both_codings_decode_to_their_samples", "crates/filters/src/jbig2/decode.rs::what_it_cannot_decode_is_an_error", "crates/cos/src/document/tests.rs::jbig2_images_decode_with_their_globals"]
|
||||
notes = "decoded through the hayro bootstrap for display; own codecs are M1.1/M1.2. CCITT fax decoding is in filters (decode_ccitt, through hayro-ccitt), which Document::decoded_image uses for Export ▸ Images, the optimizer and redaction; so is JBIG2 decoding (decode_jbig2, through hayro-jbig2, the page bounded before it is decoded), which it uses with the image's /JBIG2Globals (within 32 MiB) for Export ▸ Images and Copy Image. Byte-aligned Group 3 is read by its end-of-line codes when the data starts with one and on bytes otherwise; still limited: data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at /K 0 may come out as other rows, which the optimizer then writes back. The bootstrap renderer draws byte-aligned Group 3 whose fill ends each end-of-line code on a byte as other rows (black box, at every /K), which decode_ccitt reads right; redaction removes Group 3 on bytes whole"
|
||||
|
||||
[[feature]]
|
||||
id = "core.decompression-limits"
|
||||
@@ -1599,7 +1599,7 @@ status = "partial"
|
||||
commands = []
|
||||
tools = []
|
||||
evidence = ["crates/ui-egui/tests/it/viewer.rs::a_clicked_picture_is_selected_and_copied_in_its_original_format", "crates/ui-egui/tests/it/viewer.rs::right_click_on_a_picture_copies_it_at_full_size", "crates/ui-egui/tests/it/viewer.rs::a_picture_inside_form_artwork_under_text_is_selected_and_copied", "crates/engine/src/tests.rs::copy_image_finds_a_picture_inside_form_artwork_under_text"]
|
||||
notes = "Select tool: a click selects a raster image (also inside Form artwork or under text) and ⌘C copies it as a file in its original format (JPEG as stored, else PNG); right-click ▸ Copy Image copies its pixels at full resolution. Missing: inline images (BI/ID/EI), JPEG 2000 images and JBIG2 inside other filters, multi-format clipboard (file and bitmap at once), the web build"
|
||||
notes = "Select tool: a click selects a raster image (also inside Form artwork or under text) and ⌘C copies it as a file in its original format (JPEG as stored, else PNG); right-click ▸ Copy Image copies its pixels at full resolution. Missing: inline images (BI/ID/EI), JPEG 2000 images, multi-format clipboard (file and bitmap at once), the web build"
|
||||
|
||||
[[feature]]
|
||||
id = "view.rulers"
|
||||
@@ -7188,8 +7188,8 @@ milestone = "M10"
|
||||
status = "shipped"
|
||||
commands = ["export.all_images"]
|
||||
tools = ["doc_export_all_images"]
|
||||
evidence = ["crates/create/src/tests.rs::ccitt_fax_images_export_as_their_picture", "crates/create/src/tests.rs::group_3_ccitt_masks_and_parameters_by_reference_export_as_their_picture", "crates/create/src/tests.rs::images_export_as_jpeg_unchanged_and_others_as_png", "crates/create/src/tests.rs::indexed_and_one_bit_images_decode", "crates/create/src/tests.rs::jpeg_images_with_a_soft_mask_export_as_png_with_alpha", "crates/create/src/tests.rs::soft_masks_of_another_size_and_stencil_and_colour_key_masks_become_alpha", "crates/create/src/tests.rs::soft_mask_decode_ranges_become_exported_alpha", "crates/create/src/tests.rs::soft_mask_decode_maps_packed_and_jpeg_masks_before_resizing", "crates/automation/tests/it/automation.rs::exporting_all_images_through_tools", "crates/ui-egui/tests/it/export.rs::export_dialogs_write_images_and_text", "crates/create/src/tests.rs::lossless_decode_maps_each_component_before_colour_conversion", "crates/create/src/tests.rs::lossless_indexed_decode_rounds_in_index_units_then_clamps", "crates/create/src/tests.rs::jbig2_images_export_as_their_picture", "crates/create/src/tests.rs::jbig2_globals_are_decoded_within_a_limit_of_their_own", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/create/src/tests.rs::symbol_coded_jbig2_images_export_as_their_picture"]
|
||||
notes = "each image once (page resources and nested forms); JPEG unchanged unless it has transparency, others PNG (gray/RGB/CMYK/ICC/Indexed, 1–16 bpc, stencil masks); transparency as PNG alpha: /SMask (scaled to the image size, /Matte un-blended), stencil /Mask and colour-key /Mask, JPEG images included (#645); JPX/JBIG2 masks are left out (image exported opaque); JPX/JBIG2/separations reported as skipped; minimum size filter. Lossless images apply per-component Decode; Indexed values round and clip before palette lookup. Missing, null or malformed Decode arrays keep the default mapping. Soft-mask Decode intervals preserve opacity before resizing, including indirect arrays and endpoints. Still limited: 16-bit samples use high bytes; general JPEG image Decode, ICC transforms, and calibrated colors remain unsupported. CCITT fax images (Group 3 and 4, with their /DecodeParms, behind other filters too) export as their picture, masks included; byte-aligned Group 3 is read by the end-of-line codes when the data starts with one and on bytes otherwise, and one cut short is reported. Still limited: byte-aligned Group 3 with codes but none before the first row (not as T.4 sends them) is reported, or at /K 0 may export wrong."
|
||||
evidence = ["crates/create/src/tests.rs::ccitt_fax_images_export_as_their_picture", "crates/create/src/tests.rs::group_3_ccitt_masks_and_parameters_by_reference_export_as_their_picture", "crates/create/src/tests.rs::images_export_as_jpeg_unchanged_and_others_as_png", "crates/create/src/tests.rs::indexed_and_one_bit_images_decode", "crates/create/src/tests.rs::jpeg_images_with_a_soft_mask_export_as_png_with_alpha", "crates/create/src/tests.rs::soft_masks_of_another_size_and_stencil_and_colour_key_masks_become_alpha", "crates/create/src/tests.rs::soft_mask_decode_ranges_become_exported_alpha", "crates/create/src/tests.rs::soft_mask_decode_maps_packed_and_jpeg_masks_before_resizing", "crates/automation/tests/it/automation.rs::exporting_all_images_through_tools", "crates/ui-egui/tests/it/export.rs::export_dialogs_write_images_and_text", "crates/create/src/tests.rs::lossless_decode_maps_each_component_before_colour_conversion", "crates/create/src/tests.rs::lossless_indexed_decode_rounds_in_index_units_then_clamps", "crates/create/src/tests.rs::jbig2_images_export_as_their_picture", "crates/create/src/tests.rs::jbig2_globals_are_decoded_within_a_limit_of_their_own", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/create/src/tests.rs::symbol_coded_jbig2_images_export_as_their_picture", "crates/create/src/tests.rs::jbig2_images_behind_another_filter_export_as_their_picture"]
|
||||
notes = "each image once (page resources and nested forms); JPEG unchanged unless it has transparency, others PNG (gray/RGB/CMYK/ICC/Indexed, 1–16 bpc, stencil masks); transparency as PNG alpha: /SMask (scaled to the image size, /Matte un-blended), stencil /Mask and colour-key /Mask, JPEG images included (#645); JPX masks are left out (image exported opaque); JPX images and separations reported as skipped; minimum size filter. Lossless images apply per-component Decode; Indexed values round and clip before palette lookup. Missing, null or malformed Decode arrays keep the default mapping. Soft-mask Decode intervals preserve opacity before resizing, including indirect arrays and endpoints. Still limited: 16-bit samples use high bytes; general JPEG image Decode, ICC transforms, and calibrated colors remain unsupported. CCITT fax images (Group 3 and 4, with their /DecodeParms, behind other filters too) export as their picture, masks included; byte-aligned Group 3 is read by the end-of-line codes when the data starts with one and on bytes otherwise, and one cut short is reported. Still limited: byte-aligned Group 3 with codes but none before the first row (not as T.4 sends them) is reported, or at /K 0 may export wrong. JBIG2 images (with the /JBIG2Globals of their own /DecodeParms entry, within 32 MiB, behind other filters too) export as their picture, masks included; the JBIG2 page has to be the image's size (the renderer draws the whole page into the image)."
|
||||
|
||||
[[feature]]
|
||||
id = "create.export-html"
|
||||
@@ -7408,8 +7408,8 @@ milestone = "M11"
|
||||
status = "partial"
|
||||
commands = ["optimize.advanced", "optimize.reduce"]
|
||||
tools = ["doc_optimize", "doc_reduce"]
|
||||
evidence = ["crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols"]
|
||||
notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. Not yet: inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples."
|
||||
evidence = ["crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_images_are_left_as_they_are"]
|
||||
notes = "CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. Not yet: inline images, soft masks and /Mask images, JBIG2 images themselves (left as they are: neither downsampled nor stored again), Group 4 for rows wider than 32,767 samples."
|
||||
|
||||
[[feature]]
|
||||
id = "optimize.only-if-smaller"
|
||||
|
||||
Reference in New Issue
Block a user