mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-24 10:08:00 -06:00
fix(syntax): bound inline-image lookahead by content window
Fall back only when missing sample or terminator bytes cannot fit the content-window budget, including retained header and separator overhead. Preserve available deterministic boundaries and ordinary large streams. Add oversized, exact-budget, overflow and large-stream regressions.
This commit is contained in:
@@ -172,6 +172,8 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
|
||||
|
||||
## Progress log
|
||||
|
||||
- **2026-10-22 (M2, #3107 / PR #3438):** Inline-image parsing falls back to the existing EI heuristic when missing declared samples, headers or terminators exceed the content-window budget, instead of waiting for an impossible allocation. Exact-length parsing for available samples and ordinary large streamed images is retained. Estimates unchanged.
|
||||
|
||||
- **2026-10-19 (M14, #3373):** The main menu and its long submenus scroll in short windows, keeping their commands reachable. Estimates unchanged.
|
||||
|
||||
- **2026-10-19 (M3, #3387):** Save-prompt shortcuts yield to later consent and utility dialogs; a synthetic UI regression checks that an edited document stays open. Estimates unchanged.
|
||||
|
||||
+1
-1
@@ -111,7 +111,7 @@ total matches the parity estimate.
|
||||
|---|---|---|---|---|---|
|
||||
| M0 | Skeleton: workspace, xtask gates, CI | 15–30 | 98% | 1–2 | GitHub workflow, `deny.toml` (licence audit of every dependency), parity checklist (826 features, `xtask parity`) done. Missing: remaining crate stubs, testkit/oracle crates |
|
||||
| M1 | COS: filters, crypt, parser, xref, writer | 120–200 | 80% | 20–40 | Done:<br>- filters and crypt: every standard-security revision R2–R6 (RC4, AES-128/256), SASLprep, permissions, creating encryption;<br>- cos: parse and repair (including invalid xref object-number ranges), decoded-stream limits including unfiltered data, decrypt on load, re-encrypt on save, incremental and full writing.<br>Corpus: open/edit/save passes on 958 files, and all 7 password-protected files open.<br>Full saves now pack objects into compressed object streams. An unchanged incremental save of a classic table writes a zero-length cross-reference subsection instead of a section with no subsections (#1339). A full rewrite keeps unknown trailer entries, including objects reachable only from them (#1338). Fuzzing runs nightly (`xtask fuzz`). Raw-deflate fallback preserves decoding-limit errors, including through tolerant COS stream decoding. Missing: ≥ 250 tests, own image codecs |
|
||||
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. |
|
||||
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Oversized missing inline-image samples, including header and terminator overhead, no longer pin the content window (#3107 / PR #3438); normal large streamed images retain deterministic sample boundaries. |
|
||||
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. The renderer keeps its one-point crop-size floor in physical points after UserUnit scaling, with regression coverage for crop origins, rotations and f32 reciprocal rounding. Existing unit policy and nearly-zero fallback are retained. |
|
||||
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings |
|
||||
| M3 | Viewer app (native + web) | 80–150 | 90% | 10–20 | Acrobat-style shell, find, select, panels, tiles, web build, UI control channel for agents (opt-in). Missing: 60 fps test on a 500-page document, snapshot tests of every panel<br>Done since: fit visible, document title in the window; Linux middle-button scrolling; shared three-state theme preferences in the toolbar, View menu and control channel; numeric Linux portal events and idle wakeups completing #881, with native appearance inheritance retained (#446); late browser startup downloads preserve the current document and Save target (#171). Selected-text Copy in Edit and the canvas context menu is covered by focused regressions and native clipboard checks (#1225); document copy restrictions apply to the floating action too. #1364 candidate: absolute local GoToR links ask before target access; headless callers explicitly opt in. Native-opening candidate: chosen tools survive preparation and password retries; startup intake waits for pending preparation. Page-label navigation displays logical labels, retains the current page on unchanged submission and refreshes focused same-document replacements; clean file-watcher reload preserves the chosen scrolling mode, preventing a Single Page reader from jumping to page 1 during replacement. Synthetic control regressions cover these behaviors. #3149: page-list changes discard render demand based on old geometry; actual raster dimensions are checked against the GPU limit before upload. Synthetic crash and continued-rendering regressions cover the replacement path. Save-prompt shortcuts yield to dialogs above them; stacked link-consent UI regression covers Escape and Return. |
|
||||
|
||||
+86
-23
@@ -245,7 +245,12 @@ impl<'a> UntypedIter<'a> {
|
||||
// PdfCraft patch: an unfiltered inline image's byte length is determined by
|
||||
// its parameters (and `/L` or `/Length` when present and consistent), so
|
||||
// `EI` inside its samples does not cut the image short (#3107).
|
||||
match unfiltered_image_end(stream_data, &dict, self.complete) {
|
||||
match unfiltered_image_end(
|
||||
stream_data,
|
||||
&dict,
|
||||
self.complete,
|
||||
start_offset.saturating_sub(self.committed),
|
||||
) {
|
||||
UnfilteredEnd::Found(end_data, end_after_ei) => {
|
||||
let image_data = stream_data.get(..end_data)?;
|
||||
let stream = Stream::new(image_data, dict.clone());
|
||||
@@ -971,33 +976,37 @@ fn unfiltered_image_len(dict: &Dict<'_>) -> Option<usize> {
|
||||
}
|
||||
|
||||
/// PdfCraft patch: locate the deterministic `EI` terminator of an unfiltered inline image (#3107).
|
||||
fn unfiltered_image_end(data: &[u8], dict: &Dict<'_>, complete: bool) -> UnfilteredEnd {
|
||||
fn unfiltered_image_end(data: &[u8], dict: &Dict<'_>, complete: bool, header_bytes: usize) -> UnfilteredEnd {
|
||||
let Some(end_data) = unfiltered_image_len(dict) else {
|
||||
return UnfilteredEnd::Fallback;
|
||||
};
|
||||
// An inline image shares the content window's budget, not the larger decoded-image
|
||||
// budget. Include its dictionary/header and the shortest whitespace + EI + delimiter
|
||||
// look-ahead, or one more byte beyond any separator whitespace already present.
|
||||
// Missing bytes that cannot fit must use the existing heuristic, not pin the window
|
||||
// and discard following content. Already-available exact ends still win.
|
||||
let missing_end = if complete
|
||||
|| end_data
|
||||
.checked_add(4)
|
||||
.and_then(|minimum| data.len().checked_add(1).map(|next| minimum.max(next)))
|
||||
.and_then(|needed| needed.checked_add(header_bytes))
|
||||
.is_none_or(|needed| needed > crate::MAX_DECODED_STREAM)
|
||||
{
|
||||
UnfilteredEnd::Fallback
|
||||
} else {
|
||||
UnfilteredEnd::NeedsMore
|
||||
};
|
||||
let Some(after) = data.get(end_data..) else {
|
||||
return if complete {
|
||||
UnfilteredEnd::Fallback
|
||||
} else {
|
||||
UnfilteredEnd::NeedsMore
|
||||
};
|
||||
return missing_end;
|
||||
};
|
||||
let Some(&first) = after.first() else {
|
||||
return if complete {
|
||||
UnfilteredEnd::Fallback
|
||||
} else {
|
||||
UnfilteredEnd::NeedsMore
|
||||
};
|
||||
return missing_end;
|
||||
};
|
||||
if !is_white_space_character(first) {
|
||||
return UnfilteredEnd::Fallback;
|
||||
}
|
||||
let Some(separator) = after.iter().position(|&b| !is_white_space_character(b)) else {
|
||||
return if complete {
|
||||
UnfilteredEnd::Fallback
|
||||
} else {
|
||||
UnfilteredEnd::NeedsMore
|
||||
};
|
||||
return missing_end;
|
||||
};
|
||||
let Some(rest) = after.get(separator..) else {
|
||||
return UnfilteredEnd::Fallback;
|
||||
@@ -1006,11 +1015,7 @@ fn unfiltered_image_end(data: &[u8], dict: &Dict<'_>, complete: bool) -> Unfilte
|
||||
return UnfilteredEnd::Fallback;
|
||||
}
|
||||
let Some(&second) = rest.get(1) else {
|
||||
return if complete {
|
||||
UnfilteredEnd::Fallback
|
||||
} else {
|
||||
UnfilteredEnd::NeedsMore
|
||||
};
|
||||
return missing_end;
|
||||
};
|
||||
if second != b'I' {
|
||||
return UnfilteredEnd::Fallback;
|
||||
@@ -1022,13 +1027,14 @@ fn unfiltered_image_end(data: &[u8], dict: &Dict<'_>, complete: bool) -> Unfilte
|
||||
Some(&b) if !is_regular_character(b) => UnfilteredEnd::Found(end_data, end_after_ei),
|
||||
Some(_) => UnfilteredEnd::Fallback,
|
||||
None if complete => UnfilteredEnd::Found(end_data, end_after_ei),
|
||||
None => UnfilteredEnd::NeedsMore,
|
||||
None => missing_end,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ContentTokenizer, UntypedIter};
|
||||
use crate::reader::ReaderExt;
|
||||
|
||||
fn operators(data: &[u8]) -> Vec<Vec<u8>> {
|
||||
let mut tokenizer = UntypedIter::new(data);
|
||||
@@ -1216,6 +1222,63 @@ BT /F1 14 Tf 72 550 Td (After) Tj ET";
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn oversized_unfiltered_inline_image_does_not_hold_following_content() {
|
||||
for length in [crate::MAX_DECODED_STREAM - 1, crate::MAX_DECODED_STREAM, crate::MAX_DECODED_STREAM + 1] {
|
||||
// The next BI supplies the existing EI heuristic's bounded look-ahead. The
|
||||
// stream is not exhausted: an impossible declaration must not pin this window.
|
||||
let data = format!(
|
||||
"BI /W {length} /H 1 /CS /G /BPC 8 ID x EI BT (After) Tj ET \
|
||||
BI /W 1 /H 1 /CS /G /BPC 8 ID y EI Q"
|
||||
);
|
||||
let mut tokenizer = UntypedIter::new_window(data.as_bytes(), false);
|
||||
for expected in [b"BI".as_slice(), b"BT", b"Tj", b"ET"] {
|
||||
let instruction = tokenizer.next_instruction().expect("following content survives");
|
||||
assert_eq!(instruction.operator.as_ref(), expected, "declared length {length}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unfiltered_inline_image_budget_includes_header_and_terminator() {
|
||||
let mut reader = crate::reader::Reader::new(b"/W 1 /H 1 /CS /G /BPC 8 ID ");
|
||||
let inline = reader
|
||||
.read_without_context::<crate::object::dict::InlineImageDict<'_>>()
|
||||
.expect("synthetic inline dictionary");
|
||||
let dict = inline.get_dict();
|
||||
let fits = crate::MAX_DECODED_STREAM - 5; // One sample + whitespace/EI/delimiter.
|
||||
assert!(matches!(super::unfiltered_image_end(b"", dict, false, fits), super::UnfilteredEnd::NeedsMore));
|
||||
assert!(matches!(super::unfiltered_image_end(b"x ", dict, false, fits), super::UnfilteredEnd::Fallback));
|
||||
assert!(matches!(super::unfiltered_image_end(b"x EI", dict, false, fits + 1), super::UnfilteredEnd::Fallback));
|
||||
assert!(matches!(super::unfiltered_image_end(b"x EI", dict, true, usize::MAX), super::UnfilteredEnd::Found(1, 4)));
|
||||
for header in [fits + 1, usize::MAX] {
|
||||
assert!(matches!(super::unfiltered_image_end(b"", dict, false, header), super::UnfilteredEnd::Fallback));
|
||||
// The guard is only for missing bytes, never a reason to reinterpret available samples.
|
||||
assert!(matches!(super::unfiltered_image_end(b"x EI Q", dict, false, header), super::UnfilteredEnd::Found(1, 4)));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn large_unfiltered_inline_image_still_waits_for_its_samples() {
|
||||
// A valid image larger than the interpreter's initial 64 KiB window must
|
||||
// still use its exact length rather than execute operators in its samples.
|
||||
let length = (64 << 10) + 1;
|
||||
let mut data = format!("BI /W {length} /H 1 /CS /G /BPC 8 ID ").into_bytes();
|
||||
let sample_start = data.len();
|
||||
data.extend_from_slice(b"0EI Q BT (Injected) Tj ET q");
|
||||
data.resize(sample_start + length, b'0');
|
||||
data.extend_from_slice(b" EI BT (After) Tj ET");
|
||||
let mut partial = UntypedIter::new_window(&data[..64 << 10], false);
|
||||
assert!(partial.next_instruction().is_none());
|
||||
assert!(partial.needs_more);
|
||||
assert_eq!(partial.committed, 0);
|
||||
assert_eq!(operators(&data), [b"BI".to_vec(), b"BT".to_vec(), b"Tj".to_vec(), b"ET".to_vec()]);
|
||||
let expected = slice_instructions(&data);
|
||||
for piece in [4096, 64 << 10, data.len()] {
|
||||
assert_eq!(windowed_instructions(&data, || piece), expected, "piece size {piece}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unfiltered_inline_image_variants_and_fallbacks() {
|
||||
// ImageMask with 1-bit row padding (width 10 -> 2 bytes per row, height 2 -> 4 bytes)
|
||||
|
||||
Reference in New Issue
Block a user