Grzegorz MartensandClaude Opus 5.5
85c1005148
Each JBIG2 bitmap was bounded on its own, but a few kilobytes could still exhaust memory: nothing bounded what a page's segments hold for later ones, or how far one segment could multiply them. Measured on black-box streams, peak memory before and after: - intermediate regions, 16 in 774 bytes: 514 MiB, now 98 MiB; - an 8 MiB symbol exported 1,200 times, a 2,989-byte PDF: killed at 8 GiB, now 79 MB; - dictionaries of zero-width symbols: 781 MB, now 72 MB; - dictionaries retaining their coding contexts, 1.1 MB: 2.89 GB, now 71 MB; - 65,535 references to one dictionary, 78 KB: about 17 GB, now 6 MB; - a halftone grid of 16,384 cells a side, 98 bytes: 1.1 GB, now 2.4 MB; - a pattern dictionary of 65,535 patterns, 48 bytes: 539 MB, now 15 MB, and it still draws; - a custom Huffman table behind nested Flate, a 1,297-byte PDF: killed at 8 GiB, now 278 MB, which is mostly the decoded stream; - 11-byte segments behind nested Flate in a stream and its globals, a 4,765-byte PDF: 5 GB, now 602 MB, which is mostly the two decoded streams; - an Indexed image whose page header claims 2^20 a side, a 743-byte PDF: aborted asking for 137 GB, now 23 MB; - a chain of 4,000 /JBIG2Globals streams that are JBIG2 themselves, on a 2 MiB stack: overflowed it and aborted, now 24 MB. So PageState counts what it stores: regions, pattern and symbol dictionaries and Huffman tables, with bitmap headers, retained contexts and a charge per item, up to 64 MiB, two square bitmaps of 16,384 a side. A dictionary's new and exported symbols count against what is left as they are made, not once it is stored. A segment may refer to at most 2^20 symbols, checked before any are collected. A halftone grid has at most 2^25 cells. A pattern dictionary keeps at most 16 MiB of shifted copies, which only speed up drawing, and none that wouldn't fit what the page has left. A custom Huffman table has at most 2^16 lines, and a stream with its globals at most 2^20 segments and 2^22 references, checked as they are read. A page past the bounds on a bitmap is refused when the stream is opened, since callers size their output from its header before the page bitmap is made. And in hayro-syntax, a /JBIG2Globals stream that is itself JBIG2 isn't decoded: globals are raw segments, so it reads as globals that can't be decoded. An image past a bound isn't drawn, as with the other caps, and the rest of the page is. This bounds memory, not time: decoding work has no total yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
hayro-syntax
A low-level library for reading PDF files.
This crate implements the Syntax chapter of the PDF reference, and therefore
serves as a very good basis for building various abstractions on top of it, without having to reimplement
the PDF parsing logic.
This crate does not provide more high-level functionality, such as parsing fonts or color spaces.
Such functionality is out-of-scope for hayro-syntax, since this crate is supposed to be
as light-weight and application-agnostic as possible.
Functionality-wise, this crate is therefore close to feature-complete. The main missing feature is support for password-protected documents. In addition to that, more low-level APIs might be added in the future.
The crate is no_std compatible but requires an allocator to be available.
Example
This short example shows you how to load a PDF file and iterate over the content streams of all pages.
use hayro_syntax::Pdf;
use std::path::PathBuf;
// First load the data that constitutes the PDF file.
let data = std::fs::read(
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../hayro-tests/pdfs/custom/text_with_rise.pdf"),
)
.unwrap();
// Then create a new PDF file from it.
//
// Here we are just unwrapping in case reading the file failed, but you
// might instead want to apply proper error handling.
let pdf = Pdf::new(data).unwrap();
// First access all pages, and then iterate over the operators of each page's
// content stream and print them.
let pages = pdf.pages();
for page in pages.iter() {
let mut ops = page.typed_operations();
while let Some(op) = ops.next() {
println!("{op:?}");
}
}
Safety
When the unsafe feature is disabled, there is only use of unsafe in hayro-syntax.
We also unconditionally use the smallvec crate which uses unsafe internally, but that's
it.
For better performance, it is strongly recommended to enable the unsafe feature,
which does result in slightly more unsafe code being, but does give better performance:
- We will use the
flate2crate for decoding flate streams. - SIMD will be enabled to accelerate decoding of images.
memchrwill be used to accelerate some of the fallback parsing code.
Features
The supported features include:
- Parsing xref tables in all its possible formats, including xref streams.
- Best-effort attempt at repairing PDF files with broken xref tables.
- Parsing of all objects types (also in object streams).
- Parsing and decoding PDF streams.
- Iterating over pages as well as their content streams in a typed or untyped fashion.
- The crate is very lightweight, especially in comparison to other PDF crates.
Limitations
- There are still a few features missing, for example, support for password-protected PDFs. In addition to that, many properties (like page annotations) are currently not exposed.
- This crate is for read-only processing, you cannot directly use it to manipulate PDF files. If you need to do that, there are other crates in the Rust ecosystem that are suitable for this.
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.