mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-25 18:47:22 -06:00
sonar-fix/enforce-https
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 PiohrunandClaude Opus 5.5
|
f65f733041 |
fix(render): faster sampled functions, image decodes across zoom levels and patch meshes (vendored hayro-interpret patches 31–33) (#2436)
* fix(render): sampled functions index a flat sample array (vendored hayro-interpret patch 31) A Type 0 sampled function kept its samples in a HashMap keyed by each point of its /Size grid (two SmallVecs per key, SipHash), and the renderer builds a shading's functions again every time it is painted, so every paint hashed one insert per sample. A 2,097,152-sample ramp painted 30 times took 38 s; the pdf.js corpus's bug1721218 (InDesign) spent most of a 1.5 s page on it. The samples now stay in stream order in one Vec, and a grid point is found by strides (first input fastest). The interpolation keeps the same recursion and arithmetic, so results are bit for bit the same, and a point the stream ended before still gives no value. The repeated-paint page draws in well under a second. The maximum sample value no longer computes 2^32 in a u32 (a panic with overflow checks on at 32 bits per sample). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MoEx9Nj8KC4WKDcf5YaokQ * fix(render): one decode of an image serves every size it is drawn at (vendored hayro-interpret patch 32) The decoded-image cache (patch 24) keyed each decode by the size the image is drawn at, but only the JPX filter decodes a reduced resolution for a size: every other filter decodes the whole image. So a JPEG or Flate scan missed the cache at every zoom step, was decoded again, and was kept once per zoom level. The key now carries the target size only when JPX decodes the image, its /SMask or its /Mask (read from /Filter or /F, erring towards keeping the target). `InterpreterCache::decoded_image_decodes` counts the cache's decodes; the new test draws a Flate image at five sizes and sees one decode, and a copy whose soft mask is JPX-encoded still decodes per size. Both assertions fail with the old keying. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MoEx9Nj8KC4WKDcf5YaokQ * fix(render): patch meshes copy their vertices' colours less (vendored hayro-interpret patch 33) A Coons or tensor-product patch is drawn as 722 triangles over a 20 x 20 grid of points, and every vertex's colours (a SmallVec) were produced or copied up to five times: each grid cell interpolated its four corners itself (1,444 interpolations a patch for 400 points), cloned all six vertices of its two triangles, and the mesh sampler cloned each vertex again to move the triangle into device space. Profiling page 8 of the annual report in #1584 put a quarter of its time in those clones. Each grid point's colours are now interpolated once, a cell's second triangle takes the vertices its first copied, and the sampler moves a patch's freshly made triangles instead of copying them (`TriangleSampler::sample_owned`; a free-form triangle mesh, which the shading keeps, is still copied). The same inputs give the same values: rendering the first three pages of every pdf.js corpus file is byte for byte the same. Release build, Linux: report page 8 773 -> 696 ms, personwithdog.pdf 149 -> 129 ms, issue13520.pdf 104 -> 87 ms. The new test pins the colours near each corner, at the centre and between them for a Coons and a tensor-product patch (taken before the change); swapping two vertices' grid points makes it fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MoEx9Nj8KC4WKDcf5YaokQ * docs: roadmap entries for faster sampled functions, image decodes and patch meshes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MoEx9Nj8KC4WKDcf5YaokQ --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d415749556 |
Heavy pages: cull off-tile images, cache decodes, preview backdrops, calmer queues (#1117)
* Heavy pages: cull off-tile images, cache decodes, preview backdrops, calmer queues A large CAD sheet (page 24 of the benchmark file: ~5050 pt wide, 2104 images, 9 MB of content) cost a full page interpret per tile: 4.3 s for a 1024 px tile at 1000%, 10.5 s for the whole page, and a storm of renders on every pan frame. Tiles now cost 0.6 s, the sheet is covered by a handful of large tiles, and gestures no longer queue work faster than workers finish it. - Vendored hayro patch (10): skip images whose stream-size box misses the canvas before decoding them, so a tile no longer decodes, resamples and rasterizes every image on its page. Pixel-identical; the skip regression test now covers images too. - Vendored hayro-interpret patch (23): decoded image pixels shared across renders through the interpreter cache, keyed by stream and decode target size within a 64 MiB budget. - Preview renders (hayro patch 11, RenderRequest::preview through to the device): backdrops, thumbnails, grid cells and signature-drag backgrounds skip images under 16 device px. Tiles, whole pages, print and export always draw everything. Previews may differ by those pixels; finals don't. - Viewer: zoom/pan settle holds apply only to pages whose raster proved slow (slow_pages, 500 ms), so fast pages render with no added latency; tile sides grow with zoom and page size (giant sheets need ~8 tiles, not ~28), queued center-first and keyed by exact coordinates; Edit-text outlines capped. - vendor/README.md records the new patches with their tests. Co-Authored-By: Muse Spark <noreply@meta.com> * Decoded-image cache: key every decode input, survive a poisoned lock The key was the dictionary's bytes and the target size, so two images with byte-identical dictionaries (uncompressed, one size) shared pixels, and an image drawn under a transfer function reused pixels decoded without it. DecodedImageKey now holds the dictionary's place in the document (its data follows it), its bytes and object, mask/stencil/raster output, interpolation, the target size and preview or final quality; images under a transfer function and inline images (named colour spaces, dictionaries in content streams) are not cached, and an image over the budget is never kept. A poisoned lock decodes uncached instead of panicking. Soft masks: the per-renderer cache can't carry a preview mask into a final render (documented). Regression tests for identical dictionaries, preview then final through one cache, and transfer functions. --------- Co-authored-by: Muse Spark <noreply@meta.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
306bbcedf9 |
M3: harden render and CLI failure surfaces (#669)
* M3: harden render and CLI failure surfaces Make content truncation observable, reject unknown CLI options, expose deterministic render work counters, and scope render failures to exact requests. Refuse silently reduced whole-page export rasters while preserving tiled rendering, and add heavy-vector and CLI regression coverage. Co-authored-by: Codex <noreply@openai.com> * Render: remember a failed render by page and kind again A key with scale and tag made every zoom level and tile a new key: a pathological page tied up a worker for the watchdog limit per request (exhausting worker replacements) and was re-parsed each time. The repeated-parsing check asks for a new generation and scale again. * Export: pages too large at the chosen dpi export at the most the renderer allows, and say so #669 made every Exporter refuse oversized whole-page renders, which reached File > Export to image, doc_export_* and OCR: at 600 dpi any page side over 13.65 in (Legal, A3, Tabloid) failed instead of exporting. The exporter clamps again; Exporter::dpi_used reports the resolution a page is drawn at, which the Export dialog's message and doc_export_png's lower_dpi list mention. A source with reject_oversize still refuses (CLI render keeps its strict config). * doc_export_images: describe lower_dpi * Render: ContentTruncated is reported once per page; the hayro-interpret patch is recorded Nested forms, Type 3 glyphs and patterns are interpreted again and each reported the exhausted budget; pdfcraft-render now keeps one. The warning is part of hayro-interpret patch (18) (the page content budget): marked in the code and described in vendor/README.md. * CLI render: print render warnings (skipped content) to stderr * CLI: the option check refuses unknown options only; commands report missing values themselves text's own --page parsing (and its test on main) says what a page value must be; the generic "needs a value" pre-empted it. --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
f5c99ce059 |
Never crash: mesh shadings can't exhaust memory or run for minutes (#412)
* Never crash: mesh shadings can't exhaust memory or run for minutes `cargo xtask fuzz` (with the dash, JBIG2 and decompression fixes in place) found a page whose Coons patch shading (type 6), a 147-byte stream with 1-bit coordinates decoded onto +-40000, took the renderer past 3 GB. hayro-interpret samples mesh shadings (types 4-7) pixel by pixel over each triangle's device-space bounding box into a map, with nothing but the u16 cast bounding it: up to 65536 x 65536 entries for one triangle. A synthetic single patch passed 6 GB in six seconds; ten thousand triangles that each cover half a 1000 x 1000 page ran for minutes. Two rounds of review of earlier versions found more: every patch was cut into 722 triangles (about 150 KB) before anything limited them; a lattice mesh with `/VerticesPerRow 0` pushed empty rows forever, and one with 4294967295 read the whole stream into one row; a mesh painted a thousand times was triangulated a thousand times; and the first caps cut content without a word. Vendored hayro-interpret patch (18): - `ShadingPattern::encode_within(area)`: the vendored hayro renderer passes the area it will actually draw (the path's bounding box on the canvas, which it already computes) plus a pixel for texels at a fractional edge, and triangles are sampled only inside it, so the map holds at most that area's pixels. - `TriangleSampler`: at most 8 visits per pixel of that area plus 4 per triangle in it. A triangle's bounding box is about twice its area and a sub-pixel triangle's is at most 2 x 2 pixels, so tiling meshes and small dense ones (a patch in a few pixels is 722 triangles) still shade completely; overlapping ones stop. - Coons and tensor-product patches are triangulated one at a time. - Meshes hold at most `MAX_MESH_PATCHES` (2^16; a 200 x 200 gradient mesh is 40 thousand) patches and `MAX_MESH_TRIANGLES` (2^20) triangles, and a lattice needs two (ISO 32000-2 8.7.4.5.6) to 2^19 vertices per row. - A page samples at most `MAX_PAGE_MESH_TRIANGLES` (2^25; the 200 x 200 mesh is 29 million) mesh triangles in all, reset per page like the existing paint budget. - Each of these cuts logs a warning. `encode()` keeps its signature; with no area it samples within a fixed 2^24 visits. Synthetic regression tests in render: - `mesh_shadings_far_off_the_page_render`: a Coons patch and a triangle mesh decoded onto +-40000 render and the rest of the page draws; a patch on the page, the part of the huge one that covers it, and a patch filling a 4 x 4 pt rectangle as a pattern are all shaded. - `many_overlapping_mesh_triangles_finish`: 10,000 triangles each covering half of a 1000 x 1000 page finish, and are drawn. - `mesh_shadings_with_many_patches_or_no_rows_finish`: 40,000 patches beside the page, and a lattice with `/VerticesPerRow 0`. Red on main (out of memory, killed at 3.9 GB). Each part is pinned by a mutation: no visit budget stalls past 60 s; no clip leaves the covering patch unshaded; no per-triangle allowance leaves the small patch unshaded; triangulating all patches first runs out of memory (killed at 5.8 GB); no rows check runs out of memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Never crash: decompression bombs can't exhaust the renderer's memory The renderer's parser (vendored hayro-syntax) decoded Flate, LZW and RunLength streams with no output limit. A 2 MB page whose content stream inflates to 2 GB, or whose 8 x 8 image does, took `pdfcraft-cli render` and `text` past 3 GB in under a second, and a few more MB of file scale it further. On the desktop that is the first-page render of an emailed PDF, and the allocation-failure abort takes every open document with it. Our own layers already bound single streams (lopdf's load-time decoding at 256 MiB, cos at 1 GiB), so `info` stays at a few MB; the renderer path didn't. Random mutation almost never makes a high-ratio stream, so `cargo xtask fuzz` doesn't find this; found by hand while triaging the fuzz run. Vendored hayro-syntax patch (6): - `decode_limit` gives each filter an output limit, and Flate (both flate2 and the fallback inflater), LZW and RunLength never write past it (the buffer can still grow to under twice it): `MAX_DECODED_STREAM`, 256 MiB, lopdf's figure; or, for the last filter of an image whose size, bits per component and components are known, twice its pixel bytes (a PNG predictor tag byte per byte at most), capped at 1 GiB. Earlier filters in a chain produce compressed data and get the general limit. - `Stream::decoded_within(max)` for callers that budget. - A page's `/Contents` array shares one `MAX_DECODED_STREAM` (naming one bomb twenty times decoded 5 GB), reserving exactly what it appends. Vendored hayro-interpret patch (18): - The content a page holds at once (its contents, and the forms, tiling patterns and Type 3 glyph procedures being painted) stays within `MAX_PAGE_CONTENT`, 512 MiB, reset per page like the existing paint budget. Forms nest fifty deep and each level kept its content while the next ran: a form painting itself around a 256 MiB stream would have held over 12 GB. Each level gives its bytes back when it is dropped, so content painted again and again keeps drawing, and a stream without filters (borrowed from the file) costs nothing. - Type 0 sampled functions read every sample into a 4-byte value and then a table entry: 256 MiB of 1-bit samples asked for over 8 GB. They now take at most 2^21 table points and 2^23 values (the largest real tables, DeviceN tint transforms such as 33^4 points x 3 outputs, are 1.2 million and 3.6 million), counted with checked arithmetic, and read no further than `/Size` asks. `/Size 0` no longer underflows building the default `/Encode`. Three rounds of independent review found the aggregate cases, buffer doubling past the limit, a chain/predictor shape the first image limit would have truncated, the Type 0 amplifier, and a first, cumulative budget that would have dropped legitimately repeated content, and a first Type 0 cap low enough to refuse real tint transforms; all fixed here. Measured in a release build (render; OS peak commit): - content stream inflating to 2 GB: 389 MB, 0.3 s - 8 x 8 image inflating to 2 GB: 4 MB, 0.2 s - /Contents naming it twenty times: 516 MB, 0.3 s - a form painting itself around it: 644 MB, 0.4 s Synthetic regression tests in render (flate2 is a new dev-dependency there, already in the workspace): - `decompression_bombs_stop_at_the_stream_limit`: 288 MiB bombs through flate2, the fallback inflater (a stored block with a wrong length check, which flate2 refuses and the fallback accepts), LZW and RunLength each decode exactly 256 MiB, and at most 128 bytes as an 8 x 8 image. - `page_contents_and_nested_forms_share_one_budget`: three /Contents references decode one stream's worth; a self-painting form finishes. - `content_painted_again_and_again_keeps_drawing`: a 2 MiB form painted 300 times draws the 300th time. - `sampled_functions_with_billions_of_samples_are_refused`. Ten mutations each turn them red: no limit at all; no limit in flate2, the fallback, LZW, RunLength or for images; an unshared /Contents limit; no page budget; no refund on drop; no Type 0 sample cap. (Taking a form out of the budget no longer compiles: its content is the budget guard.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * M2: normalize JPEG2000 samples without overflowing Use bounded integer sample scaling for 1-32-bit JPX images and reject empty dimensions before chunking. Cover public decoding and continued page rendering with synthetic regressions. Co-Authored-By: OpenAI Codex <codex@openai.com> * Never crash: decompression cut at its limit is logged, not silent; vendor README renumbered after #575 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Streetblock <github@streetblock.de> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
e88def3465 |
Never crash: decompression bombs can't exhaust the renderer's memory (#411)
* Never crash: decompression bombs can't exhaust the renderer's memory The renderer's parser (vendored hayro-syntax) decoded Flate, LZW and RunLength streams with no output limit. A 2 MB page whose content stream inflates to 2 GB, or whose 8 x 8 image does, took `pdfcraft-cli render` and `text` past 3 GB in under a second, and a few more MB of file scale it further. On the desktop that is the first-page render of an emailed PDF, and the allocation-failure abort takes every open document with it. Our own layers already bound single streams (lopdf's load-time decoding at 256 MiB, cos at 1 GiB), so `info` stays at a few MB; the renderer path didn't. Random mutation almost never makes a high-ratio stream, so `cargo xtask fuzz` doesn't find this; found by hand while triaging the fuzz run. Vendored hayro-syntax patch (6): - `decode_limit` gives each filter an output limit, and Flate (both flate2 and the fallback inflater), LZW and RunLength never write past it (the buffer can still grow to under twice it): `MAX_DECODED_STREAM`, 256 MiB, lopdf's figure; or, for the last filter of an image whose size, bits per component and components are known, twice its pixel bytes (a PNG predictor tag byte per byte at most), capped at 1 GiB. Earlier filters in a chain produce compressed data and get the general limit. - `Stream::decoded_within(max)` for callers that budget. - A page's `/Contents` array shares one `MAX_DECODED_STREAM` (naming one bomb twenty times decoded 5 GB), reserving exactly what it appends. Vendored hayro-interpret patch (18): - The content a page holds at once (its contents, and the forms, tiling patterns and Type 3 glyph procedures being painted) stays within `MAX_PAGE_CONTENT`, 512 MiB, reset per page like the existing paint budget. Forms nest fifty deep and each level kept its content while the next ran: a form painting itself around a 256 MiB stream would have held over 12 GB. Each level gives its bytes back when it is dropped, so content painted again and again keeps drawing, and a stream without filters (borrowed from the file) costs nothing. - Type 0 sampled functions read every sample into a 4-byte value and then a table entry: 256 MiB of 1-bit samples asked for over 8 GB. They now take at most 2^21 table points and 2^23 values (the largest real tables, DeviceN tint transforms such as 33^4 points x 3 outputs, are 1.2 million and 3.6 million), counted with checked arithmetic, and read no further than `/Size` asks. `/Size 0` no longer underflows building the default `/Encode`. Three rounds of independent review found the aggregate cases, buffer doubling past the limit, a chain/predictor shape the first image limit would have truncated, the Type 0 amplifier, and a first, cumulative budget that would have dropped legitimately repeated content, and a first Type 0 cap low enough to refuse real tint transforms; all fixed here. Measured in a release build (render; OS peak commit): - content stream inflating to 2 GB: 389 MB, 0.3 s - 8 x 8 image inflating to 2 GB: 4 MB, 0.2 s - /Contents naming it twenty times: 516 MB, 0.3 s - a form painting itself around it: 644 MB, 0.4 s Synthetic regression tests in render (flate2 is a new dev-dependency there, already in the workspace): - `decompression_bombs_stop_at_the_stream_limit`: 288 MiB bombs through flate2, the fallback inflater (a stored block with a wrong length check, which flate2 refuses and the fallback accepts), LZW and RunLength each decode exactly 256 MiB, and at most 128 bytes as an 8 x 8 image. - `page_contents_and_nested_forms_share_one_budget`: three /Contents references decode one stream's worth; a self-painting form finishes. - `content_painted_again_and_again_keeps_drawing`: a 2 MiB form painted 300 times draws the 300th time. - `sampled_functions_with_billions_of_samples_are_refused`. Ten mutations each turn them red: no limit at all; no limit in flate2, the fallback, LZW, RunLength or for images; an unshared /Contents limit; no page budget; no refund on drop; no Type 0 sample cap. (Taking a form out of the budget no longer compiles: its content is the budget guard.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * M2: normalize JPEG2000 samples without overflowing Use bounded integer sample scaling for 1-32-bit JPX images and reject empty dimensions before chunking. Cover public decoding and continued page rendering with synthetic regressions. Co-Authored-By: OpenAI Codex <codex@openai.com> * Never crash: decompression cut at its limit is logged, not silent; vendor README renumbered after #575 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Streetblock <github@streetblock.de> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
a747f2521f |
M2.6: W/W* clip after whichever path-painting operator ends the path (#362)
ISO 32000-2 §8.5.4: W and W* mark the current path to become the clip once the path-painting operator that ends it has painted it with the old clip, and n is only one such operator. The vendored interpreter applied the clip in n alone. After S, s, f, F, f*, B, B*, b or b* it was never applied, so everything painted afterwards ignored it, and it stayed pending until some later, unrelated n (even after Q) clipped content that should show. Issue #260's test file uses `W* S` on pages 3, 5 and 9: a shading meant for a circle flooded page 3, an image meant for a rounded rectangle and nested clips on 5 and 9 spilled out. Every painting operator now applies a pending clip after painting and clears it (apply_pending_clip, shared with n). The now unused Context::clip getter is removed. Test: raster::tests::clipping_path_applies_after_any_painting_operator. W and W* with each of n, S, s, f, F, f*, B, B*, b, b*: the later red fill stays inside the square, and a stroking operator's outer half still shows outside it (painted under the old clip). A clip used by S doesn't linger for a later n, inside or after q/Q. On main it fails at `W S`. MuPDF, Poppler and PDFium agree on every case. Page 3 of #260's file now differs from PDFium by 0.84/255 on average (was 164). Vendored patch (16). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
Brandon ThomasandClaude Opus 5.5
|
d3db99818f |
Rename PrintCraft to PdfCraft (#174)
The app, crates, binaries, bundle id, packaging, icons, screenshots and docs are now PdfCraft / pdfcraft (repo storytold/pdfcraft). On first launch the desktop app moves the PrintCraft settings and crash-recovery folders to the new name, and both apps read settings saved under the old storage key, so upgrades keep recent files, preferences and unsaved work. The signing test fixtures keep their PrintCraft subject names (they are binary). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brandon ThomasandClaude Opus 5.5
|
2f364665b8 |
Never crash: the fuzz job's eleven hangs finish (parser retries, inline images, Type 3 fan-out)
Every hang the nightly fuzz job found (11 findings, 8 source files) now finishes in 0.1-2 s; each cause has a synthetic regression test that times out without its fix. - cos (7 findings, our own parser): a stream whose /Length pointed at itself, with no endstream, made every parse attempt fail and each attempt reload the same object: ~2^32 parses. Objects being loaded are a seen-set, /Length values are memoised per load, and the shifted-header retry only runs when the header is shifted. - hayro-syntax (3): deciding where an inline image's data ends re-parsed the rest of the stream, nesting through every inline image; 50 images ran for over ten minutes. The look-ahead no longer nests and is bounded (256). - hayro-interpret (1): a Type 3 glyph showing eight glyphs of its own font, sixteen deep. Nested paints share a per-page budget (50,000). The pdf.js corpus sweep matches its baseline (963 ok, same 20 known). xtask check writes its report under CARGO_TARGET_DIR too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brandon ThomasandClaude Opus 5.5
|
ee254a91a2 |
M2.6: fuzz findings in the bootstrap renderer: paint-nesting cap now covers Type 3 glyphs (stack overflow), hayro skips images over 2^28 px (hang); regression tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brandon Thomas
|
108fdc108c | more progress, roadmap, wip |