#3670 raised the form depth limit to 50 and added its own 50,000-visit cap. Main's
per-page DecodeBudget (e2a677164) has the same 50,000-form limit, and the visit cap
fired first, so Apply removed the forms instead of refusing the page as main does.
The visit cap goes; the fan-out test expects the refusal, and the depth-limit
sanitization test builds its chain from MAX_DEPTH.
crates/redact/src/interp.rs: kept both sides: main's per-page DecodeBudget and the PR's
visits counter (MAX_FORM_VISITS, same 50,000 as MAX_PAGE_FORMS; the visit check fires first).
* chore: resolve SonarCloud maintainability issues
* fix(packaging): use verify_sha256 helper function
* style: cargo fmt
* fix: restore wildcard imports and fix method-reference type errors
Revert the glob-import expansions (they missed SomIndices and left unused
imports), revert char::is_ascii_digit where the iterator yields char by
value, and pass the content slice to prune_doomed_xobjects as &[Vec<u8>].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep XObject decodability check without binding content in interp
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Dylan Taylor <dylan@craftmaker.dev>
Redact: refuse Redact marks written in place; remove replies and pop-ups listed on
other pages; delete form fields by object, not name; resolve Do by its last operand
like the renderer; sanitize drops PCAdded and flattens widgets before hidden-content
cleanup. Sign: freed signed objects and edits to an already-signed widget are
modifications. Cos: only reading trusts /Type /XRef to skip encryption.
Redaction reads page content and form XObjects strictly (upstream) and
within one page's decode budget (this branch): both now go through
DecodeBudget::decode_strict.
Filtered inline image data has no length, so its end is an EI chosen
among candidates, and the content parser chose by a different rule
than the renderer's tokenizer (hayro-syntax): the renderer also takes an
EI with no whitespace before it, and checks a candidate by reading what
follows as operators. With image data such as "7FEI 1 0 0 1 200 0 cm",
the renderer ran a cm that redaction read as image data, so text was
drawn 200 pt from where removal and its verify pass looked.
The parser now takes each inline image's end from the renderer's
tokenizer (already in the tree through pdfcraft-cos), so both read the
same operators after the image; a test runs both on the same fixtures.
Only where that tokenizer reads no image (no EI it accepts, or
parameters it can't read) does the parser fall back to its own rules,
counting the image in Parsed::ambiguous_images, and applying redactions
refuses such a page as unreadable.
Structure cleanup cleared alternate text, actual text and expansions
only on the element that directly owned changed marked content, and an
element that lost an annotation reference kept its descriptions unless
it lost every kid. An ancestor's /ActualText covering a redacted span,
or a link element's /Alt repeating the address of a removed link, stayed
in the saved file. Elements that lose an annotation now always lose
their descriptions, and every ancestor of an affected element (found
through both /P and the tree's /K, which may disagree) loses its own.
An element written in place above one with a description to clear
refuses the operation, as other in-place cases already do.
Finding which marked content a redaction changed copied every operator
into the buffer of each MCID enclosing it, so content nested n deep was
held n times: a modest stream of nested BDCs could exhaust memory.
Each operator inside marked content is now serialized once, and each
MCID keeps the ranges its sequences span. The bytes compared are capped;
MCIDs past the cap count as changed, so their descriptions are cleared
rather than kept.
Clearing the pixels of a partly covered image decoded its stream with
the global 1 GiB allowance and only then cut it to the raster length its
Width, Height and bit depth declare. A small Flate stream could make
every such image allocate and inflate far more than it can use. Decoding
now stops at twice the raster plus a byte per row (filter-chain and
predictor overhead) and a little slack; an image past that is removed
whole, as an undecodable one already is.
After applying redactions, removed images and forms were unlinked from
page-tree resource dictionaries but not from the resources of forms the
page still draws. A form sharing the page's original resource dictionary
(a common producer layout) kept the removed image reachable, so a full
save wrote it. The walk over surviving forms now drops XObject entries
for removed objects that the form's own content does not draw, and fails
closed when that cannot be determined.
The redaction interpreter started every form XObject with an empty text
state (no font, size 0, no spacing) although a form inherits its caller's
graphics state, and it ignored the /Font entry of an ExtGState set by gs.
Text shown that way was placed at the wrong positions, so removal and
the verify pass could both miss glyphs under a mark. Forms now start
from the caller's state, and gs sets the font and size as Tf does.
The engine's renderer cross-check test used the gs case as its example
of interpreter/renderer disagreement; it now runs the check directly on
an unredacted page, which also covers an input stream labelled as a
redaction overlay.
Both checks for text left under applied redaction marks (redaction's own
verify pass and the engine's renderer cross-check) skipped any page
content stream whose dictionary said /PCMark /Redaction, so that the
overlay's label would not count. An input file could put that label on
its own content stream and keep it out of both checks. Applying
redactions now records the overlay streams it adds (stamp returns the
new stream), and only those are skipped.
Deleting a field took it off its pages and out of the field tree but left
its entry in the form's /CO array. That reference kept the field
dictionary, its value and its widgets reachable, so a full save after
applying redactions over the field still wrote the redacted value.
The name-tree count cloned each node's /Kids array and skipped repeated
references without charging the budget, so a node listing the same kid
many times cost work proportional to every entry while the budget saw one
step. The walk now borrows the array, charges every entry it does not
descend into, and stops once the budget is spent.
Metadata, private application data, associated files and additional
actions were removed from the catalog and pages only. The same entries on
an image or font (XMP), a Form XObject (PieceInfo), a page or annotation
(/AF with its embedded file) or a form field (/AA scripts) survived the
full rewrite, and the scan did not count them, so removing only such a
category reported nothing to remove. A walk of every object reachable from
the catalog now counts and strips these entries wherever they are, leaving
name maps such as resource dictionaries alone (their keys are names a
producer chose), and refuses a graph nested too deeply to walk in full.
Removing hidden text or hidden layers rewrote page and Form content but
left the structure tree alone, unlike applying redactions. An element's
ActualText, Alt or expansion text still described the removed content,
and a marked-content reference into a Form kept the original Form, with
the removed text, in the full rewrite. The hidden-content pass now cleans
the tags of what each page no longer shows and moves Form references to
the rewritten copy, refusing when the tree can't be cleaned completely.
Removing form fields flattens them and then drops the interactive form,
but flattening keeps hidden and no-view widgets (they draw nothing) and
widgets with no appearance. The pages still listed them, so their values
and their parent fields' values survived the full rewrite. Every widget
left after flattening now leaves the pages and the structure tree, inline
annotations included.
Page and Form content was decoded tolerantly, keeping what came before a
corrupt spot and dropping the rest without saying so. A page whose content
broke at its first bytes therefore read as empty: the hidden-text scan
counted nothing, the removal was skipped and the original stream, which
other viewers still draw, stayed in the file. Content is now decoded
strictly; an unreadable page counts as something to remove, so the removal
runs and refuses the page, and a Form read only in part is treated like an
undecodable one.
The hidden-information count followed every /Kids entry of the attachment
and document-script name trees with only a depth limit, so a node listed
twice by each level was read once per path: a few hundred bytes made the
scan take 2^32 steps. Indirect nodes are now read once each, under a total
node budget.
The interpreter entered a form again at every Do with only a nesting
limit, so a few shared forms drawing each other produced exponential
work in apply, sanitize and verify. Scope carries a per-page
DecodeBudget for form entries and form bytes; past it, apply refuses
the page and verification counts the form as possible residue. Page
content streams are decoded against a page budget too.
Review of #3670: with the depth limit at 50, depth was the only bound on the
form walk. A form that draws the next level twice, or a cycle of two forms,
costs 2^depth interpretations — at 12 that was 4096, at 50 it is a hang on a
file anyone can send.
The pass now also counts the forms it enters, and the budget is Edit text's own
`MAX_FORM_VISITS`, referenced rather than copied. Running out is treated as a
form that cannot be read, which is what the depth limit already did: removed in
Apply, the page left alone in Verify.
Measured with the new test:
depth 50, fan-out and a 50-deep cycle, with the budget 5.9 s
depth 22, fan-out, budget removed still running after
5 minutes
A `const` assertion ties this pass's depth limit to Edit text's, so the build
fails rather than a redaction quietly removing what a reader still shows (the
review's third point). Checked by setting the limit back to 12:
error[E0080]: evaluation panicked: assertion failed:
MAX_DEPTH >= pdfcraft_edit::text::MAX_FORM_DEPTH
Tests: a page whose forms each draw the next twice, and the same with the last
form drawing the first again — both must finish and still redact the page.
* fix(redact): evaluate OCMD visibility during sanitization
Fixes#3499 by evaluating OCMD policies and bounded visibility expressions against the default optional-content configuration before removing hidden groups. Unknown or malformed visibility data is handled conservatively.
* fix(redact): preserve OCMD visibility during sanitization
* fix(redact): count hidden XObjects during verification
* test(redact): cover hidden form depth limit
The total budget of the outline count (100,000 items) was checked against the
number of indirect items the walk had reached. Items written inline, as a
direct dictionary, were counted in the result but never entered that set, so
an outline of 60,000 indirect siblings, each with one inline child, counted
120,000 items.
The walk now keeps a separate running total of every item it counts, inline
items included, and stops when that total reaches the budget. The set of
indirect items is still used to count shared items once. Ordinary outlines
count as before.
The doc comment of the walk now states the depth limit in one sentence: an
item first reached through a deeper path is not visited again from a shallower
one, so items nested past depth 64 may stay uncounted. The depth behaviour is
unchanged.
Test added in crates/redact/src/tests.rs:
inline_outline_children_count_towards_the_total_budget. It reads 120000 on the
previous head and 100000 now. Reverting the budget check to the indirect-only
count fails it; reverting the increment fails it and the existing total budget
test. The other outline tests pass unchanged.
Checked with cargo fmt --all -- --check, clippy with -D warnings on
pdfcraft-redact, and cargo test for pdfcraft-redact (85 passed, 0 failed).
A Redact annotation with neither a usable QuadPoints array nor a usable
Rect was dropped silently by marks(). Applying the readable marks beside
it then reported success, and the text under the dropped mark stayed in
the file.
apply() now counts such marks on the pages being applied and returns
RedactError::UnreadableMark(n) before anything changes. The message goes
out through the existing error text: the engine edit error, the
automation reply and the UI failure line all print it. Marks that were
readable through the existing fallbacks (a Rect alone, a usable
QuadPoints array) work as before.
Tests: three in crates/redact/src/tests.rs (refusal with the count and
no change, Rect-only still redacts, an unreadable mark on another page
does not block the selected page) and one end to end in
crates/automation/tests/automation.rs (the reply carries the reason and
the page keeps its text). On upstream main with only the variant added,
the two refusal tests and the automation test fail; the Rect test passes.
Reverting each pinned production line turns the tests red again.
Gates: cargo fmt --check, clippy -D warnings for pdfcraft-redact and
pdfcraft-automation, and the redact test suite pass.
An outline whose items name shared children (/First and /Next pointing at
nodes that other items also name) was counted by following every path. With
60 levels of two items each, a walk that follows every path takes 2^60
steps. The outline count never finished, and Remove Hidden Information and
Sanitize both run that count first.
The count now keeps the set of items it has reached. An item reached a
second time is not counted again, and the level stops there: the walk that
reached the item first goes on to count the rest of that level. A total
budget of 100,000 distinct items bounds the work for an outline with very
many items. The per-level cap is unchanged.
Ordinary outlines count as before. The existing hidden-information test
(two bookmarks) passes unchanged, and a new nested outline test (five
items) pins the count for nesting and siblings.
Tests added in crates/redact/src/tests.rs:
- an_outline_whose_items_share_children_is_counted_without_spinning
- removing_bookmarks_from_an_outline_whose_items_share_children_finishes
- an_outline_counts_no_more_items_than_its_total_budget
- a_nested_outline_is_counted_item_by_item
The first, second and third were seen to fail on the code before this
change: the first two timed out after 10 seconds, and the third read 120000
where 100000 is expected. The fourth passes before and after the change and
guards the ordinary count. The spin tests run the walk on a thread with a
10 second bound, so a regression fails the test instead of hanging the run.
Reverting only the shared-item check, or only the budget check, fails the
tests they pin.
Checked with cargo fmt --all -- --check, clippy with -D warnings on
pdfcraft-redact, and cargo test for pdfcraft-redact.
Redaction removed annotations and form fields under a mark from the pages
but left the structure tree's /OBJR references to them, so a full rewrite of
a tagged PDF kept the comment's /Contents and the field's /V. apply now
drops references to every annotation no page lists afterwards (widgets of a
removed field on other pages included), as Sanitize does since #2532, and
fails closed with RedactError::Tags when an element can't be edited.
A form past the pass's depth limit is removed rather than walked. That
is the fail-safe choice for a redaction -- marked content inside a form
we cannot read would otherwise survive -- but it also removes whatever
else that form drew, marked or not. At a limit of 12, a page with a
13-deep chain lost the text of its innermost form with nothing in it
marked.
Measured on a page that draws HEAD, a chain of forms whose innermost
draws DEEP, then TAIL, with only HEAD marked:
chain before after (12) after (50)
1 HEAD DEEP TAIL DEEP TAIL DEEP TAIL
12 HEAD DEEP TAIL DEEP TAIL DEEP TAIL
13 HEAD DEEP TAIL TAIL DEEP TAIL
14 HEAD DEEP TAIL TAIL DEEP TAIL
50 HEAD DEEP TAIL TAIL DEEP TAIL
51 HEAD TAIL TAIL TAIL
The limit is now 50, which is what the renderer follows
(MAX_NESTED_INTERPRETATION_DEPTH) and what Edit text's reader follows
(text::MAX_FORM_DEPTH). So anything a reader can see, this pass can
walk. The last row is why 50 and not more: at 51 the renderer does not
draw DEEP either, so removing that form costs nothing visible and the
fail-safe still holds.
Fixes#3645's data loss. The other half of that report -- that nothing
tells the user when a form is removed -- is untouched: Report.forms_removed
is counted but the engine's edit pipeline does not carry it out, and the
one channel that reaches the result (doc.edit_notices) is typed
TextNotice, for text that could not be drawn. Widening that type or
adding a channel is a design choice for the maintainers rather than
something to slip into this fix.
`[(FIRSTARR)] [(LASTARR)] TJ` is malformed (TJ takes one array
operand), but the operator still shows the one immediately before it,
as the renderer, MuPDF and pdftotext agree. Three readers here counted
operands from the front instead and showed the discarded first array:
- crates/edit/src/text.rs (text_lines/text_paragraphs, and the Office
export through reading_lines)
- crates/redact/src/interp.rs (search-based redaction's content
interpreter)
- crates/a11y/src/headings.rs (autotagged heading detection)
So the editor could offer a word the page never draws, a redaction
search could miss it, and a heading could read as text nobody sees.
Take the last array for TJ in all three, matching the `_ =>
op.operands.last()` arm already used for Tj/'/" alongside it.
Added a regression test per crate (`a_tj_with_a_surplus_array_reads_
the_last_one`, `a_tj_surplus_arrays_undrawn_glyphs_are_not_seen_
under_a_mark`, `a_heading_with_a_tj_surplus_array_reads_the_one_the_
page_draws`), confirmed each fails without its fix and passes with it.
cargo test -p pdfcraft-edit (196 passed) -p pdfcraft-redact (81
passed) -p pdfcraft-a11y (44 passed), fmt, clippy -D warnings, xtask
layers and xtask assets all clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Maintainer review on #3285: an element written in place that owns redacted
marked content or tags a removed annotation, an inherited page the lookup gave
up on (over 256 ancestors or a loop), and a walk past its limit are now
RedactError::Tags, so nothing is changed. Removing only /StructTreeRoot left
outline /SE and GoTo /SD reaching the elements. Inherited pages are cached per
ancestor. clear_every_element and drop_tree are gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
Review follow-ups on #3285: walk /K before the parent and ID trees and look up
an element's inherited /Pg through /P; drop the structure tree when a cut-short
walk could leave MCRs naming a replaced stream; drop it when an element written
in place tags a removed annotation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
tags::clean, which drops /Alt, /ActualText and /E that repeat redacted
text, walked the structure tree from the root's /K only, so an element
listed only in /ParentTree or /IDTree kept the redacted words. It now
starts from both trees too, as drop_object_refs does since 3f543ca1.
Past its 2,000,000-reference cap the walk used to skip the rest and
report success. It now fails closed: every structure element in the
document, found by reading each object rather than through the tree,
loses its alternate and actual text.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
Follow-up to #2532 (review by aiongg): the structure-tree walk started
only at the root's /K, so an element listed only in /ParentTree or
/IDTree kept its object reference to a removed link or comment, and the
root kept that annotation reachable in a full rewrite. The walk now
also starts from both trees and follows their nodes (/Nums, /Names,
/Kids), written in place or as their own objects.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
aiongg