Conflicts: docs/roadmap.md: all status lines kept, M10 row from main, M11 row with #3227's two sentences. parity/acrobat-features.toml: save-with-object-streams entry from #3227 with its test paths under tests/it/.
The signing saves wrote with default options, so a redaction mark added
and cleared in the session (or a comment added and deleted) reached the
file through a signature, a timestamped signature, embedded LTV evidence
or a document timestamp, with its QuadPoints, overlay text and authors.
A signed revision can't be changed afterwards, so nothing could take it
out again.
`sign`, `sign_with_timestamp` and `dss::embed` now save with
`drop_unlinked`: everything they add is reachable (the signature through
the field's `/V`, the store from the catalog), and the ByteRange
placeholder is found in the written bytes.
`timestamp_document` can't: its `/DocTimeStamp` dictionary has no field
pointing to it, so the option would drop it and every document timestamp
would fail with "ByteRange placeholder not found". It frees the leftovers
first with the new `Document::free_unlinked_additions`, which frees
exactly what the option leaves out, and then adds the dictionary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A section naming an /XRefStm that a newer section had already read
skipped the stream entirely, so an entry in its own table that
disagreed with the stream was no longer recorded in xref_oddities.
Signature validation reads those records to refuse an appended section
another reader could follow. The reader keeps each hybrid stream's
entries (up to MAX_XREF_OBJECTS in all) and compares every later
naming section's table with them, without decoding the stream again.
The materialized cross-reference reader reread a section's /XRefStm
for every revision that named it, decoding the stream and inserting
its rows each time, although the first read had already merged every
entry. Up to 10,000 revisions naming one large stream repeated that
work 10,000 times. A stream already read is skipped, with one repair
log line.
A stream whose /Length is wrong or missing is ended at the next
endstream, which the parser found by searching the rest of the file.
Many such streams before one late endstream (a damaged file being
reconstructed reads every one) cost the file size per stream. The
document lists where endstream occurs the first time it needs it, and
each stream looks its end up there.
The object caches weigh what each cached object owns, but the walk
stopped four levels down and counted anything deeper as nothing. Many
objects holding a large string inside a few nested arrays stayed cached
far past the byte budget (34 MiB against 32 MiB in the test, and without
limit as objects grow). The walk now goes past the parser's nesting
limit; anything deeper counts as too large to keep.
A stream that named its own crypt filter (/Filter /Crypt with
/DecodeParms /Name) kept that name when the document was given new
password protection. The new /Encrypt defines only /StdCF, so the
inherited name resolved to /Identity and the stream was written in
clear into a file the user asked to protect. When saving with protection
applied by set_encryption, a selector the new handler does not define is
removed with its parameters and the stream follows the new policy.
Saving with the original handler keeps the source's selectors.
crates/cos/src/document.rs: #2220 moved the header scan into object_headers; it also returns the position after `obj`, and reconstruct applies this PR's /XRef candidate check (same 1024-byte window) to each header there. Tests: kept both sides.
crates/cos/src/document.rs: kept #2220's shifted_subsection_end and object_headers and this PR's prev_offset.
vendor/README.md: hayro-syntax row already has (22) and (23); this PR's patch and tests are numbered (24).
With junk before %PDF the table's offsets are relative to the header, so the "1 G obj" check is also tried there, as load does; otherwise object 1 stayed free and a full save dropped the image.
iText's indirectPrev.pdf writes /Prev 6 0 R, and object 6, listed in the
same update, holds 470, the offset of the first section. Both readers took
the object number as the offset, so cos found no section at 6, rebuilt the
file from object headers and lost its revision list, and hayro's chain
stopped there.
When /Prev is a reference, the sections read so far (cos) or the section's
own entries (hayro, which now reads them before following /Prev) are asked
for that object first. If it holds a non-negative integer, that is the
offset; otherwise the object number still is, as the tiff2pdf files need.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NH3itd85jPuM6wNjgcGfEH
The JHOVE PDF-HUL-157 files (tiff2pdf output) end their first trailer with
`/Prev 297298 0 R`: the offset written as a reference. The chain reader took
/Prev as an integer only, so the chain stopped after the first section without
a word, and every object of the earlier sections (here pages 2-4 and their
images) read as null. The document showed one page where Poppler and MuPDF
show four, and a full save wrote the missing kids as null, so pages 2-4 were
unreadable afterwards ("cycle in page tree", 1x1 renders).
A reference-valued /Prev is now read as the offset it names, with a repair-log
note; any other non-offset value is noted and ends the chain as before.
Regression test: document::tests::a_prev_written_as_a_reference_still_reaches_the_earlier_sections.
Verified on JA3-1998-PDF.pdf: four pages read, and after a Subject edit and a
full save Poppler renders all four pages without a warning (before: "Kid
object (page 2) is not an indirect reference (null)").
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011K1peyrh5MdZ6ttHkVq8Gh
A later cross-reference stream (or trailer) whose /Root points at another
dictionary, such as /Info, replaced an earlier good trailer, and a full save
then wrote that dictionary as /Root. A candidate whose /Root passes the
root_is_catalog rule (/Type /Catalog or /Pages) now beats one whose /Root is
only some dictionary, wherever it is in the file. The looser rule still
applies when nothing better is there, which FDF files (whose /Root has
neither key) and the repair path for a non-catalog /Root rely on.
Test: reconstruction_skips_a_trailer_whose_root_is_not_the_catalog
crates/cos/src/document.rs: both new accessors, header_offset (#2525) and reconstructed (#2029).
parity/acrobat-features.toml: the malformed-xref entry has both sides' evidence and notes (#2029 full save of a repaired file, #2717 table numbered from 1).
* fix(engine): Save writes a repaired file in full even without edits (#1937)
A damaged file is repaired on open (its cross-reference index rebuilt from
object headers, as Document Properties reports), but Save wrote it back as it
was: `Session::save_bytes` returned the bytes as opened whenever nothing was
edited, and a repair is not an edit. The "repaired" file on disk was still the
damaged one.
`save_bytes` now writes a reconstructed document in full even with no edits.
`Document::reconstructed()` in cos names the condition (no revision chain:
rebuilt, or built from scratch), and `write_incremental` uses it too, so the
writer and the engine share one definition.
Tests: `engine::tests::a_repaired_file_is_rewritten_by_save_even_without_edits`
(table offsets shifted; no table at all) and
`automation::saving_a_repaired_file_puts_the_repair_on_disk` (doc_open,
doc_info, doc_save over the same path, reopen). Both fail with the condition
removed: the saved bytes equal the damaged file. parity: core.repair-broken-files
cites them and notes that a repair still does not count as an unsaved change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
* fix(engine): a signed repaired file keeps its bytes on Save without edits (#1937)
Review follow-up: writing a reconstructed file in full on Save puts the
repair on disk, but for a signed file it would also invalidate its
signatures without warning. Save now skips the early return only when the
document is reconstructed and not signed. A regression test signs a
document, appends a `startxref` that points nowhere (the signed bytes stay
intact, the cross-reference data must be rebuilt) and checks that Save
without edits returns those bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
ROADMAP.md, docs/roadmap.md: union of both sides' log lines and milestone rows, each line once.
parity/acrobat-features.toml, crates/xfdf/README.md: next's text, a superset of main's (#1917, #2686 plus #2688).
crates/edit/src/tests.rs: both sides' new tests (#2558 and #3084).
crates/ui-egui/src/editing.rs: the picked picture is cleared on every edit, and a new drawing commits its raster (#3149).
When the cross-reference chain is unusable (here: a second, truncated PDF
appended after the first, so no startxref is in reach), the index is rebuilt
from object headers and the trailer taken from the last `trailer` dictionary
with a usable /Root. A file written with cross-reference streams has no
`trailer` keyword, so the rebuild fell through to the catalog alone and /Info
(and /ID, /Encrypt) were lost: Document Properties showed nothing, and a full
save after editing one Info field wrote only that field and /ModDate.
reconstruct() now also considers every /Type /XRef stream dictionary with a
usable /Root, taking whichever trailer comes last in the file.
Regression test: document::tests::reconstruction_takes_the_trailer_from_a_cross_reference_stream.
Verified on the issue's fixture (openpreserve PDF-HUL-138/6.2017-0960.pdf):
Creator, Producer, CreationDate and ModDate are read and kept by a full save.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011K1peyrh5MdZ6ttHkVq8Gh
* Issue #2367: an object stream whose entry cannot place it is found by its header
qpdf's xref-compressed-in-compressed.pdf marks object stream 69 as itself
compressed inside stream 81, which the format forbids and which does not
exist. objstm() loaded the container through its cross-reference entry only,
so every object inside (the outlines, the page labels) read as null and a
full save wrote `/Outlines null /PageLabels null`, while an incremental save
(which copies the original bytes) kept them, and MuPDF recovers them.
The container is now found by its own `N G obj` header when its entry cannot
load it (scan_for, the same last resort a plain object with a wrong offset has).
Regression test: document::tests::an_object_stream_marked_as_compressed_is_found_by_its_header.
Verified on the fixture: objects 70 and 82 load (Count 6; 12 page-label
ranges) and a full save keeps both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011K1peyrh5MdZ6ttHkVq8Gh
The fallback is for file entries only: once the stream is freed or replaced in the overlay,
its objects must not come back from the file's copy (sign's
object_stream_changes_cannot_reuse_old_candidate_membership caught this after the rebase);
a cos test covers both cases.
* cos: a container the saved table marks free is not found by its header either
Freeing an object stream, saving incrementally and reopening left the
overlay empty and the table saying the stream is free; the header
fallback then rescanned the file and the stream's objects came back on
the next full save. The fallback now also stops at an on-disk free
entry.
* M1: put the #2367 test's description on the test, not its fixture
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
The engine caches a page's text blocks for opening a paragraph and
applying its edit, but only while the document is unmodified: it was
keyed by the source identity, which edits keep. After the first edit the
cache never hit again, and the page was grouped again on every call (the
inline editor and the panel each call it after an apply).
pdfcraft_cos::Document::read_state() returns what the document reads as:
its source identity, its trailer and every changed object, compared as
allocations that the state keeps alive. A clone or a snapshot restored by
undo has the same state; any set, add, free or trailer change makes a
different one. The cache is keyed by it, so it hits after edits and never
returns another state's blocks.
On a page of 120 rows, text_blocks after an edit took 0.215 ms per call
and now 0.006 ms (release build, best of three; a throwaway timing test).
Tests: a read state matches a clone and not a document changed by set,
free or a trailer change, nor another document; after an edit the engine
groups the page once and undo and redo get their own blocks. The engine
test fails without this change.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Object-stream members were indexed with unchecked casts: `/First` and each
offset went through `as usize` and an unchecked addition, which truncate or
wrap on 32-bit targets such as the wasm32 build, and the member number went
through `as u32` on every target, so a member numbered 2^32 + 3 was read as
object 3. A negative offset was clamped to zero and read the stream's first
member instead.
Each member is now converted and added with checks, and kept as an unusable
slot (preserving positions, which cross-reference entries refer to) when its
number is not an object number or its start falls outside the decoded data.
Looking such a member up is a MissingObject error, as for any other absent
object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some producers head a classic xref table "1 n" and then write object 0's
free entry first, so every row is one number off and object 1 reads as
free. The renderer coped, but cos did not: printing a scanned invoice
built from cos dropped object 1 (page 1's image) and printed the page
blank. Such a subsection is now read from 0 when its second row locates
an object that is numbered 1, in both xref readers, and the repair is
logged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
crates/cos/src/document.rs: union with #2525: header_relative revisions, the renumbering log line, and log passed to read_section on the first try and the header-relative retry
Some producers head a classic xref table "1 n" and then write object 0's
free entry first, so every row is one number off and object 1 reads as
free. The renderer coped, but cos did not: printing a scanned invoice
built from cos dropped object 1 (page 1's image) and printed the page
blank. Such a subsection is now read from 0 when its second row locates
an object that is numbered 1, in both xref readers, and the repair is
logged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
Some files carry bytes before `%PDF-`, and their offsets count from the header rather than from
the first byte. We read those, retrying a section relative to the header when the absolute
offset misses, but the incremental writer then appended a section whose offsets and `startxref`
were absolute while its `/Prev` kept the header-relative value it was given. qpdf reported "xref
not found" and reconstructed the file, so a save that reported success left a damaged-looking
update behind (#2364, found with KillerPDF's reserve3.pdf and a Subject edit).
The reader now records on each `Revision` whether it was found counting from the header, and the
writer counts the new section the same way as the one its `/Prev` points to. A file whose
offsets already include the prefix keeps absolute offsets, which the second test pins so the fix
cannot overcorrect.
Seen red with the writer change reverted: the appended section's `startxref` missed its `xref`.
Fixesstorytold/pdfcraft#2364
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsT1y8xBvkXeBxEUQqdDtj
* M1: edit finds the pages a damaged file's readers find (fixes#1656)
* M1: rebuild a lost page tree only when the reader shows the same pages, and refuse edits to signed rebuilt files (#1656)
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* M7: share Windows local-link fixture portability fixes (#1265)
Use the identical test-only prerequisite already proposed in #1263: Unix-only mutable vectors, ordinary Windows drive paths and escaped PDF literal strings. This is shared CI support for the separate xref overflow fix; production link handling is unchanged.
Generated with Codex.
* M1: return an error for exhausted xref object numbers (#1337)
Check the next-object addition and return CosError at the xref offset rather than overflowing or wrapping to object zero. Classic-table and stream regressions reproduced the original panic; ordinary, zero-offset and repaired-range controls preserve catalog access and allocation.
Validation: 2732 passed, 0 failed, 13 ignored; strict all-target Clippy, fmt, 31 WASM crates, layering, assets, deny and parity passed. The rebuilt CLI opened and rendered a clean synthetic page.
Generated with Codex.
* M1: place overflow notes beside stable milestone entries (#1337)
Keep the tested code unchanged while avoiding progress-note conflicts with newly landed main updates. Formatting and diff checks passed.
Generated with Codex.
* Merge main: adapt exhausted-xref repair to the compact xref index
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
To classify what later revisions changed, the signed bytes were reopened
with Document::open, without a password, so for a file opened with a
nonempty user password the earlier revision was unreadable and the earlier
signature became Invalid as soon as it was counter-signed. Document::
open_with_same_credentials opens such bytes with the current document's own
security handler when their /Encrypt and /ID are the same (no password is
needed again or kept), and the signature validator uses it.
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
* M4: keep the last saved revision across undo (#1310, #1311, #1312)
Undo treated a return to the opened file as unsaved work, and after a save it followed the restored snapshot's older base. Revert and the next save now stay on the last saved file, so that revision remains in the PDF.
* Run a tool after a background open of the file it asked for
A tool chosen with no document opened the picked file on the worker, but only ran when that open finished inside the picker callback. Remember the command on the pending open, and let the command test wait for the worker.
* Wait for background opens in the command tests
Opening a picked file or a recent file finishes on the worker, so a fixed handful of frames can assert before the tab exists.
* Compare batch copies to the PDF that was written
A new text PDF stamps the current second, so a later pdf() call is a different file. The numbered copy and the source must match the bytes that were saved.
* Drop the unused batch fixture left by the merge
Main already keeps the written PDF in original, so the older source binding is unread and fails -D warnings.
* M8: redaction unlinks removed images and forms from inherited and shared resources
Apply Redactions and Remove Hidden Information give a rewritten page its own
pruned /Resources, but the dictionary the page drew through before could still
name the image or form that was removed:
- a /Resources inherited from a /Pages node;
- one resource dictionary object shared by several pages;
- the page's own dictionary, when a form without /Resources drew the image
through it.
A full-rewrite save keeps whatever is still referenced, so the removed image
samples and form text stayed in the saved file while the operation reported
success. Per-page resources (the case fixed in #1099) were not affected.
The interpreter now records the original object behind every XObject it drops
or replaces. After the page loop, each page-tree resource dictionary that still
names one loses that entry unless a page drawing through that dictionary still
draws the name (the same image on another page, or again outside a mark). A
dictionary that loses an entry is replaced by a private copy on its node, so a
dictionary object shared with other nodes is left alone for them. If a page's
use of a name cannot be told (unreadable content, a form that will not decode),
the operation fails closed.
Tests: redacted_xobjects_do_not_survive_through_inherited_or_shared_resources,
an_image_another_page_still_draws_through_the_same_resources_is_kept,
an_image_a_form_drew_through_the_pages_resources_does_not_survive,
a_hidden_layers_image_does_not_survive_through_inherited_resources.
Not covered here: originals named by a form XObject's own /Resources, by an
annotation appearance or by a pattern.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* M9: a cross-reference entry the reader sets aside is never an unchanged signed object
The reader that judges post-signing changes and the renderer that draws the
page resolve some cross-reference entries differently. A revision appended
after signing could redefine a page's content stream and give the object an
entry the reader sets aside while the renderer follows it:
- an in-use entry at offset 0 (the reader skips it; the renderer fails to read
the object there, rebuilds its index and takes the last definition);
- a second entry for the same object in one section (the reader keeps the
first, the renderer the last).
The classifier compared the signed object with itself, found nothing changed
and reported the signature Valid while the page showed the new content.
pdfcraft-cos now records the entries it does not take at face value
(Document::xref_oddities: offset-0 in-use entries other than object 0,
duplicate entries that differ, reserved cross-reference stream types, and a
hybrid /XRefStm entry that disagrees with the table), and can say whether an
entry locates the object itself (Document::xref_entry_is_exact). What it
resolves is unchanged.
classify_changes treats any such entry added after the signed revision as a
change that is never permitted (page content when the object was a signed page
or its content, other changes otherwise). An object whose content compares
equal at a new place is unchanged only when its entry is exact: where the
reader had to search for the object, another reader may arrive elsewhere.
Tests: a_cross_reference_entry_the_reader_sets_aside_does_not_hide_new_page_content,
a_cross_reference_entry_that_misses_its_object_is_not_an_unchanged_object,
cross_reference_entries_that_are_set_aside_are_recorded,
an_entry_that_misses_its_object_is_found_by_searching_but_is_not_exact.
This closes the two demonstrated triggers and their near variants. It does not
prove that the two parsers agree on every file: other differences in what each
tolerates may remain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* M8: Apply Redactions is refused when the page still shows text under a mark
Redaction decides what lies under a mark from the positions its own
interpreter computes, and its verify pass reuses them. Where those differ from
the renderer's, a mark drawn over the text as displayed lies over glyphs the
interpreter places elsewhere, or does not see at all: they stayed in the file
and the operation reported success.
The case the test pins: a font and size set through an ExtGState /Font entry
(`gs`), which the renderer honours and the interpreter does not model. The same
kind of gap was observed for a font a form XObject takes from its caller, for
some predefined CMaps and for a rotated Type 3 font matrix. The standard-14
width error that first showed it is fixed at its root by d0d1ad81.
The engine now asks the renderer, which is what the user saw when placing the
marks. After redaction runs, the result is written as a save would write it
(without the marks' own overlay), its text is extracted, and a glyph still at
least half inside a mark fails the edit with the existing "could not be
verified" error. The edit runs on a copy, so the document is unchanged. A page
whose text cannot be read back fails the same way.
This makes such mismatches fail closed; it does not make those documents
redactable. Teaching the interpreter each of them is separate work.
Test: redaction_is_refused_when_the_page_still_shows_text_under_a_mark (the
line whose font is set by `gs` is refused and left unchanged; the same line set
with `Tf` is redacted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A PDF protected with only a permissions (owner) password opens without
asking, restricted, and nothing let the user enter that password
afterwards: the security notice only led to a read-only Security tab and
the hint said to reopen the file with the password, which the app had no
way to do (#995, #1077).
Now the security notice and Document Properties > Security offer Unlock...,
and Change settings / Remove security on a restricted document ask for the
permissions password first and then carry on, as Acrobat does. The open
password is refused there.
- crypt: SecurityHandler::authenticate_owner accepts only the owner
password and refuses a different file key.
- cos: Document::unlock upgrades the open document in place; owner and user
passwords unlock the same file key, so nothing is decoded again.
- engine: Session::unlock (not an edit: no undo step, not marked unsaved)
also unlocks the undo and redo states and reopens saves as owner;
Document::can_unlock, EditError::WrongPassword, command protect.unlock.
- ui: the permissions password prompt shares the open prompt's modal; the
notice and Security tab get Unlock...; the disabled-command hints no
longer point at reopening the file.
- i18n: the new strings in all 15 full catalogs (catalog hashes updated).
Tests cover crypt, cos and engine (all four algorithms, wrong and open
passwords, undo, redo, save, removing security afterwards) and the UI
(notice flow, Remove security asking first).
Closes#995Closes#1077
Co-authored-by: Brandon Thomas <echelon@gmail.com>
aiongg