with_missing_appearances_cached checked each /Annots entry against the source document, which never gets the appearance, so an annotation listed K times added K appearance streams to the display copy. The engine writes that copy on open and after every edit, so the display bytes grew with the number of listings: 20,000 listings of one Ink comment in a 136 KB file came to 428 MB.
Each annotation reference is now handled once per call, across all pages.