lightcraft

364 Commits 97 Branches 19 Tags

2 Commits

Include renames

Author	SHA1	Message	Date
Brandon ThomasandClaude Opus 5.5	ae1cbbe87b	Issue #37 : Catalog: stream the snapshot into the temp file instead of building two strings Journal::snapshot serialised the catalog into a String and then copied it into a second `format!`-ed String before writing (2x the snapshot size in memory, ~620 MB at 85k photos). It now streams `{"format":…,"seq":N,"catalog":` + serde_json::to_writer + `}\n` through a 1 MiB BufWriter into the store (new `Store::write_atomic_with`; FsStore writes the temp file directly, then flush + sync_all + rename + dir fsync as before; other stores buffer, as before). Durability and format are unchanged: same temp + fsync + atomic rename; a failed write leaves the old snapshot (and now removes the temp file); the bytes are identical to the old format (test `streamed_snapshot_keeps_the_on_disk_format`, which also loads a legacy snapshot), so no version bump. Also tested: `failed_streamed_write_keeps_the_old_file`. persist_bench, 85k photos (310 MB snapshot), compaction at the 2,000-record threshold, before/after binaries alternated in the same session (load avg ~130-170), building+compacting only (PERSIST_BENCH_NO_LOAD): compaction median: before 3.5 s / 4.6 s, after 1.9 s / 2.5 s peak RSS: before 926 MB / 1263 MB, after 308 MB / 366 MB 10k photos peak RSS: 149-235 MB -> 61-68 MB (times too noisy to compare) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-04 15:43:13 -04:00
Brandon ThomasandClaude Opus 5.5	db6242d2d3	Issue #37 : Catalog: instrument persistence (append, snapshot stages) and whole-frame update time - Journal::stats(): last/max append time (encode + write + sync_data), snapshot count, last snapshot by stage (serialise, write+sync+rename, log reset, bytes, records), slowest snapshot. Reported by `library.info` → `persistence`; printed per write to stderr under LIGHTCRAFT_PROFILE. - UI: `perf.frame_ms` only covers layout (`LightcraftApp::ui`); `logic()` (control channel, shortcuts, render polling, `persist_if_dirty`) ran untimed. Add `logicMs`, `updateMs` (logic + layout) and `maxUpdateMs` to `ui.inspect` → perf and a "Frame update" row to Help ▸ System Info. - `crates/catalog/examples/persist_bench.rs`: synthetic 1k/10k/85k-photo catalogs (90 % Local browse records) on FsStore: append latency, compaction at the 2,000-record threshold, load. Baseline (release, load avg ~240, so absolute times are inflated; median of 3 compactions): 1k photos (3.6 MB snap): append 7 ms, compaction 31 ms (serialise 5, write+sync 12, reset 14) 10k photos (36.5 MB): append 15 ms, compaction 528 ms (serialise 463, write+sync 56) 85k photos (310 MB): append 12 ms, compaction 4.6 s (serialise 2.2-4.4 s, write+sync 0.25-0.8 s); peak RSS 963 MB; load 4.7 s Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-04 15:43:13 -04:00

Author

SHA1

Message

Date

Brandon ThomasandClaude Opus 5.5

ae1cbbe87b

Issue #37 : Catalog: stream the snapshot into the temp file instead of building two strings

Journal::snapshot serialised the catalog into a String and then copied it into a second
`format!`-ed String before writing (2x the snapshot size in memory, ~620 MB at 85k photos).
It now streams `{"format":…,"seq":N,"catalog":` + serde_json::to_writer + `}\n` through a
1 MiB BufWriter into the store (new `Store::write_atomic_with`; FsStore writes the temp file
directly, then flush + sync_all + rename + dir fsync as before; other stores buffer, as before).

Durability and format are unchanged: same temp + fsync + atomic rename; a failed write leaves
the old snapshot (and now removes the temp file); the bytes are identical to the old format
(test `streamed_snapshot_keeps_the_on_disk_format`, which also loads a legacy snapshot), so
no version bump. Also tested: `failed_streamed_write_keeps_the_old_file`.

persist_bench, 85k photos (310 MB snapshot), compaction at the 2,000-record threshold,
before/after binaries alternated in the same session (load avg ~130-170), building+compacting
only (PERSIST_BENCH_NO_LOAD):
  compaction median: before 3.5 s / 4.6 s, after 1.9 s / 2.5 s
  peak RSS:          before 926 MB / 1263 MB, after 308 MB / 366 MB
  10k photos peak RSS: 149-235 MB -> 61-68 MB (times too noisy to compare)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-04 15:43:13 -04:00

Brandon ThomasandClaude Opus 5.5

db6242d2d3

Issue #37 : Catalog: instrument persistence (append, snapshot stages) and whole-frame update time

- Journal::stats(): last/max append time (encode + write + sync_data), snapshot count, last
  snapshot by stage (serialise, write+sync+rename, log reset, bytes, records), slowest
  snapshot. Reported by `library.info` → `persistence`; printed per write to stderr under
  LIGHTCRAFT_PROFILE.
- UI: `perf.frame_ms` only covers layout (`LightcraftApp::ui`); `logic()` (control channel,
  shortcuts, render polling, `persist_if_dirty`) ran untimed. Add `logicMs`, `updateMs`
  (logic + layout) and `maxUpdateMs` to `ui.inspect` → perf and a "Frame update" row to
  Help ▸ System Info.
- `crates/catalog/examples/persist_bench.rs`: synthetic 1k/10k/85k-photo catalogs (90 % Local
  browse records) on FsStore: append latency, compaction at the 2,000-record threshold, load.

Baseline (release, load avg ~240, so absolute times are inflated; median of 3 compactions):
  1k photos  (3.6 MB snap): append 7 ms, compaction 31 ms (serialise 5, write+sync 12, reset 14)
  10k photos (36.5 MB):     append 15 ms, compaction 528 ms (serialise 463, write+sync 56)
  85k photos (310 MB):      append 12 ms, compaction 4.6 s (serialise 2.2-4.4 s, write+sync
                            0.25-0.8 s); peak RSS 963 MB; load 4.7 s

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-04 15:43:13 -04:00