Skip to content

feat(core): bootstrap records from record packs with the cursor-exact manifest reconcile - #5268

Merged
h4yfans merged 5 commits into
mainfrom
feat/5232-core-pack-bootstrap
Nov 20, 2026
Merged

h4yfans merged 5 commits into
mainfrom
feat/5232-core-pack-bootstrap

Conversation

@h4yfans

@h4yfans h4yfans commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #5221
Refs #5232

Step 09, PR 1 of 2: records. A fresh core device now bootstraps record metadata from record packs (chapter 08 §8.4.2). The feed is read from where the packs end, and one cursor-exact manifest reconcile (§8.10) settles the rest, before any body is fetched. Bodies, the core oracle scenario and the staging bench come in PR 2 (decision D-BODYMANIFEST in the epic log: a separate GET /sync/crdt/manifest route is built first).

What changes

  • GET /sync/packs lists every pack kind plus the first page's coverage. tail_start implements the §8.10 walk. The snapshot run() filters to crdt_snapshot, so its behavior is unchanged.
  • Record packs:
    • 3 parallel downloads.
    • Each entry is rebuilt as a pull item and goes through the same decode and apply as fetch_ids (pull/shared_apply.rs).
    • Chunks are cut at 250 entries or 4 MiB, whichever comes first.
    • An unusable entry lowers the tail to sortKey-1. Undeclared types are skipped, as /sync/pull would never serve them.
    • Finished packs are recorded durably, so a resumed run does not download them again.
  • First sync order:
    1. The record packs pass runs only when there is no record cursor. It writes the cursor T and the "packs used" marker in one transaction.
    2. The refs pass runs from T.
    3. When packs were used, the reconcile runs.
    4. The metadata pass runs.
    5. Snapshots, then bodies, as before.
  • The refs pass writes through record_ref, which keeps a held row's server_cursor. A ref names an item, not a write.
  • Reconcile, in one unpaged GET /sync/manifest:
    • An exact cursor match counts as held.
    • Anything else is pulled by id. Cursors are forgotten before that pull, which also covers blobMissing.
    • With no serverCursor in the manifest, every item is pulled.
    • A pack-held live row the manifest does not list is pulled by id. If it is still live at its old cursor after the pull, it gets a local tombstone with no clock. Rows with outbox entries are never touched, rechecked in the transaction that deletes. Both the stale filter and the cursor forget before the by-id pull are keyed on (type, id), so a listed item of another type sharing the id does not shield a shed row.

Request ceiling

A covered, unchanged vault of N records with P listed packs of every kind (record and snapshot) needs:

requests <= 2 * ceil(P / 50) + P + 2

That is two listing walks of every kind (records, then snapshots; each walk pages all P packs, so 230 snapshot packs on staging is 5 pages per walk), P downloads (record and snapshot), one feed page from the coverage boundary, and one manifest. The formula has no N term. the_request_count_follows_the_packs_and_not_the_items asserts the same count at N=40 and N=4000, and at 3 record + 60 snapshot packs (two pages per walk, ceiling(63)). The two walks stay separate: the snapshot pass and the record pass are separate phases that resume independently after a kill, and a listing's presigned URLs expire, so sharing one listing is not cheap.

Proof

Command Result
cargo fmt --all --check ok
cargo clippy --workspace --all-targets -- -D warnings ok
cargo test -p memry-core --no-fail-fast 1805 passed, 0 failed, 153 binaries
cargo test -p memry-core --test sync_record_packs 9 passed
node scripts/check-line-ceilings.mjs passed. pull.rs 630 -> 534; pull/shared_apply.rs 112
pnpm lint && pnpm typecheck && pnpm format:check && git diff --check ok
pnpm test:sync-harness 57/57 on 2 consecutive runs. One earlier run had 1 red in bootstrap-zero-loss.test.ts. That suite is TS-only and has no core path, so this is a flake outside this diff.
Mutation: record_ref ON CONFLICT also sets server_cursor = NULL packs_without_coverage_tail_from_zero_and_still_save_every_pull goes red. Restored.
Mutation: the reconcile call removed 4 tests in sync_record_packs go red

api_vault_sync.rs changes only its script. The scripted transport answers in order, and the first sync now asks for the record packs listing (a 404 on an old server, which means "no packs") before the refs page.

Rule 7 (seed demotion) on main core: no hole

On feat/tauri-v2 (#5052), a failed metadata pass pushed a demoted, clockless seed. Main core cannot reach that state:

  • The Inbox seed is a clocked create with its outbox row in the same write: crates/memry-core/src/domain/projects/write.rs:157 ensure_inbox, then insert, which uses next_clock and init_all_field_clocks. It is written only when no Inbox row exists, tombstones included. No clock IS NULL seeder exists in crates/memry-core (grep).
  • The settings seed is INSERT ... ON CONFLICT DO NOTHING (domain/settings.rs:192).
  • Neither refs-pass writer demotes an existing row:
    • sync_items::upsert_metadata_only (storage/repositories/sync_items.rs:103) updates only server_cursor and updated_at on conflict.
    • record_ref (sync/first_sync_store.rs:107) updates only updated_at. payload_state and the payload's clock are never changed.
  • A row that is still metadata-only is refused at seal time rather than pushed empty (protocol/account/seal.rs:388 test).
  • The new reconcile never tombstones a row with an outbox entry, and it only considers pack-held rows (non-null server_cursor). A seed has a NULL cursor.

No tag_mint exists on main. Nothing to fix here.

Review checklist

  1. Parity. No REFERENCE_IMPL. The reference is Bootstrap step 09: Core: bootstrap from packs for every kind, then tail the feed #5232's acceptance and chapter 08 §8.4.2, §8.10 and §8.12. Bodies are deferred to PR 2 by D-BODYMANIFEST.
  2. Partial failure. Each pack's apply is one transaction. The done set is durable per pack. The cursor T and the "packs used" marker are written in one transaction. The reconcile writes its done marker last. a_kill_in_each_record_phase_resumes_without_downloading_a_finished_pack_again kills the run in the pack download, the changes, the manifest and the pull phases.
  3. Waiters. The downloads run in a tokio JoinSet (records.rs:128). It is drained by join_next, and dropping it on an early return aborts its tasks. No new channels or oneshots.
  4. Races. First sync is already exclusive: a pass asked for during it waits (a_pass_asked_for_during_the_first_sync_waits_for_it). Pack downloads run in parallel, but every apply goes through the single Db actor.
  5. Platforms. No platform-conditional code.
  6. Untrusted input.
    • Each entry is verified by the same decode as a pull, so a forged entry is pulled again and not recorded corrupt (an_entry_that_does_not_verify...).
    • A corrupt pack or a 404 is a gap, not a failure (a_corrupt_pack_and_a_pack_gone...).
    • Chunks are bounded at 250 entries or 4 MiB.
  7. Tests. The two mutations above. sync_record_packs.rs drives a route-level fake server through the real FirstSync.
  8. PR body. This one. Departures below.
  9. Whole workspace. Workspace clippy is green. The UniFFI surface is unchanged: record progress is reported as the existing Metadata phase, so there are no binding changes.

Departures and assumptions

  • Today's server writes no coverFrom/coverage, so T=0 and the refs pass reads the whole feed. That is spec-correct, and it is why record_ref must keep held cursors. The ceiling above holds once coverage ships.
  • Core does not declare purged_tombstones. A shed tombstone therefore ends as a local tombstone with no clock. This is the conservative choice: the row was gone server-side, and nothing is pushed for it.
  • A listing error or 404 means "no record packs", and the item-granular path finishes the sync. A manifest error fails the first sync with FirstSyncError::Pull. Its done marker is still unwritten, so the resume runs the reconcile again (the kill-in-manifest case). It never skips the reconcile.

Touched files (for step 12, the Tauri port)

  • crates/memry-core/src/sync/pack_bootstrap.rs: module wiring; the snapshot run() kind filter
  • crates/memry-core/src/sync/pack_bootstrap/listing.rs: list :62, tail_start :168
  • crates/memry-core/src/sync/pack_bootstrap/records.rs: run_records :86, apply_record_entries :212, rebuild_item :307
  • crates/memry-core/src/sync/first_sync.rs: run() order :253, record_packs_pass :304, refs_pass :363
  • crates/memry-core/src/sync/first_sync/refs.rs: refs page reading, split out of first_sync.rs
  • crates/memry-core/src/sync/first_sync/reconcile.rs: reconcile :48, remove_unanswered :114, plan :164
  • crates/memry-core/src/sync/first_sync_store.rs: record_ref :107, held_rows :136, forget_cursors :165
  • crates/memry-core/src/sync/pull.rs: apply_items shared by the page and by-id paths
  • crates/memry-core/src/sync/pull/shared_apply.rs: fetch_ids :24, apply_pack_items :57, apply_decoded :69
  • crates/memry-core/tests/sync_record_packs.rs: route-level fake server, 9 tests
  • crates/memry-core/tests/api_vault_sync.rs: script order only

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Fold of REVIEW-5268

  • Finding 1: an_undeclared_last_entry_does_not_drop_the_chunk_before_it (red: entries_applied 0, want 3). One flush site now runs whenever the chunk is non-empty and full or at the pack's last entry.
  • Finding 2: a_row_edited_locally_during_the_reconcile_is_not_tombstoned (red: removed 1, want 0). remove_unanswered reads held rows inside its delete transaction and skips local_change.
  • Finding 3: a_shed_row_sharing_an_id_with_another_types_pulled_item_is_removed (red: note:x1 stayed live). The id-keyed seen filter is gone (a stale row is unlisted, so it never overlaps a wanted listed key), and forget_cursors is keyed on (type, id): the id-keyed forget also cleared the shed row's cursor and let it escape removal.
  • Finding 4: the fake lists snapshot packs and pages at 50; formula above restated with P = every listed pack.

@github-actions github-actions Bot added enhancement New feature or request test labels Nov 20, 2026
@h4yfans

h4yfans commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Review (REVIEW-5268) on 6840976: FIX-FIRST

Two probes are red. No data is lost, because the reconcile catches finding 1, but the ceiling breaks.

  1. records.rs:229-245 medium — When a pack's last entry is an undeclared or non-record type, continue skips the end-of-pack flush. Up to 249 verified entries are dropped silently and the pack is still marked done. The reconcile pulls them back, which costs ceil(k/100) requests that grow with N. Probe: review_5268_an_undeclared_last_entry_does_not_drop_the_chunk_before_it.
  2. reconcile.rs:119-145 low — remove_unanswered does not recheck local_change, so a row edited during the stale pull round trip is still tombstoned (breaks §8.10 rule 2). Probe: review_5268_a_row_edited_locally_during_the_reconcile_is_not_tombstoned.
  3. reconcile.rs:192-201 low — seen is keyed on id, not (type, id). A shed type-A row that shares an id with a live type-B manifest item stays live.
  4. PR body and sync_record_packs.rs:403 low — Each listing walk pages every kind, so P is all listed packs (230 snapshot packs on staging is 5 pages). The fake lists only record packs on one page.

Clean: stale packed record pulled, writes during bootstrap, tombstones and re-creates, same apply path, the tail starts at coverThrough+1, kill/resume done-set ordering, #5247 lows, rule 7 claim, line ceilings.

@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

A record pack ending on an undeclared or non-record entry skipped its
final flush. The stale-row removal now rechecks local changes inside the
delete transaction, and the stale filter and cursor forget are keyed on
(type, id). The test fake lists snapshot packs, paged by 50.
@h4yfans

h4yfans commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Verify (VERIFY-5268) on b12091b: VERIFIED

Real Worker test (harness miniflare D1/R2): the real core FirstSync ran over the CLI NativeTransport.

  • Seed: 700 records, with undeclared types at both ends of the packs.
  • Unchanged vault: 695/695 live, every cursor exact, no extras, 0 reconcile pulls.
  • After post-pack edits: an update, a delete+re-create before and after the build, a deleted tombstone and a shed tombstone. Result: 693/693, no extras, no duplicates, 0 cursor mismatches.
  • Kill at 9 points (listing, each download, changes, manifest, each pull): every resume ended 693/693, and no finished pack was downloaded again.
  • The three fold probes: red at 6840976, green at head.
  • Gates: memry-core 1808, clippy, fmt, line ceilings, harness 57 ×3.

Request counts: 8 requests (P=3) and 11 requests (P=4 plus edits).

Lows (collector, step 09 PR 2):

  1. first_sync_store.rs:76-78 — A by-id pull clears cursors of every type under that id (server /sync/pull filters on id only). Either keep (type, id) when applying, or drop the claim. Cost: one extra pull after a kill.
  2. PR body — The ceiling depends on Bootstrap step 07: Sync server: guaranteed pack coverage in the bootstrap response #5230.

@h4yfans
h4yfans merged commit 2737110 into main Nov 20, 2026
21 checks passed
@h4yfans
h4yfans deleted the feat/5232-core-pack-bootstrap branch November 20, 2026 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant