Repository navigation
feat(core): bootstrap records from record packs with the cursor-exact manifest reconcile - #5268
Merged
Merged
Conversation
… manifest reconcile
Collaborator
Author
|
Review (REVIEW-5268) on 6840976: FIX-FIRST Two probes are red. No data is lost, because the reconcile catches finding 1, but the ceiling breaks.
Clean: stale packed record pulled, writes during bootstrap, tombstones and re-creates, same apply path, the tail starts at coverThrough+1, kill/resume done-set ordering, #5247 lows, rule 7 claim, line ceilings. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A record pack ending on an undeclared or non-record entry skipped its final flush. The stale-row removal now rechecks local changes inside the delete transaction, and the stale filter and cursor forget are keyed on (type, id). The test fake lists snapshot packs, paged by 50.
Collaborator
Author
|
Verify (VERIFY-5268) on b12091b: VERIFIED Real Worker test (harness miniflare D1/R2): the real core FirstSync ran over the CLI NativeTransport.
Request counts: 8 requests (P=3) and 11 requests (P=4 plus edits).
Lows (collector, step 09 PR 2):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #5221
Refs #5232
Step 09, PR 1 of 2: records. A fresh core device now bootstraps record metadata from record packs (chapter 08 §8.4.2). The feed is read from where the packs end, and one cursor-exact manifest reconcile (§8.10) settles the rest, before any body is fetched. Bodies, the core oracle scenario and the staging bench come in PR 2 (decision D-BODYMANIFEST in the epic log: a separate
GET /sync/crdt/manifestroute is built first).What changes
GET /sync/packslists every pack kind plus the first page'scoverage.tail_startimplements the §8.10 walk. The snapshotrun()filters tocrdt_snapshot, so its behavior is unchanged.fetch_ids(pull/shared_apply.rs).sortKey-1. Undeclared types are skipped, as/sync/pullwould never serve them.record_ref, which keeps a held row'sserver_cursor. A ref names an item, not a write.GET /sync/manifest:blobMissing.serverCursorin the manifest, every item is pulled.Request ceiling
A covered, unchanged vault of N records with P listed packs of every kind (record and snapshot) needs:
That is two listing walks of every kind (records, then snapshots; each walk pages all P packs, so 230 snapshot packs on staging is 5 pages per walk), P downloads (record and snapshot), one feed page from the coverage boundary, and one manifest. The formula has no N term.
the_request_count_follows_the_packs_and_not_the_itemsasserts the same count at N=40 and N=4000, and at 3 record + 60 snapshot packs (two pages per walk,ceiling(63)). The two walks stay separate: the snapshot pass and the record pass are separate phases that resume independently after a kill, and a listing's presigned URLs expire, so sharing one listing is not cheap.Proof
cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p memry-core --no-fail-fastcargo test -p memry-core --test sync_record_packsnode scripts/check-line-ceilings.mjspull.rs630 -> 534;pull/shared_apply.rs112pnpm lint && pnpm typecheck && pnpm format:check && git diff --checkpnpm test:sync-harnessbootstrap-zero-loss.test.ts. That suite is TS-only and has no core path, so this is a flake outside this diff.record_refON CONFLICT also setsserver_cursor = NULLpacks_without_coverage_tail_from_zero_and_still_save_every_pullgoes red. Restored.sync_record_packsgo redapi_vault_sync.rschanges only its script. The scripted transport answers in order, and the first sync now asks for the record packs listing (a 404 on an old server, which means "no packs") before the refs page.Rule 7 (seed demotion) on main core: no hole
On feat/tauri-v2 (#5052), a failed metadata pass pushed a demoted, clockless seed. Main core cannot reach that state:
crates/memry-core/src/domain/projects/write.rs:157ensure_inbox, theninsert, which usesnext_clockandinit_all_field_clocks. It is written only when no Inbox row exists, tombstones included. Noclock IS NULLseeder exists incrates/memry-core(grep).INSERT ... ON CONFLICT DO NOTHING(domain/settings.rs:192).sync_items::upsert_metadata_only(storage/repositories/sync_items.rs:103) updates onlyserver_cursorandupdated_aton conflict.record_ref(sync/first_sync_store.rs:107) updates onlyupdated_at.payload_stateand the payload's clock are never changed.protocol/account/seal.rs:388test).server_cursor). A seed has a NULL cursor.No
tag_mintexists on main. Nothing to fix here.Review checklist
REFERENCE_IMPL. The reference is Bootstrap step 09: Core: bootstrap from packs for every kind, then tail the feed #5232's acceptance and chapter 08 §8.4.2, §8.10 and §8.12. Bodies are deferred to PR 2 by D-BODYMANIFEST.a_kill_in_each_record_phase_resumes_without_downloading_a_finished_pack_againkills the run in the pack download, the changes, the manifest and the pull phases.JoinSet(records.rs:128). It is drained byjoin_next, and dropping it on an early return aborts its tasks. No new channels or oneshots.a_pass_asked_for_during_the_first_sync_waits_for_it). Pack downloads run in parallel, but every apply goes through the singleDbactor.an_entry_that_does_not_verify...).a_corrupt_pack_and_a_pack_gone...).sync_record_packs.rsdrives a route-level fake server through the realFirstSync.Metadataphase, so there are no binding changes.Departures and assumptions
coverFrom/coverage, so T=0 and the refs pass reads the whole feed. That is spec-correct, and it is whyrecord_refmust keep held cursors. The ceiling above holds once coverage ships.purged_tombstones. A shed tombstone therefore ends as a local tombstone with no clock. This is the conservative choice: the row was gone server-side, and nothing is pushed for it.FirstSyncError::Pull. Its done marker is still unwritten, so the resume runs the reconcile again (the kill-in-manifest case). It never skips the reconcile.Touched files (for step 12, the Tauri port)
crates/memry-core/src/sync/pack_bootstrap.rs: module wiring; the snapshotrun()kind filtercrates/memry-core/src/sync/pack_bootstrap/listing.rs:list:62,tail_start:168crates/memry-core/src/sync/pack_bootstrap/records.rs:run_records:86,apply_record_entries:212,rebuild_item:307crates/memry-core/src/sync/first_sync.rs:run()order :253,record_packs_pass:304,refs_pass:363crates/memry-core/src/sync/first_sync/refs.rs: refs page reading, split out offirst_sync.rscrates/memry-core/src/sync/first_sync/reconcile.rs:reconcile:48,remove_unanswered:114,plan:164crates/memry-core/src/sync/first_sync_store.rs:record_ref:107,held_rows:136,forget_cursors:165crates/memry-core/src/sync/pull.rs:apply_itemsshared by the page and by-id pathscrates/memry-core/src/sync/pull/shared_apply.rs:fetch_ids:24,apply_pack_items:57,apply_decoded:69crates/memry-core/tests/sync_record_packs.rs: route-level fake server, 9 testscrates/memry-core/tests/api_vault_sync.rs: script order onlyOpen in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Fold of REVIEW-5268
an_undeclared_last_entry_does_not_drop_the_chunk_before_it(red: entries_applied 0, want 3). One flush site now runs whenever the chunk is non-empty and full or at the pack's last entry.a_row_edited_locally_during_the_reconcile_is_not_tombstoned(red: removed 1, want 0).remove_unansweredreads held rows inside its delete transaction and skipslocal_change.a_shed_row_sharing_an_id_with_another_types_pulled_item_is_removed(red: note:x1 stayed live). The id-keyedseenfilter is gone (a stale row is unlisted, so it never overlaps a wanted listed key), andforget_cursorsis keyed on (type, id): the id-keyed forget also cleared the shed row's cursor and let it escape removal.