Skip to content

Fix backup consistency after concurrent TRUNCATE. - #145

Open
woblerr wants to merge 1 commit into
apache:mainfrom
woblerr:snapshot_lock_retry
Open

woblerr wants to merge 1 commit into
apache:mainfrom
woblerr:snapshot_lock_retry

Conversation

@woblerr

@woblerr woblerr commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Problem

gpbackup takes its snapshot before acquiring table locks. Concurrent TRUNCATE can replace storage while it waits, so backup and restore report success but restore an empty table, hiding data loss.

Reproduction

  1. Create a distributed heap table containing {1,2,3} on Cloudberry.
  2. In another transaction, run TRUNCATE and insert {4,5,6} without committing.
  3. Start gpbackup, wait until pg_locks shows the backup waiting for AccessShareLock on this table, then commit the writer.
  4. Restore to another database: both commands exit 0, but the restored table is empty.

Fix

  • On Cloudberry, validate table storage and identity after locking, including descendant partitions.
  • Add --snapshot-attempts (default 3) to retry with a fresh shared snapshot and fail the backup when attempts are exhausted.
  • Re-resolve table filters on each retry and disable heap file hashes for that backup to avoid skipping heap data.

A concurrent TRUNCATE can replace table storage after the backup
snapshot but before table locks are acquired. Backup and restore can
then report success while restoring an empty table.

On Cloudberry, validate storage and table identity after locking,
including descendant partitions. Add --snapshot-attempts (default 3) to
retry with a fresh shared snapshot, rebuild filters, and discard heap
file hashes. Fail the backup when the limit is reached.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant