Skip to content

feat: strengthen the download and unpacking of the Cardano database archives - #3708

Open
jpraynaud wants to merge 6 commits into
mainfrom
jpraynaud/cardano-database-extraction-staging
Open

jpraynaud wants to merge 6 commits into
mainfrom
jpraynaud/cardano-database-extraction-staging

Conversation

@jpraynaud

@jpraynaud jpraynaud commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Content

This PR includes a strengthening of the download and unpacking of the Cardano database archives in the client and the client CLI:

  • Add a shared tar unpacker in mithril-client (TarArchiveUnpacker with UnpackedFilePermissions) that unpacks only the regular files and directories of an archive, and reports any other entry with a typed error
  • Give the unpacked files consistent permissions (fixed 0o644 for the database files, archived mode without group and other write for the binaries unpacked by the client CLI) and create directories with the default permissions
  • Unpack each immutable and ancillary archive in its own staging directory, check that its content is the expected one (exactly the immutable trio, or the files of the signed ancillary manifest), and move only these files to the target directory
  • Remove the staging directories when a download fails, is cancelled or panics, and clean up the ones left by an earlier download
  • Accept only plain relative paths in an ancillary manifest, when deserializing a manifest and when building one from paths, and verify the manifest signature before hashing the files it names
  • Update the location fallback: a local error that no other location can fix stops the task, any other failure tries the next location, and the last failure is kept as the cause of the final error
  • Skip the uncompressed immutable and ancillary locations with a warning
  • Fail before any download when the target directory does not exist or is not a directory
  • Support a target directory whose subdirectories are on another filesystem, by copying the downloaded files when they can not be moved
  • Update the client CLI tar.gz unpacker to use the shared unpacker

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

@jpraynaud jpraynaud self-assigned this Nov 13, 2026
@jpraynaud jpraynaud added the optimization 🛠️ Optimization and/or small enhancements label Nov 13, 2026
@jpraynaud
jpraynaud requested a lite review from Copilot November 13, 2026 18:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Approval is blocked by unresolved critical security and atomicity issues in archive permission sanitization and multi-file promotion, plus a public error-type API gap.

4 open issues
What changed in this PR

This PR strengthens Cardano database archive downloads by safely unpacking untrusted tar archives, validating archive contents before promotion, improving staging and cleanup behavior, and supporting cross-filesystem file moves. It also applies the shared unpacking logic to the client CLI and validates ancillary manifest paths more strictly.

Changes:

  • Adds a shared tar unpacker with path, entry-type, and permission controls.
  • Adds per-download staging, archive-content validation, cleanup, location fallback, and cross-filesystem promotion.
  • Hardens ancillary manifest validation and updates the CLI tar.gz unpacker.
File Description
mithril-client/​src/​utils/​mod.rs Exposes ancillary verification helpers needed by download-task tests and logic.
mithril-client/​src/​utils/​fs.rs Adds move-with-cross-filesystem-copy support and related tests.
mithril-client/​src/​utils/​ancillary_verifier.rs Verifies signatures before file hashing and promotes validated ancillary files.
mithril-client/​src/​file_downloader/​streaming.rs Uses the shared tar unpacker for streamed database archives and improves stream and unpack error handling.
mithril-client/​src/​file_downloader/​mod.rs Exports the shared archive unpacker types.
mithril-client/​src/​file_downloader/​mock_builder.rs Adds mock behavior for creating immutable archive contents in staging directories.
mithril-client/​src/​file_downloader/​archive_unpacker.rs Introduces secure tar extraction, path checks, entry filtering, permissions, and extensive tests.
mithril-client/​src/​cardano_database_client/​download_unpack/​internal_downloader.rs Validates targets, skips uncompressed locations, manages staging cleanup, and coordinates task shutdown.
mithril-client/​src/​cardano_database_client/​download_unpack/​download_task.rs Adds staged downloads, archive validation, promotion, fallback behavior, and cancellation cleanup.
mithril-client/​src/​cardano_database_client/​api.rs Documents the single-download-per-target-directory staging constraint.
mithril-client-cli/​src/​utils/​archive_unpacker/​tar_gz_unpacker.rs Replaces the CLI-specific tar unpacking implementation with the shared unpacker.
mithril-client-cli/​Cargo.toml Moves the tar dependency to the CLI's development dependencies.
internal/​cardano-node/​mithril-cardano-node-internal-database/​src/​entities/​ancillary_files_manifest.rs Restricts manifest paths to plain relative paths during deserialization and manifest construction.

Review effort: Lite

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Test Results

     5 files  ±  0     211 suites  ±0   1h 1m 0s ⏱️ - 52m 54s
 4 491 tests + 38   4 491 ✅ + 38  0 💤 ±0  0 ❌ ±0 
14 675 runs  +315  14 675 ✅ +315  0 💤 ±0  0 ❌ ±0 

Results for commit cd4ba13. ± Comparison against base commit 1a3c425.

This pull request removes 61 and adds 99 tests. Note that renamed tests count towards both.
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::index_out_of_bounds
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::index_too_large_for_circuit_range
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::indices_not_increasing
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::leaf_merkle_path_mismatch
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::leaf_swap_keep_merkle_path
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::leaf_wrong_verification_key
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_corrupt_sibling
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_flip_position
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_length_long
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_length_short
…
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::can_deserialize_a_manifest_with_a_nested_relative_path
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::cannot_deserialize_a_manifest_with_a_current_directory_path
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::cannot_deserialize_a_manifest_with_a_nested_parent_directory_path
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::cannot_deserialize_a_manifest_with_a_parent_directory_path
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::cannot_deserialize_a_manifest_with_an_absolute_path
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::cannot_deserialize_a_manifest_with_an_empty_path
mithril-cardano-node-internal-database ‑ entities::ancillary_files_manifest::tests::ancillary_files_manifest::from_paths_rejects_a_path_outside_the_base_directory
mithril-client ‑ cardano_database_client::download_unpack::download_task::tests::download_unpack_ancillary_file::a_panicking_download_removes_the_ancillary_staging_directory
mithril-client ‑ cardano_database_client::download_unpack::download_task::tests::download_unpack_ancillary_file::aborting_the_download_future_removes_the_ancillary_staging_directory
mithril-client ‑ cardano_database_client::download_unpack::download_task::tests::download_unpack_ancillary_file::falls_back_to_the_next_location_when_the_ancillary_promotion_fails_with_another_io_error_kind
…

♻️ This comment has been updated with latest results.

@jpraynaud
jpraynaud force-pushed the jpraynaud/cardano-database-extraction-staging branch from f6d8495 to 71e74f9 Compare November 16, 2026 10:26
@jpraynaud
jpraynaud deployed to testing-preview November 16, 2026 10:47 — with GitHub Actions Active
@jpraynaud
jpraynaud deployed to testing-2-preview November 16, 2026 10:47 — with GitHub Actions Active
@jpraynaud
jpraynaud force-pushed the jpraynaud/cardano-database-extraction-staging branch from 71e74f9 to 374b74b Compare November 16, 2026 16:34
@jpraynaud
jpraynaud marked this pull request as ready for review November 16, 2026 16:36
@jpraynaud
jpraynaud deployed to testing-preview November 16, 2026 16:47 — with GitHub Actions Active
@jpraynaud
jpraynaud deployed to testing-2-preview November 16, 2026 16:47 — with GitHub Actions Active

@damrobi damrobi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍
I left a few small suggestions

Comment thread mithril-client/src/utils/fs.rs Resolved

@turmelclem turmelclem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good to me 👍

Never set the setuid, setgid and sticky bits with fixed unpack permissions.
Pin the tests of the staging cleanup, of the stream failure and of the partial copy removal.
@jpraynaud
jpraynaud deployed to testing-2-preview November 19, 2026 17:00 — with GitHub Actions Active
@jpraynaud
jpraynaud deployed to testing-preview November 19, 2026 17:00 — with GitHub Actions Active

This branch was successfully deployed

2 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimization 🛠️ Optimization and/or small enhancements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants