Skip to content

feat(client): managed kubo node for database download - #3717

Open
Alenar wants to merge 24 commits into
mainfrom
djo/3655/client-managed-kubo-node
Open

Alenar wants to merge 24 commits into
mainfrom
djo/3655/client-managed-kubo-node

Conversation

@Alenar

@Alenar Alenar commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Content

This PR adds a managed IPFS node mode to cardano-db download: with the new --ipfs-managed-node option, the client downloads, runs and cleans up its own Kubo node, so users don't need to install and configure one to download a Cardano database through IPFS. Bringing your own node with --ipfs-rpc-url is still supported, both options are mutually exclusive.

Managed Kubo node (mithril-client-cli/src/utils/kubo_node)

  • Pinned distribution: Kubo v0.43.1, with the SHA-256 of the release archive of each supported platform: Linux x86_64 and aarch64, macOS aarch64, Windows x86_64.
  • Archive provider: the release archive is downloaded from GitHub on first use (optional GITHUB_TOKEN environment variable to avoid the API rate limit, read from the environment only so that the token doesn't appear in the process list), and cached in the user cache directory (~/.cache/mithril/kubo/v0.43.1/).
    • The checksum of the cached archive is verified at each start, an invalid archive is downloaded again.
    • The install is serialized between concurrent clients with a lock file, and done in a staging directory renamed once verified, so the cache never holds a partial archive.
  • Node lifecycle: at each start, the binary is unpacked in a new repository directory, unique per run (mithril-client-kubo-<pid>-<timestamp>), created in the system temporary directory and only accessible to the current user on Unix (0700), so another user of a shared /tmp can't tamper with the binary run from it. The node is initialized, configured with a single edit of its config file, then its daemon is started and the client waits for its RPC API to answer.
    • When the download ends, the daemon is asked to shut down (killed after a timeout), and its repository is removed.
    • On failure or cancellation of the start, the started processes are killed and the repository is removed.
    • If the daemon doesn't exit in time after being killed, or the repository can't be removed, an error is printed instead of silently leaving them behind.
  • Node configuration, for a download-only node:
    • RPC API on 127.0.0.1 with a random port, no gateway.
    • RPC API protected by a basic authentication with a random secret generated at each run, so another user of the host can't control the node. The secret is written in the repository config file (never on a command line), given to the downloader in the RPC URL credentials, and removed from the errors of the RPC calls.
    • Swarm doesn't listen for inbound connections (not needed by a download-only node, and avoids a firewall prompt on each run on Windows and macOS), and dials its peers on TCP only (no QUIC, WebTransport, WebRTC, relays or AutoTLS).
    • autoclient routing, no content providing, Bitswap server disabled, telemetry off.
    • Datastore bounded to 2 GB with a GC every 15 s, so a full database download doesn't fill the disk.
    • The Kubo environment variables of the client (IPFS_*, GOLOG_*, LIBP2P_*, matched case-insensitively as on Windows) are not inherited by the node.

Shutdown signals

  • All commands now stop on Ctrl-C/SIGINT or SIGTERM, so the managed node is stopped and its repository removed when a download is interrupted.
  • The runtime is shut down with a bounded timeout, and the async logger is flushed before exiting.

Fixes found along the way

  • CLI HTTP downloader: it now fails on an error response status (a 404 used to surface as an invalid checksum).
  • CLI HTTP downloader and GitHub API client: they now share a reqwest client builder with a connect timeout (10 s) and a read timeout (30 s), so a stalled server fails the request instead of hanging it. This applies to the snapshot converter downloads too.
  • JSON warnings: their message is now escaped, it produced invalid JSON when it contained quotes or backslashes (e.g. Windows paths).
  • mithril-client library: the IPFS downloader now bypasses the system proxy when the RPC URL is a loopback address (localhost, IPv4, IPv6 or IPv4-mapped IPv6 loopback). With HTTP(S)_PROXY set, the calls to a local Kubo node went through the proxy and failed.
  • Snapshot converter: its --github-token option is deprecated, as the token appears in the process list: it prints a deprecation warning when used, and no longer reads GITHUB_TOKEN itself (the environment variable is read directly when the option is not set, and its value no longer appears in --help).

CI

  • The Windows static binaries check allows the shell32.dll and combase.dll system DLLs, imported by the dirs crate used to locate the user cache directory.

Documentation

  • The client node page of the website has a new "IPFS download" section (managed node and your own node), and documents the ipfs_rpc_url and ipfs_managed_node parameters, and the GITHUB_TOKEN environment variable.
  • The page also gets the missing cardano-block sections, and the mentions of the removed v1 backend are removed.
  • CHANGELOG entries under UNSTABLE › IPFS, and under DEPRECATED for the --github-token option of the snapshot converter.

Manual checks

On testing-preview, full snapshot (≈15.5 GB), release build:

Run Result Time Node repository
Cloud storage only verified 475 s –
--ipfs-managed-node --ipfs-share 100 verified, 0 errors 481 s max 2.02 GB (GC keeps it bounded)
Same, SIGINT after 2 min exit 0.6 s after the signal – no repository, process or temp files left

The loopback proxy bypass was checked with a Squid proxy in Docker.

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • Update documentation website (if relevant)
    • No new TODOs introduced

Comments

  • The managed node tests that run a real Kubo binary are #[ignore]d. Run them with cargo test -p mithril-client-cli -- --include-ignored.
  • With a plain cargo test, the tests running a fake Kubo binary can rarely fail with Text file busy: another test thread spawning a process can briefly hold the just unpacked binary open for writing. cargo nextest, used by the CI, runs each test in its own process and is not affected.
  • The TEMP(DO NOT MERGE) commit adds a step to test-client.yml that runs 10 managed node downloads on each OS, to check whether an antivirus lock on the Kubo files makes the node fail on Windows. It will be dropped before merge.
  • On dev-mainnet, IPFS downloads currently fall back to cloud storage: the aggregator's Kubo node doesn't keep its root CIDs announced (its provider queue is saturated). This is an infra issue, followed separately.

Issue(s)

Closes #3655

@Alenar Alenar self-assigned this Nov 18, 2026
@Alenar Alenar added the feature 🚀 New implemented feature label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Test Results

     5 files  ±  0     211 suites  ±0   59m 19s ⏱️ - 1h 3m 47s
 4 460 tests +  7   4 460 ✅ +  7  0 💤 ±0  0 ❌ ±0 
14 564 runs  +204  14 564 ✅ +204  0 💤 ±0  0 ❌ ±0 

Results for commit c3c09ff. ± Comparison against base commit e50bd4e.

This pull request removes 61 and adds 68 tests. Note that renamed tests count towards both.
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::index_out_of_bounds
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::index_too_large_for_circuit_range
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::indices_not_increasing
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::leaf_merkle_path_mismatch
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::leaf_swap_keep_merkle_path
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::leaf_wrong_verification_key
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_corrupt_sibling
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_flip_position
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_length_long
mithril-stm ‑ circuits::halo2::tests::golden::cases::negative::slow::merkle_path_length_short
…
mithril-client ‑ utils::url::tests::loopback_hosts_are_loopback
mithril-client ‑ utils::url::tests::other_hosts_are_not_loopback
mithril-client-cli ‑ commands::cardano_db::download::tests::ipfs_managed_node::conflicts_with_ipfs_rpc_url
mithril-client-cli ‑ commands::cardano_db::download::v2::tests::ipfs_managed_node_requires_unstable
mithril-client-cli ‑ commands::tools::utxo_hd::snapshot_converter::tests::snapshot_conversion_cancellation::kills_the_converter_when_cancelled
mithril-client-cli ‑ utils::http_client::tests::request_fails_when_the_server_does_not_send_data_within_the_read_timeout
mithril-client-cli ‑ utils::http_downloader::reqwest::tests::downloads_the_file_content
mithril-client-cli ‑ utils::http_downloader::reqwest::tests::fails_without_writing_a_file_when_the_response_status_is_an_error
mithril-client-cli ‑ utils::kubo_node::archive::tests::fails_when_the_archive_can_not_be_unpacked
mithril-client-cli ‑ utils::kubo_node::archive::tests::fails_when_the_binary_is_not_in_the_archive
…

♻️ This comment has been updated with latest results.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two unresolved issues remain: inaccurate guidance for the managed IPFS mode and cleanup that can leave a Kubo process and temporary repository behind after interruption.

2 open issues
What changed in this PR

Adds an unstable managed Kubo-node mode for cardano-db download, including platform-pinned binary caching, node lifecycle management, signal-aware cleanup, and IPFS proxy handling.

Changes:

  • Adds managed Kubo distribution, checksum verification, caching, configuration, startup, shutdown, and cleanup.
  • Adds --ipfs-managed-node and --github-token, with mutual exclusion against --ipfs-rpc-url.
  • Adds shutdown signal handling, bounded runtime cleanup, HTTP client timeouts, improved status handling, JSON warning escaping, documentation, and CI coverage.
File Description
mithril-client/​src/​utils/​url.rs Adds loopback URL detection for proxy bypassing.
mithril-client/​src/​file_downloader/​ipfs.rs Disables system proxies for loopback Kubo nodes.
mithril-client-cli/​src/​utils/​shutdown_signal.rs Adds Ctrl-C and SIGTERM cancellation helpers.
mithril-client-cli/​src/​utils/​mod.rs Exposes new HTTP, Kubo, shutdown, and warning utilities.
mithril-client-cli/​src/​utils/​kubo_node/​node_config.rs Defines managed Kubo node configuration and runtime settings.
mithril-client-cli/​src/​utils/​kubo_node/​mod.rs Registers the managed Kubo node modules.
mithril-client-cli/​src/​utils/​kubo_node/​managed_node.rs Implements Kubo unpacking, startup, readiness checks, shutdown, and cleanup.
mithril-client-cli/​src/​utils/​kubo_node/​distribution.rs Defines supported pinned Kubo distributions and checksums.
mithril-client-cli/​src/​utils/​kubo_node/​archive.rs Unpacks and validates Kubo archives.
mithril-client-cli/​src/​utils/​kubo_node/​archive_provider.rs Downloads, verifies, locks, and caches Kubo archives.
mithril-client-cli/​src/​utils/​http_downloader/​reqwest.rs Adds shared HTTP client settings and response-status validation.
mithril-client-cli/​src/​utils/​http_client.rs Provides common Reqwest connect and read timeout configuration.
mithril-client-cli/​src/​utils/​github_release_retriever/​reqwest.rs Applies shared HTTP client settings to GitHub API calls.
mithril-client-cli/​src/​main.rs Adds runtime signal cancellation and asynchronous logger flushing.
mithril-client-cli/​src/​lib.rs Re-exports shutdown helpers.
mithril-client-cli/​src/​commands/​tools/​utxo_hd/​snapshot_converter.rs Makes converter execution cancellable and terminates interrupted processes.
mithril-client-cli/​src/​commands/​cardano_db/​download/​v2.rs Starts and stops managed Kubo nodes around database downloads.
mithril-client-cli/​src/​commands/​cardano_db/​download/​mod.rs Adds managed-node and GitHub-token CLI options and validation.
mithril-client-cli/​Cargo.toml Adds dependencies and features required for Kubo management and signal handling.
docs/​website/​root/​manual/​develop/​nodes/​mithril-client.md Documents managed and user-provided IPFS download modes and related parameters.
CHANGELOG.md Records the new managed-node download option.
Cargo.lock Locks newly introduced and updated dependency versions.
.github/​workflows/​test-client.yml Adds temporary repeated managed-node download checks across supported platforms.

Review effort: Lite

Kubo v0.43.1 release asset name and GitHub SHA-256 digest per supported
platform (linux amd64/arm64, macOS arm64, Windows amd64), with the
procedure to update the pinned version.
…PFS node

Retrieve the pinned Kubo release asset from GitHub, verify its SHA-256,
and install it in the user cache directory through a staging directory
so that a partial install is never used.
Installs are serialized between clients with a lock file, and leftovers
of an interrupted install are removed.

The Kubo distribution now also gives the path of the binary in the
release archive.
Previously the body of an error response (e.g. a 404 page) was written
to the target file as if the download had succeeded.
Loopback-only RPC API on a random port, no gateway, server profile,
TCP-only swarm, and no trace left on the network once removed (no DHT
provide, DHT client only, no bitswap server, no relay, no AutoTLS).
The datastore is soft bounded by a 15s automatic GC.
Initialize and configure a Kubo repository in a new directory, named
after the process id, then start its daemon and wait for its RPC API,
whose address is read from the repository `api` file. A failed start
reports the daemon output and removes the repository.

The daemon is stopped through the RPC shutdown command, and killed if it
does not exit in time. The repository is removed once the node is
stopped or dropped.

The Kubo commands do not inherit the `IPFS_*`, `GOLOG_*` and `LIBP2P_*`
environment variables of the client. The RPC client bypasses the system
proxy and has a timeout.
… download`

Unstable flag (env `IPFS_MANAGED_NODE`) conflicting with
`--ipfs-rpc-url`, and a `--github-token` option (env `GITHUB_TOKEN`)
used to download the Kubo binary.
The node is not started yet.
With `--ipfs-managed-node`, the Kubo binary is downloaded (or reused
from the user cache), a node is started with its repository in the
system temporary directory, and the download goes through its RPC API.

The node is stopped and its repository removed at the end of the
download, even when it fails.
…errupted

When the download goes through the managed IPFS node, `Ctrl-C` and
`SIGTERM` (on unix) interrupt it, then the node is stopped and its
repository removed before exiting, instead of leaving them behind.
Instead of only for the downloads through the managed IPFS node, all
commands are now interrupted by `Ctrl-C` and `SIGTERM` (on unix): the
command is dropped, releasing its resources (e.g. the managed IPFS node
is killed and its repository removed).

The tokio runtime is built manually, to shut it down with a timeout so a
blocking task still running when interrupted can not hang the client.
…errupted

Interrupting the client while the managed IPFS node was starting left
its repository directory, and the setup command or daemon then running.

The repository directory and the Kubo processes (setup commands and
daemon) are now guards, respectively removed and killed when dropped,
including when the start is cancelled.
- Stop listening to `SIGHUP`: it was also listened to when ignored at
  startup, so a `nohup` client was interrupted when its terminal closed.
- Poll the shutdown signal before running the command, so that its
  handlers are always installed before the command starts, instead of
  depending on the random polling order of `tokio::select!`.
- Run the snapshot converter asynchronously, killing it when the command
  is interrupted.
  It ran synchronously, so a shutdown signal was only handled once it
  exited, and a converter killed by `Ctrl-C` was taken as a failed
  conversion and retried with the next snapshot.
- Document that `SIGINT` and `SIGTERM` are listened to even if ignored
  at startup (e.g. `SIGINT` for the background jobs of a script).
… managed IPFS node

The Kubo release archive is now cached instead of its unpacked binary,
and its SHA-256 is verified each time it is provided: a cached archive
that is corrupted or replaced is downloaded again.

At each start, the managed node unpacks the binary in its repository
directory, so it is removed with it.
…between the HTTP downloader and GitHub API client
…ack address

With the `system-proxy` feature, reqwest also sends requests to loopback
addresses through the proxy set in `HTTP_PROXY`/`HTTPS_PROXY`: only an explicit
`NO_PROXY` entry excludes them.

So, behind an HTTP proxy, the IPFS file downloader sent its calls to a local
Kubo node (e.g. the CLI managed node on `127.0.0.1`) through the proxy. A proxy
that can't reach the client's loopback failed the Kubo RPC API check, and the
local node was never used.

The HTTP client of `IpfsFileDownloader` now skips the system proxy when the
host of the RPC URL is a loopback address (`127.0.0.0/8`, `::1`) or
`localhost`. A remote RPC URL still uses the system proxy.

Checked manually with a Squid proxy in Docker: without the fix, the calls to
`127.0.0.1:<port>/api/v0/version` went through the proxy and failed; with it,
none reach the proxy and the immutables are downloaded through IPFS.
- Document the IPFS download, with a node managed by the client or with
  your own node, and add the `--ipfs-rpc-url`, `--ipfs-managed-node` and
  `--github-token` parameters of `cardano-db download`.
- Add the missing `cardano-block` commands and parameters.
- Remove the references to the `v1` backend of the `cardano-db`
  commands, removed since a while.
- Create the Kubo repository directly in the system temporary directory,
  only accessible to the current user on Unix, instead of a
  `mithril-client` directory shared by all users, which let another
  local user tamper with the Kubo binary run from it, or prevent the
  other users from starting a managed node.
- Fail early if the parent directory of the Kubo repositories does not
  exist instead of creating it.
- Hide the value of the `GITHUB_TOKEN` environment variable in the help
  of `cardano-db download` and `tools utxo-hd snapshot-converter`.
- Do not listen for inbound swarm connections on the managed node, as it
  only downloads, which also avoids a firewall prompt on each run on
  Windows and macOS.
- Treat IPv4-mapped IPv6 loopback addresses and `localhost.` as loopback
  when bypassing the system proxy for the IPFS RPC URL.
- Match the Kubo environment variables not inherited by the managed node
  case-insensitively, as their names are on Windows.
…its config file

Apply the Kubo settings directly to the JSON config file of the
repository after its init, instead of running one `ipfs config` command
per setting, which spawned the Kubo binary 16 times and slowed down the
start of the node.
… secret

Require a basic authentication on the RPC API of the managed node with a
random secret generated at each start, so that another user of the host
can not control the node during a download.

The secret is written in the configuration file of the repository, never
in a process command line, is given to the downloader in the RPC URL
credentials, and is removed from the errors of the RPC calls.
…ic binaries check

They are imported by the `dirs` crate, used by the client CLI to locate
the user cache directory of the managed IPFS node.
…t out of the process list

- Remove the `--github-token` option of `cardano-db download`, new with
  the managed IPFS node, which reads the token from the `GITHUB_TOKEN`
  environment variable only, as documented in the help of
  `--ipfs-managed-node`.
- Deprecate the `--github-token` option of `tools utxo-hd
  snapshot-converter`: it no longer reads the `GITHUB_TOKEN` environment
  variable, which is read directly when the option is not set, and a
  deprecation warning is printed when it is used.
… cleaned up

When the managed node is dropped without being stopped (e.g. on an
interrupted download), print an error if its daemon has not exited in
time after being killed, or if its repository can not be removed,
instead of silently leaving them behind.
@Alenar
Alenar force-pushed the djo/3655/client-managed-kubo-node branch from 82c45c6 to a6486f6 Compare November 19, 2026 17:41
@Alenar
Alenar requested a review from jpraynaud November 19, 2026 17:41
@Alenar
Alenar deployed to testing-2-preview November 20, 2026 09:11 — with GitHub Actions Active
@Alenar
Alenar deployed to testing-preview November 20, 2026 09:11 — with GitHub Actions Active

@jpraynaud jpraynaud left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🔥

@Alenar
Alenar force-pushed the djo/3655/client-managed-kubo-node branch from a6486f6 to c3c09ff Compare November 20, 2026 10:44
@Alenar
Alenar deployed to testing-preview November 20, 2026 10:59 — with GitHub Actions Active
@Alenar
Alenar deployed to testing-2-preview November 20, 2026 10:59 — with GitHub Actions Active

@turmelclem turmelclem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 🐎

This branch was successfully deployed

2 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature 🚀 New implemented feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement a managed Kubo node download mode in the client

4 participants