Repository navigation
Conversation
Alenar
requested review from
damrobi,
jpraynaud and
turmelclem
as code owners
November 18, 2026 17:25
Test Results 5 files ± 0 211 suites ±0 59m 19s ⏱️ - 1h 3m 47s Results for commit c3c09ff. ± Comparison against base commit e50bd4e. This pull request removes 61 and adds 68 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
10 tasks done
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Two unresolved issues remain: inaccurate guidance for the managed IPFS mode and cleanup that can leave a Kubo process and temporary repository behind after interruption.
2 open issues
What changed in this PR
Adds an unstable managed Kubo-node mode for cardano-db download, including platform-pinned binary caching, node lifecycle management, signal-aware cleanup, and IPFS proxy handling.
Changes:
- Adds managed Kubo distribution, checksum verification, caching, configuration, startup, shutdown, and cleanup.
- Adds
--ipfs-managed-nodeand--github-token, with mutual exclusion against--ipfs-rpc-url. - Adds shutdown signal handling, bounded runtime cleanup, HTTP client timeouts, improved status handling, JSON warning escaping, documentation, and CI coverage.
| File | Description |
|---|---|
| mithril-client/src/utils/url.rs | Adds loopback URL detection for proxy bypassing. |
| mithril-client/src/file_downloader/ipfs.rs | Disables system proxies for loopback Kubo nodes. |
| mithril-client-cli/src/utils/shutdown_signal.rs | Adds Ctrl-C and SIGTERM cancellation helpers. |
| mithril-client-cli/src/utils/mod.rs | Exposes new HTTP, Kubo, shutdown, and warning utilities. |
| mithril-client-cli/src/utils/kubo_node/node_config.rs | Defines managed Kubo node configuration and runtime settings. |
| mithril-client-cli/src/utils/kubo_node/mod.rs | Registers the managed Kubo node modules. |
| mithril-client-cli/src/utils/kubo_node/managed_node.rs | Implements Kubo unpacking, startup, readiness checks, shutdown, and cleanup. |
| mithril-client-cli/src/utils/kubo_node/distribution.rs | Defines supported pinned Kubo distributions and checksums. |
| mithril-client-cli/src/utils/kubo_node/archive.rs | Unpacks and validates Kubo archives. |
| mithril-client-cli/src/utils/kubo_node/archive_provider.rs | Downloads, verifies, locks, and caches Kubo archives. |
| mithril-client-cli/src/utils/http_downloader/reqwest.rs | Adds shared HTTP client settings and response-status validation. |
| mithril-client-cli/src/utils/http_client.rs | Provides common Reqwest connect and read timeout configuration. |
| mithril-client-cli/src/utils/github_release_retriever/reqwest.rs | Applies shared HTTP client settings to GitHub API calls. |
| mithril-client-cli/src/main.rs | Adds runtime signal cancellation and asynchronous logger flushing. |
| mithril-client-cli/src/lib.rs | Re-exports shutdown helpers. |
| mithril-client-cli/src/commands/tools/utxo_hd/snapshot_converter.rs | Makes converter execution cancellable and terminates interrupted processes. |
| mithril-client-cli/src/commands/cardano_db/download/v2.rs | Starts and stops managed Kubo nodes around database downloads. |
| mithril-client-cli/src/commands/cardano_db/download/mod.rs | Adds managed-node and GitHub-token CLI options and validation. |
| mithril-client-cli/Cargo.toml | Adds dependencies and features required for Kubo management and signal handling. |
| docs/website/root/manual/develop/nodes/mithril-client.md | Documents managed and user-provided IPFS download modes and related parameters. |
| CHANGELOG.md | Records the new managed-node download option. |
| Cargo.lock | Locks newly introduced and updated dependency versions. |
| .github/workflows/test-client.yml | Adds temporary repeated managed-node download checks across supported platforms. |
Review effort: Lite
jpraynaud
reviewed
Nov 19, 2026
Kubo v0.43.1 release asset name and GitHub SHA-256 digest per supported platform (linux amd64/arm64, macOS arm64, Windows amd64), with the procedure to update the pinned version.
…PFS node Retrieve the pinned Kubo release asset from GitHub, verify its SHA-256, and install it in the user cache directory through a staging directory so that a partial install is never used. Installs are serialized between clients with a lock file, and leftovers of an interrupted install are removed. The Kubo distribution now also gives the path of the binary in the release archive.
Previously the body of an error response (e.g. a 404 page) was written to the target file as if the download had succeeded.
Loopback-only RPC API on a random port, no gateway, server profile, TCP-only swarm, and no trace left on the network once removed (no DHT provide, DHT client only, no bitswap server, no relay, no AutoTLS). The datastore is soft bounded by a 15s automatic GC.
Initialize and configure a Kubo repository in a new directory, named after the process id, then start its daemon and wait for its RPC API, whose address is read from the repository `api` file. A failed start reports the daemon output and removes the repository. The daemon is stopped through the RPC shutdown command, and killed if it does not exit in time. The repository is removed once the node is stopped or dropped. The Kubo commands do not inherit the `IPFS_*`, `GOLOG_*` and `LIBP2P_*` environment variables of the client. The RPC client bypasses the system proxy and has a timeout.
… download` Unstable flag (env `IPFS_MANAGED_NODE`) conflicting with `--ipfs-rpc-url`, and a `--github-token` option (env `GITHUB_TOKEN`) used to download the Kubo binary. The node is not started yet.
With `--ipfs-managed-node`, the Kubo binary is downloaded (or reused from the user cache), a node is started with its repository in the system temporary directory, and the download goes through its RPC API. The node is stopped and its repository removed at the end of the download, even when it fails.
…errupted When the download goes through the managed IPFS node, `Ctrl-C` and `SIGTERM` (on unix) interrupt it, then the node is stopped and its repository removed before exiting, instead of leaving them behind.
Instead of only for the downloads through the managed IPFS node, all commands are now interrupted by `Ctrl-C` and `SIGTERM` (on unix): the command is dropped, releasing its resources (e.g. the managed IPFS node is killed and its repository removed). The tokio runtime is built manually, to shut it down with a timeout so a blocking task still running when interrupted can not hang the client.
…errupted Interrupting the client while the managed IPFS node was starting left its repository directory, and the setup command or daemon then running. The repository directory and the Kubo processes (setup commands and daemon) are now guards, respectively removed and killed when dropped, including when the start is cancelled.
- Stop listening to `SIGHUP`: it was also listened to when ignored at startup, so a `nohup` client was interrupted when its terminal closed. - Poll the shutdown signal before running the command, so that its handlers are always installed before the command starts, instead of depending on the random polling order of `tokio::select!`. - Run the snapshot converter asynchronously, killing it when the command is interrupted. It ran synchronously, so a shutdown signal was only handled once it exited, and a converter killed by `Ctrl-C` was taken as a failed conversion and retried with the next snapshot. - Document that `SIGINT` and `SIGTERM` are listened to even if ignored at startup (e.g. `SIGINT` for the background jobs of a script).
… managed IPFS node The Kubo release archive is now cached instead of its unpacked binary, and its SHA-256 is verified each time it is provided: a cached archive that is corrupted or replaced is downloaded again. At each start, the managed node unpacks the binary in its repository directory, so it is removed with it.
… and GitHub API client
…between the HTTP downloader and GitHub API client
…ack address With the `system-proxy` feature, reqwest also sends requests to loopback addresses through the proxy set in `HTTP_PROXY`/`HTTPS_PROXY`: only an explicit `NO_PROXY` entry excludes them. So, behind an HTTP proxy, the IPFS file downloader sent its calls to a local Kubo node (e.g. the CLI managed node on `127.0.0.1`) through the proxy. A proxy that can't reach the client's loopback failed the Kubo RPC API check, and the local node was never used. The HTTP client of `IpfsFileDownloader` now skips the system proxy when the host of the RPC URL is a loopback address (`127.0.0.0/8`, `::1`) or `localhost`. A remote RPC URL still uses the system proxy. Checked manually with a Squid proxy in Docker: without the fix, the calls to `127.0.0.1:<port>/api/v0/version` went through the proxy and failed; with it, none reach the proxy and the immutables are downloaded through IPFS.
- Document the IPFS download, with a node managed by the client or with your own node, and add the `--ipfs-rpc-url`, `--ipfs-managed-node` and `--github-token` parameters of `cardano-db download`. - Add the missing `cardano-block` commands and parameters. - Remove the references to the `v1` backend of the `cardano-db` commands, removed since a while.
- Create the Kubo repository directly in the system temporary directory, only accessible to the current user on Unix, instead of a `mithril-client` directory shared by all users, which let another local user tamper with the Kubo binary run from it, or prevent the other users from starting a managed node. - Fail early if the parent directory of the Kubo repositories does not exist instead of creating it. - Hide the value of the `GITHUB_TOKEN` environment variable in the help of `cardano-db download` and `tools utxo-hd snapshot-converter`. - Do not listen for inbound swarm connections on the managed node, as it only downloads, which also avoids a firewall prompt on each run on Windows and macOS. - Treat IPv4-mapped IPv6 loopback addresses and `localhost.` as loopback when bypassing the system proxy for the IPFS RPC URL. - Match the Kubo environment variables not inherited by the managed node case-insensitively, as their names are on Windows.
…its config file Apply the Kubo settings directly to the JSON config file of the repository after its init, instead of running one `ipfs config` command per setting, which spawned the Kubo binary 16 times and slowed down the start of the node.
… secret Require a basic authentication on the RPC API of the managed node with a random secret generated at each start, so that another user of the host can not control the node during a download. The secret is written in the configuration file of the repository, never in a process command line, is given to the downloader in the RPC URL credentials, and is removed from the errors of the RPC calls.
…ic binaries check They are imported by the `dirs` crate, used by the client CLI to locate the user cache directory of the managed IPFS node.
…t out of the process list - Remove the `--github-token` option of `cardano-db download`, new with the managed IPFS node, which reads the token from the `GITHUB_TOKEN` environment variable only, as documented in the help of `--ipfs-managed-node`. - Deprecate the `--github-token` option of `tools utxo-hd snapshot-converter`: it no longer reads the `GITHUB_TOKEN` environment variable, which is read directly when the option is not set, and a deprecation warning is printed when it is used.
… cleaned up When the managed node is dropped without being stopped (e.g. on an interrupted download), print an error if its daemon has not exited in time after being killed, or if its repository can not be removed, instead of silently leaving them behind.
Alenar
force-pushed
the
djo/3655/client-managed-kubo-node
branch
from
November 19, 2026 17:41
82c45c6 to
a6486f6
Compare
Alenar
force-pushed
the
djo/3655/client-managed-kubo-node
branch
from
November 20, 2026 10:44
a6486f6 to
c3c09ff
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Content
This PR adds a managed IPFS node mode to
cardano-db download: with the new--ipfs-managed-nodeoption, the client downloads, runs and cleans up its own Kubo node, so users don't need to install and configure one to download a Cardano database through IPFS. Bringing your own node with--ipfs-rpc-urlis still supported, both options are mutually exclusive.Managed Kubo node (
mithril-client-cli/src/utils/kubo_node)v0.43.1, with the SHA-256 of the release archive of each supported platform: Linux x86_64 and aarch64, macOS aarch64, Windows x86_64.GITHUB_TOKENenvironment variable to avoid the API rate limit, read from the environment only so that the token doesn't appear in the process list), and cached in the user cache directory (~/.cache/mithril/kubo/v0.43.1/).mithril-client-kubo-<pid>-<timestamp>), created in the system temporary directory and only accessible to the current user on Unix (0700), so another user of a shared/tmpcan't tamper with the binary run from it. The node is initialized, configured with a single edit of itsconfigfile, then its daemon is started and the client waits for its RPC API to answer.127.0.0.1with a random port, no gateway.configfile (never on a command line), given to the downloader in the RPC URL credentials, and removed from the errors of the RPC calls.autoclientrouting, no content providing, Bitswap server disabled, telemetry off.IPFS_*,GOLOG_*,LIBP2P_*, matched case-insensitively as on Windows) are not inherited by the node.Shutdown signals
Ctrl-C/SIGINTorSIGTERM, so the managed node is stopped and its repository removed when a download is interrupted.Fixes found along the way
mithril-clientlibrary: the IPFS downloader now bypasses the system proxy when the RPC URL is a loopback address (localhost, IPv4, IPv6 or IPv4-mapped IPv6 loopback). WithHTTP(S)_PROXYset, the calls to a local Kubo node went through the proxy and failed.--github-tokenoption is deprecated, as the token appears in the process list: it prints a deprecation warning when used, and no longer readsGITHUB_TOKENitself (the environment variable is read directly when the option is not set, and its value no longer appears in--help).CI
shell32.dllandcombase.dllsystem DLLs, imported by thedirscrate used to locate the user cache directory.Documentation
ipfs_rpc_urlandipfs_managed_nodeparameters, and theGITHUB_TOKENenvironment variable.cardano-blocksections, and the mentions of the removedv1backend are removed.--github-tokenoption of the snapshot converter.Manual checks
On
testing-preview, full snapshot (≈15.5 GB), release build:--ipfs-managed-node --ipfs-share 100SIGINTafter 2 minThe loopback proxy bypass was checked with a Squid proxy in Docker.
Pre-submit checklist
Comments
#[ignore]d. Run them withcargo test -p mithril-client-cli -- --include-ignored.cargo test, the tests running a fake Kubo binary can rarely fail withText file busy: another test thread spawning a process can briefly hold the just unpacked binary open for writing.cargo nextest, used by the CI, runs each test in its own process and is not affected.TEMP(DO NOT MERGE)commit adds a step totest-client.ymlthat runs 10 managed node downloads on each OS, to check whether an antivirus lock on the Kubo files makes the node fail on Windows. It will be dropped before merge.dev-mainnet, IPFS downloads currently fall back to cloud storage: the aggregator's Kubo node doesn't keep its root CIDs announced (its provider queue is saturated). This is an infra issue, followed separately.Issue(s)
Closes #3655