Repository navigation
Conversation
Preparation for moving to hermod-tracing, valid on contra-tracer as it is: - FilePermissions test: build the capturing tracer with mkTracer instead of the Tracer constructor and Control.Tracer.Arrow's emit. - Tracing/API.hs: the disabled forwarding traces are mempty instead of `Trace nullTracer`, which relied on trace-dispatcher's Trace constructor wrapping a contra-tracer Tracer. - Import Tracer abstractly where its constructor is not used (Cardano.Node.Tracing, tx-generator's NodeToNode). - Drop the "contra-tracer" and "trace-dispatcher" package qualifiers, which no longer disambiguate anything, and the PackageImports pragmas that only served them (16 files). No behaviour change. stylish-haskell 0.14.4.0 on the touched files without CPP; in cardano-tracer's Notifications/Timer.hs that also sorts the imports.
NodeInfo and NodeStartupInfo are the DataPoints cardano-node forwards at
start-up and cardano-tracer decodes. They were defined in trace-dispatcher,
but they are cardano-node's own payload types, and hermod-tracing (which
replaces trace-dispatcher) does not have them.
trace-forward is the one package both cardano-node and cardano-tracer depend
on, so the two types move there verbatim, as Trace.Forward.Types.NodeInfo and
Trace.Forward.Types.NodeStartupInfo; trace-forward gains a `time` dependency.
The JSON instances are derived as before, so the DataPoints are encoded
exactly as they were, and old and new nodes and tracers keep understanding
each other. cardano-node's MetaTrace instances for them stay in
Cardano.Node.Tracing.{NodeInfo,NodeStartupInfo}; Cardano.Node.Startup
re-exports the new modules as it did the old ones.
Valid with trace-dispatcher as it is, whose copies are simply no longer used.
hermod-tracing, which replaces trace-dispatcher, reads the host name it puts into every log line from HERMOD_TRACING_LOGGING_HOSTNAME; TRACE_DISPATCHER_LOGGING_HOSTNAME is only a deprecated fallback there. The workbench relies on it to give each node of a local cluster its own "host", which locli uses to tell the forger from the observers. Exporting both names keeps that working before and after the switch, and lets the old name go later.
The trace-dispatcher types trace-forward uses (TraceObject, DataPoint, the
forwarding options) come from hermod-tracing-core, which is trace-dispatcher
renamed and split.
Its protocol tracers stay on contra-tracer: acceptorTracer and forwarderTracer
are tracers over Ouroboros.Network.Driver's TraceSendRecv and are handed
straight to runPeer, and Forwarding.hs needs contra-tracer's nullTracer for
ouroboros-network's HandshakeArguments. hermod-tracing-api is a dependency of
neither stanza: only hermod-tracing-core is used, and -Werror=unused-packages
rejects the rest.
This commit starts the switch of the repository; together with the following
ones up to "submit-api, chairman, testnet, ...", it moves every package. Each
of them builds its own packages against the hermod-based ouroboros-network,
ouroboros-consensus, ekg-forward, cardano-api and cardano-cli, but the whole
tree builds again only after that last one, so they are best landed together.
The mechanical part of this and the following commits is
hermod-tracing's scripts/contra-tracer-to-hermod.sh (Control.Tracer ->
Hermod.Tracing.API.Tracer, contra-tracer -> hermod-tracing-api:public) plus
these substitutions:
Cardano.Logging.Resources[.Types] -> Hermod.Tracing.Resources[.Types]
Cardano.Logging.Prometheus.* -> Hermod.Tracing.Prometheus.*
Cardano.Logging.{DocuGenerator,Utils,Formatter,Configuration,
ConfigurationParser,Consistency,FrequencyLimiter,Tracer.*}
-> Hermod.Tracing.<same>
Cardano.Logging.TraceDispatcherMessage -> Hermod.Tracing.HermodTracingMessage
Cardano.Logging.Types, Cardano.Logging.Trace, Cardano.Logging
-> Hermod.Tracing (the umbrella; the new
home of Types/Trace is not for user code)
mkCardanoTracer(') -> mkHermodTracer(')
TraceDispatcherMessage -> HermodTracingMessage
PrometheusM -> LabelSetM
tcNodeName -> tcApplicationName
CounterM n Nothing -> CounterM n CounterIncrement
By hand here: Forwarding.hs imports the five types it needs from the
Hermod.Tracing umbrella explicitly (the umbrella is much larger than
Cardano.Logging.Types was, and an open import would shadow its local
mkTracer); the cabal entries; the README sentence about who feeds the library.
The tracers in the forwarder/acceptor configurations keep contra-tracer's
carrier, so nothing breaks for their users.
The same move as the rest of the repository (see the trace-forward commit): Cardano.Logging -> Hermod.Tracing (hermod-tracing-core) and Cardano.Logging.Resources -> Hermod.Tracing.Resources (hermod-trace-resources), whose API is identical. cardano-tracer depends on this package for its timeseries traces, so it moves first. Breaking for users of the library: TimeseriesTrace's LogFormatting / MetaTrace instances are hermod's now. The traces emitted and the resource statistics read are unchanged.
cardano-tracer receives what trace-forward and ekg-forward hand it, and both now speak hermod: TraceObject, DataPoint and the forwarding options come from hermod-tracing-core, the Prometheus exposition lives in hermod-tracing-prometheus and the resource statistics in hermod-trace-resources. The forwarding protocol tracers stay on contra-tracer, because trace-forward hands them straight to ouroboros-network's runPeer. Mechanical as described in the trace-forward commit, plus by hand: - MetaTrace: machineFormatter is pure in hermod, so it is applied before filterSeverityFromConfig instead of being bound; the removed contramapM' is replaced by mkTrace, which, like it, prints messages and ignores control messages; withNames/withSeverity are imported from Hermod.Tracing.Trace.Annotations, which the umbrella does not re-export. - ReForwarder: the separate DataPoint import is redundant behind the umbrella. - bench: drop `hiding (LocalSocket)`, a name hermod no longer exports. - cabal: hermod-tracing-core ^>= 1.0.1, the release that reads the forwarder options (also in cardano-tracer's own hasForwarding) and the host-name variable as trace-dispatcher did. - docs: name ApplicationName / MetricsPrefix, and say that the TraceOption* spellings are still read in the deprecated TraceOptions layout. The configuration format, the logs written and the metrics served are unchanged; old and new nodes and tracers interoperate (the TraceObject and DataPoint encodings are the same).
The node's tracing is built on hermod-tracing-core (trace-dispatcher renamed
and split), hermod-tracing-prometheus and hermod-trace-resources.
ouroboros-network's and ouroboros-consensus' public libraries hold
contra-tracer Tracers in their tracer records, so Cardano.Node.Tracing's
Tracers record is mixed rather than uniformly hermod: a field takes
contra-tracer's carrier exactly when its message type or its consumer belongs
to one of those libraries -- chainDBTracer, churnModeTracer, rpcTracer, the KES
agent signatures in Run.hs and the four library-owned tracer records -- and the
node's own tracers are hermod traces.
Mechanical as described in the trace-forward commit. By hand:
- Tracers.hs keeps its mkTracer (traceWith tr) wrappers. The plain mkTracer
comes from the Hermod.Tracing umbrella, which re-exports contra-tracer's, and
builds the library-facing fields; the node's own fields take the facade's,
qualified as HT. Behaviour is unchanged: nothing sends control messages
through the tracers the libraries hold. (The wrappers go when tracer
construction is reworked.)
- CounterM n (Just i) -> CounterM n (CounterAdd (fromIntegral i)).
- The resource and ledger-metrics periods are read from PeriodicTracers
("resources", "ledgerMetrics") with the same fallbacks as before, clamped to
Int; DefaultTraceConfig no longer needs to discard a library default.
- hermod-tracing-prometheus and cardano-rpc still take contra-tracer tracers:
bridged with toContraTracer (Tracing/API.hs, Run.hs).
- contramapMCond is pure in hermod (BlockReplayProgress).
- Documentation imports HermodTracingMessage, which the umbrella does not
re-export.
- README: the dependency diagram names hermod-tracing.
Existing configurations keep working unchanged: hermod-tracing-core 1.0.1
still reads the TraceOptions* layout (deprecated), with the same meaning and
precedence. Log output, namespaces, metrics and forwarding are the same as
before, with two visible exceptions in content: the generated trace
documentation's title ("Tracing Documentation"), and the effective
configuration logged at start-up (Reflection.TracerInfoConfig), which now
includes the PeriodicTracers settings that trace-dispatcher left out.
…ermod-tracing The remaining packages follow cardano-node onto hermod-tracing and the hermod tracer facade (mechanical part as described in the trace-forward commit): - cardano-submit-api: Cardano.Logging -> Hermod.Tracing; CounterM takes a CounterAction. Its configuration keeps working (deprecated TraceOptions). - cardano-node-chairman: Control.Tracer -> facade. - cardano-testnet: Control.Tracer -> facade, except at the ouroboros-network boundary, where the ping client of cardano-diffusion keeps contra-tracer. Breaking for library users: Testnet.ChainWatchdog's chainStallWatchdog and stderrTracer use hermod traces. Generated node configurations are unchanged (they use the deprecated layout, as the comment there now says). - tx-generator: forwardFormatter/machineFormatter are pure in hermod; the annotation combinators come from Hermod.Tracing.Trace.Annotations; the remaining mkTracer (traceWith ...) wrapper takes the umbrella's mkTracer, and NodeToNode.hs keeps contra-tracer's Tracer for the ouroboros-network boundary. - locli: Cardano.Logging.Resources.Types -> Hermod.Tracing.Resources.Types. - trace-schema-gen: scan hermod's sources (api, core, prometheus, trace-resources) instead of trace-dispatcher's. With this commit the whole repository builds again.
The profiled build variant lists the tracing packages to build with -fprof-auto; trace-resources is now hermod-trace-resources.
hermod-tracing-prometheus 1.1 and cardano-rpc now take hermod traces, so the node passes its tracers to runPrometheusSimple(With), mkNodeKernelAccess and runRpcServer directly. The three toContraTracer bridges and both Hermod.Tracing.API.ContraTracer imports go. This removes the bridges, not the dependency. contra-tracer is still named directly wherever a type crosses the boundary with a library that kept it in its public API -- ouroboros-network, ouroboros-consensus and cardano-rpc -- which is why several packages here still depend on it. cardano-node requires hermod-tracing-prometheus ^>= 1.1. cardano-tracer uses only the Exposition module, which 1.1 leaves unchanged, so it accepts 1.0 and 1.1.
The checked-in list had drifted from what `cardano-node trace-documentation` produces, for reasons that have nothing to do with the hermod migration. The Startup namespaces it gains (BlockForgingUpdateError, the DiffusionInit ones, NetworkConfigUpdateInfo/Warning, RpcUnsupportedBlockType) were already in the node's sources at the branch point and merely missing from the file. The ChainDB rename IgnoreBlockOlderThanK -> IgnoreBlockOlderThanImmTip, the removal of the LMDB backing-store namespaces and the Peras, PerasVoteDb and KESAgent additions follow the ouroboros-consensus version this branch builds against. Regenerated with bench/trace-schemas/scripts/schema-gen/RegenerateTraceSchemas.sh. None of this comes from the migration commits that precede it, which is why it stands on its own rather than being folded into one of them.
The node declares its tracers three times by hand: the runtime construction in Tracers.hs, the documentation in Documentation.hs and the consistency check in Consistency.hs. Only the first is compiler-checked, and the other two have drifted from it (Forge.Stats is documented as Forge.ThreadStats and not checked; Net.DNS is checked as Net.DNSResolver; the KES-info and peer-selection-counter prefixes differ; several tracers are in one list and not the others). This commit makes the runtime construction the single declaration. The new module Cardano.Node.Tracing.Registry provides the constructors: newTrace builds a hermod trace and records it (prefix, trace, how it is listed), so the same declaration can later drive the documentation and the consistency check. buildNodeTracers in Tracers.hs is that declaration: one `newTrace registry ["A", "B"]` per tracer, with the element types inferred from the library records as before, so no type annotations are needed. mkDispatchTracers builds, then configures every recorded entry, then performs the start-up messages as before. Its unused NodeKernelData parameter goes. Entries that only exist for the documentation (Reflection, the BlockFetch client metrics) are declared with docOnly and are not built at runtime. Entries whose MetaTrace instances in ouroboros-network would make the documentation or the check warn (ChurnMode, DNS, the Mux bearer traces) are declared as undocumented or runtime-only; the comments name the deficiency. The two hooks (ChainDB's AddedToCurrentChain limiter, the forging statistics) are applied to the constructed trace instead of inside mkHermodTracer', which had applied them separately to the message and the metrics path. Each now has one state instead of two identical ones; the output is the same. The runtime consistency check now reads the recorded entries, so Tracers.hs no longer imports Consistency. Documentation.hs and Consistency.hs are unchanged in this commit; they follow in the next one.
…k from it The documentation and the configuration consistency check no longer keep their own lists of the node's tracers. Both now run buildNodeTracers, with backends that only answer documentation requests, and read the registry it fills: the documentation documents every recorded trace, the check takes the recorded prefixes and the MetaTrace instances' namespaces. The 74 type-annotated reconstructions in Documentation.hs and the 77 annotated namespace blocks in Consistency.hs go, as does the Proxy trick that fixed the block type; one type application does that now. Both therefore follow the runtime names, which changes the documented and checked namespaces where the lists had drifted: - Forge.StateInfo and Forge.ThreadStats.ForgingStats become Forge.StateInfo.StateInfo and Forge.Stats.ForgingStats, the names the node emits (the schema override moves with the first); - Net.PeerSelection.Counters.Counters becomes Net.PeerSelection.Counters; - the check knows Net.DNS instead of Net.DNSResolver, and NodeInfo, NodeStartupInfo and Reflection, which it rejected; - the Mux channel and the handshake tracers are documented; - Net.AcceptPolicy and Net.PeerSelection.Responder, which no tracer emits, are dropped from both. The metrics help text is unchanged. The consistency test's allowlists shrink to the three tracers declared undocumented, and a new golden test keeps bench/trace-schemas/newNamespaces.txt equal to what trace-documentation generates. The test fixtures use the current names.
Net.ErrorPolicy, Net.AcceptPolicy and Net.PeerSelection.Responder name no tracer the node has; with the consistency check now derived from the tracers the node declares, they would be reported as illegal namespaces at start-up. Net.DNSResolver is the check's old spelling of Net.DNS; in the workbench it had the severity of its parent key, so the rename changes nothing. The changelog entry covers the previous two commits as well.
Hermod has always advertised hot reconfiguration -- configureTracers' haddock says "Call this function at initialisation, and later for reconfiguration" -- but the node never used it: changing a severity meant a restart. Everything needed was already in place after the registry work; the only thing missing was that the registry, the entries and the TraceConfig were all local to mkDispatchTracers and discarded on return, so nothing survived for a later SIGHUP to use. Registry: the apply-and-report step that mkDispatchTracers had inlined becomes ApplyTraceConfig, built by `applier` next to configureAll, so the start-up pass and the reload pass are the same code and cannot drift apart. It has two entry points: applyTraceConfig reports the reflection, the consistency warnings and the effective configuration; restoreTraceConfig reports only the effective configuration, for putting the previous one back after a failure, where the rest is noise. Each pass builds a fresh ConfigReflection, because traceTracerInfo empties the one it reports. New Cardano.Node.Tracing.Reconfigure holds the reload itself. An MVar TraceConfig is both the mutex and the last configuration known to apply cleanly, which makes "store it only on success" the same step as releasing the lock. The order matters: trace the start marker under the configuration still in force, then read -- so a malformed or missing file never perturbs a running trace -- then apply, and on failure restore before reporting, because a half-applied pass can leave the startup tracer itself discarding everything. try @someexception and not a narrower type: this path throws ParseException (malformed YAML and a missing file alike), AesonException, and a bare error for more than four "HermodTracing" redirects. readConfigurationWithDefault with the running configuration as the default is what makes this the trace-options part and not the whole configuration: mergeWithDefault replaces tcOptions wholesale and takes the forwarder, application name, metrics prefix, periodic tracers and PrometheusSimple from the default. An operator who edits only TraceOptions changes only the options. Run.hs: initTraceDispatcher returns the handle alongside the tracers, and it is threaded to both SIGHUP handlers -- the early one and the one installSigHUPHandler reinstalls once the node kernel exists (the non-UNIX stub grows from 14 to 15 wildcards). Reconfiguring runs first in both, so the handler's own later messages are emitted under the new settings; unlike block forging there is no "not effective until consensus is up" caveat, and DB replay is exactly when an operator wants to change ChainDB severity. The handler's tracing job is that single call with no logic in it, so what cannot be tested through a real signal is also trivial. Reporting is three Startup constructors -- TraceConfigUpdate and TraceConfigUpdated at Notice, TraceConfigUpdateError at Error -- modelled on NetworkConfigUpdate/RpcConfigUpdateError. No per-field "needs a restart" detection; the fields that cannot be reloaded are listed in the ChangeLog instead, along with the non-atomic drop window and the pointer to hermod's Reflection.TracerInfoConfig, which traceEffectiveConfiguration emits unconditionally and so is the authoritative record of what is in force. bench/trace-schemas/newNamespaces.txt is regenerated, not hand-edited. Test/Cardano/Tracing/NewTracing/Reconfigure.hs drives the node's own registry over a capturing stdout backend, so it covers every declared trace: that A -> B -> A changes the output and is reversible rather than latching (the property hermod's own runScriptMultithreadedWithReconfig cannot assert, its oracle being pinned to the pre-reconfiguration config); that invalid YAML, a bad enum and a deleted file are each a complete no-op and each reported (the deleted-file case is why readConfiguration' must not be used here -- it silently substitutes a minimal config); and that two concurrent reloads serialise. Nothing anywhere covered two concurrent reconfigurations, which is exactly the SIGHUP hazard: with the mutex removed the second pass throws HermodConfigException "Trace not reset before reconfiguration" and the node goes deaf, and the property catches it. Verified: `cabal build all -j` green; `cardano-node-test` 30/30 (from cardano-node/, including the namespace-list golden, which the regenerated newNamespaces.txt keeps passing); stylish-haskell 0.14.4.0 clean on every touched non-CPP module. Verified on a real node (mainnet config, empty db, three real SIGHUPs; wt/logs/smoke/n19-sighup): root Notice -> Debug Startup.TraceConfigUpdate, Startup.TraceConfigUpdated and Reflection.TracerInfoConfig once each, then 131210 new Debug-severity lines where there had been none. malformed YAML Startup.TraceConfigUpdateError once ("YAML parse exception at line 1, column 2"), no Updated, Debug still flowing, node alive: a complete no-op. root Debug -> Notice Updated once, then no Debug line at all; the four remaining Info lines are ChainDB, which the mainnet config pins to Info. The workbench cluster could not be used: `make shell` in this worktree fails in haskell.nix's plan-to-nix with "unknown package: hermod-tracing-prometheus", because nothing is published to CHaP yet. The single-node harness above exercises the same code -- a real signal, the real handler, the real file.
…builds Lets CI build this branch before CHaP has hermod-tracing-api 1.1.0, hermod-tracing-core 1.0.1 and hermod-tracing-prometheus 1.1.0, which are IntersectMBO/hermod-tracing#21. ouroboros-network is pinned for a different reason: its tracing sublibraries provide their LogFormatting instances against hermod-tracing-api rather than trace-dispatcher. The flipped packages keep their released version numbers, so no version bound can ask for that build. Without the pin the node library fails to compile with "Could not deduce LogFormatting (Mux.WithBearer (ConnectionId RemoteAddress) Mux.Trace)" and seven more of the same shape. Nothing else needs pinning. ouroboros-consensus, cardano-api, cardano-rpc, cardano-cli, ekg-forward and dmq-node all resolve from CHaP: cardano-node builds and its test suite passes against exactly this set. Both stanzas must be removed before this merges.
c9c0cf1 to
52cd97e
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Runtime reload defaults are incorrectly latched, asynchronous cancellation is swallowed, and the explicitly temporary dependency pins remain present.
5 open findings
What changed in this PR
Migrates repository tracing from trace-dispatcher to hermod-tracing, centralizes node tracer registration, and adds SIGHUP tracing reconfiguration.
Changes:
- Replaces tracing/resource dependencies and imports across packages.
- Introduces a unified node trace registry for runtime configuration, documentation, and namespace checks.
- Adds serialized runtime trace-option reloads and related tests.
| File | Description |
|---|---|
trace-forward/trace-forward.cabal |
Updates tracing dependencies and exports node data types. |
trace-forward/src/Trace/Forward/Utils/TraceObject.hs |
Uses hermod trace-object utilities. |
trace-forward/src/Trace/Forward/Utils/DataPoint.hs |
Uses hermod data-point APIs. |
trace-forward/src/Trace/Forward/Types/NodeStartupInfo.hs |
Adds shared startup information type. |
trace-forward/src/Trace/Forward/Types/NodeInfo.hs |
Adds shared node information type. |
trace-forward/src/Trace/Forward/Protocol/TraceObject/Type.hs |
Migrates TraceObject import. |
trace-forward/src/Trace/Forward/Protocol/DataPoint/Type.hs |
Migrates data-point name import. |
trace-forward/src/Trace/Forward/Forwarding.hs |
Migrates forwarding options and utilities. |
trace-forward/src/Trace/Forward/Configuration/TraceObject.hs |
Reorders trace-object configuration imports. |
trace-forward/src/Trace/Forward/Configuration/DataPoint.hs |
Reorders data-point configuration imports. |
trace-forward/README.md |
Updates tracing architecture description. |
trace-forward/CHANGELOG.md |
Documents migration and shared types. |
README.md |
Updates the dependency diagram. |
nix/workbench/service/tracing.nix |
Updates network trace namespaces. |
nix/workbench/service/nodes.nix |
Adds hermod logging hostname. |
nix/nixos/cardano-tracer-service.nix |
Updates metrics-prefix documentation. |
nix/haskell.nix |
Profiles the renamed resource package. |
cardano-tracer/test/Cardano/Tracer/Test/Utils.hs |
Migrates test network types. |
cardano-tracer/test/Cardano/Tracer/Test/Restart/Tests.hs |
Migrates restart tests. |
cardano-tracer/test/Cardano/Tracer/Test/Queue/Tests.hs |
Migrates queue tests. |
cardano-tracer/test/Cardano/Tracer/Test/Logs/Tests.hs |
Migrates log tests. |
cardano-tracer/test/Cardano/Tracer/Test/ForwardingStressTest/Types.hs |
Migrates stress-test tracing types. |
cardano-tracer/test/Cardano/Tracer/Test/ForwardingStressTest/Script.hs |
Migrates stress-test tracer setup. |
cardano-tracer/test/Cardano/Tracer/Test/ForwardingStressTest/Messages.hs |
Migrates stress-test formatting. |
cardano-tracer/test/Cardano/Tracer/Test/ForwardingStressTest/Config.hs |
Migrates stress-test configuration. |
cardano-tracer/test/Cardano/Tracer/Test/Forwarder.hs |
Migrates forwarder test helpers. |
cardano-tracer/test/Cardano/Tracer/Test/DataPoint/Tests.hs |
Migrates data-point tests. |
cardano-tracer/test/Cardano/Tracer/Test/Acceptor.hs |
Migrates acceptor tests. |
cardano-tracer/test/cardano-tracer-test-ext.hs |
Uses mkHermodTracer. |
cardano-tracer/src/Cardano/Tracer/Utils.hs |
Uses shared NodeInfo. |
cardano-tracer/src/Cardano/Tracer/Run.hs |
Migrates resource tracing. |
cardano-tracer/src/Cardano/Tracer/MetaTrace.hs |
Reworks hermod meta tracing. |
cardano-tracer/src/Cardano/Tracer/Handlers/State/TraceObjects.hs |
Migrates trace-object state handling. |
cardano-tracer/src/Cardano/Tracer/Handlers/ReForwarder.hs |
Migrates re-forwarding. |
cardano-tracer/src/Cardano/Tracer/Handlers/Notifications/Utils.hs |
Migrates notification traces. |
cardano-tracer/src/Cardano/Tracer/Handlers/Notifications/Types.hs |
Migrates notification severities. |
cardano-tracer/src/Cardano/Tracer/Handlers/Notifications/Timer.hs |
Migrates timer traces. |
cardano-tracer/src/Cardano/Tracer/Handlers/Notifications/Send.hs |
Migrates notification logging. |
cardano-tracer/src/Cardano/Tracer/Handlers/Notifications/Check.hs |
Migrates severity checks. |
cardano-tracer/src/Cardano/Tracer/Handlers/Metrics/TimeseriesServer.hs |
Uses shared node information types. |
cardano-tracer/src/Cardano/Tracer/Handlers/Metrics/Prometheus.hs |
Migrates Prometheus exposition. |
cardano-tracer/src/Cardano/Tracer/Handlers/Logs/TraceObjects.hs |
Migrates trace-object logging. |
cardano-tracer/src/Cardano/Tracer/Handlers/Logs/Journal/Systemd.hs |
Migrates systemd journal tracing. |
cardano-tracer/src/Cardano/Tracer/Handlers/Logs/Journal/NoSystemd.hs |
Migrates no-systemd stub types. |
cardano-tracer/src/Cardano/Tracer/Handlers/Logs/File.hs |
Migrates file trace objects. |
cardano-tracer/src/Cardano/Tracer/Environment.hs |
Migrates tracer environment types. |
cardano-tracer/src/Cardano/Tracer/Configuration.hs |
Migrates tracing configuration types. |
cardano-tracer/src/Cardano/Tracer/CLI.hs |
Migrates CLI tracing imports. |
cardano-tracer/src/Cardano/Tracer/Acceptors/Server.hs |
Migrates server acceptor tracing. |
cardano-tracer/src/Cardano/Tracer/Acceptors/Run.hs |
Migrates acceptor runtime utilities. |
cardano-tracer/src/Cardano/Tracer/Acceptors/Client.hs |
Migrates client acceptor tracing. |
cardano-tracer/docs/cardano-tracer.md |
Updates application-name documentation. |
cardano-tracer/demo/ssh/forwarder.hs |
Migrates demo network types. |
cardano-tracer/CHANGELOG.md |
Documents tracer migration. |
cardano-tracer/cardano-tracer.cabal |
Replaces tracing dependencies. |
cardano-tracer/bench/cardano-tracer-bench.hs |
Migrates benchmark tracing. |
cardano-testnet/src/Testnet/Ping.hs |
Reorders tracer import. |
cardano-testnet/src/Testnet/Defaults.hs |
Documents deprecated configuration layout. |
cardano-testnet/src/Testnet/ChainWatchdog.hs |
Migrates watchdog tracers. |
cardano-testnet/cardano-testnet.cabal |
Adds hermod API dependency. |
cardano-testnet/.changes/20261001_120000_juergen.nicklisch_hermod_tracing.yml |
Records testnet API migration. |
cardano-submit-api/src/Cardano/TxSubmit/Web.hs |
Migrates web tracing. |
cardano-submit-api/src/Cardano/TxSubmit/Util.hs |
Migrates exception logging. |
cardano-submit-api/src/Cardano/TxSubmit/Tracing/TraceSubmitApi.hs |
Migrates formatting and counter actions. |
cardano-submit-api/src/Cardano/TxSubmit/Rest/Web.hs |
Migrates REST tracing. |
cardano-submit-api/src/Cardano/TxSubmit/Metrics.hs |
Migrates metrics tracing. |
cardano-submit-api/src/Cardano/TxSubmit.hs |
Uses hermod tracer construction. |
cardano-submit-api/cardano-submit-api.cabal |
Replaces dispatcher dependency. |
cardano-submit-api/.changes/20261001_120000_juergen.nicklisch_hermod_tracing.yml |
Records submit API migration. |
cardano-node/test/Test/Cardano/Tracing/NewTracing/Reconfigure.hs |
Adds runtime reconfiguration tests. |
cardano-node/test/Test/Cardano/Tracing/NewTracing/data/goodConfig.yaml |
Updates valid namespace fixture. |
cardano-node/test/Test/Cardano/Tracing/NewTracing/data/badConfig.yaml |
Updates invalid namespace fixture. |
cardano-node/test/Test/Cardano/Tracing/NewTracing/Consistency.hs |
Derives consistency checks from registry output. |
cardano-node/test/Test/Cardano/Node/FilePermissions.hs |
Migrates capturing tracer. |
cardano-node/test/cardano-node-test.hs |
Registers reconfiguration tests. |
cardano-node/src/Cardano/Node/Tracing/Tracers/Startup.hs |
Adds reload events and migrates metrics. |
cardano-node/src/Cardano/Node/Tracing/Tracers/Shutdown.hs |
Migrates shutdown formatting. |
cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs |
Migrates RPC tracing and counters. |
cardano-node/src/Cardano/Node/Tracing/Tracers/Resources.hs |
Migrates resource tracer APIs. |
cardano-node/src/Cardano/Node/Tracing/Tracers/NodeVersion.hs |
Migrates version metrics. |
cardano-node/src/Cardano/Node/Tracing/Tracers/NodeToNode.hs |
Migrates node-to-node formatting. |
cardano-node/src/Cardano/Node/Tracing/Tracers/NodeToClient.hs |
Migrates node-to-client formatting. |
cardano-node/src/Cardano/Node/Tracing/Tracers/LedgerMetrics.hs |
Migrates periodic ledger metrics. |
cardano-node/src/Cardano/Node/Tracing/Tracers/KESInfo.hs |
Migrates KES tracing. |
cardano-node/src/Cardano/Node/Tracing/Tracers/ForgingStats.hs |
Migrates forging statistics. |
cardano-node/src/Cardano/Node/Tracing/Tracers/ConsensusStartupException.hs |
Migrates startup exception formatting. |
cardano-node/src/Cardano/Node/Tracing/Tracers/Consensus.hs |
Migrates consensus metrics and tracers. |
cardano-node/src/Cardano/Node/Tracing/Tracers/ChainDB.hs |
Migrates ChainDB metrics. |
cardano-node/src/Cardano/Node/Tracing/Tracers/BlockReplayProgress.hs |
Adapts replay tracer construction. |
cardano-node/src/Cardano/Node/Tracing/Tracers.hs |
Centralizes all tracer declarations. |
cardano-node/src/Cardano/Node/Tracing/StateRep.hs |
Migrates node-state tracing. |
cardano-node/src/Cardano/Node/Tracing/Render.hs |
Migrates rendering classes. |
cardano-node/src/Cardano/Node/Tracing/Registry.hs |
Adds unified tracer registry and applier. |
cardano-node/src/Cardano/Node/Tracing/Reconfigure.hs |
Adds serialized SIGHUP trace reloads. |
cardano-node/src/Cardano/Node/Tracing/NodeStartupInfo.hs |
Uses shared startup information type. |
cardano-node/src/Cardano/Node/Tracing/NodeInfo.hs |
Uses shared node information type. |
cardano-node/src/Cardano/Node/Tracing/Formatting.hs |
Migrates formatting classes. |
cardano-node/src/Cardano/Node/Tracing/Era/Shelley.hs |
Migrates Shelley tracing instances. |
cardano-node/src/Cardano/Node/Tracing/Era/HardFork.hs |
Migrates hard-fork tracing instances. |
cardano-node/src/Cardano/Node/Tracing/Era/Byron.hs |
Migrates Byron tracing instances. |
cardano-node/src/Cardano/Node/Tracing/DefaultTraceConfig.hs |
Migrates default trace configuration. |
cardano-node/src/Cardano/Node/Tracing/Consistency.hs |
Replaces handwritten namespace inventory. |
cardano-node/src/Cardano/Node/Tracing/API.hs |
Integrates hermod setup and reload handle. |
cardano-node/src/Cardano/Node/Tracing.hs |
Splits hermod and contra tracer fields. |
cardano-node/src/Cardano/Node/TraceConstraints.hs |
Migrates formatting constraint. |
cardano-node/src/Cardano/Node/Startup.hs |
Adds reload events and shared data types. |
cardano-node/src/Cardano/Node/Run.hs |
Invokes trace reloads on SIGHUP. |
cardano-node/src/Cardano/Node/Parsers.hs |
Migrates parser tracing types. |
cardano-node/src/Cardano/Node/Handlers/Shutdown.hs |
Migrates shutdown handler tracer. |
cardano-node/src/Cardano/Node/Configuration/TopologyP2P.hs |
Migrates topology tracer type. |
cardano-node/src/Cardano/Node/Configuration/POM.hs |
Migrates configuration tracing types. |
cardano-node/ChangeLog.md |
Documents migration, registry, and reloads. |
cardano-node/cardano-node.cabal |
Adds hermod dependencies and new modules. |
cardano-node-chairman/cardano-node-chairman.cabal |
Replaces contra dependency. |
cardano-node-chairman/app/Cardano/Chairman/Commands/Run.hs |
Migrates command tracers. |
cardano-node-chairman/app/Cardano/Chairman.hs |
Migrates chairman tracers. |
cardano-node-chairman/.changes/20261001_120000_juergen.nicklisch_hermod_tracing.yml |
Records chairman migration. |
cabal.project |
Temporarily pins migration dependencies. |
bench/tx-generator/tx-generator.cabal |
Adds hermod dependencies. |
bench/tx-generator/src/Cardano/Benchmarking/Tracer.hs |
Migrates generator tracer setup. |
bench/tx-generator/src/Cardano/Benchmarking/Script/Submission.hs |
Migrates submission tracing. |
bench/tx-generator/src/Cardano/Benchmarking/Script/Env.hs |
Migrates script environment tracing. |
bench/tx-generator/src/Cardano/Benchmarking/Script/Core.hs |
Migrates core tracing. |
bench/tx-generator/src/Cardano/Benchmarking/LogTypes.hs |
Migrates logging instances. |
bench/tx-generator/src/Cardano/Benchmarking/GeneratorTx/SubmissionClient.hs |
Migrates submission client tracing. |
bench/tx-generator/src/Cardano/Benchmarking/GeneratorTx.hs |
Migrates generator tracing. |
bench/tx-generator/src/Cardano/Benchmarking/Command.hs |
Migrates benchmark control tracing. |
bench/tx-generator/CHANGELOG.md |
Documents tx-generator migration. |
bench/trace-schemas/scripts/schema-gen/GhciSchemaGen.hs |
Updates schema source roots. |
bench/trace-schemas/overrides/messages/Forge/StateInfo/StateInfo.schema.override.json |
Updates namespace explanation. |
bench/trace-schemas/newNamespaces.txt |
Regenerates namespace inventory. |
bench/locli/src/Cardano/Unlog/Resources.hs |
Migrates resource types. |
bench/locli/src/Cardano/Unlog/LogObjectDB.hs |
Migrates stored resource types. |
bench/locli/src/Cardano/Unlog/LogObject.hs |
Migrates log resource types. |
bench/locli/src/Cardano/Analysis/API/Types.hs |
Migrates analysis resource types. |
bench/locli/src/Cardano/Analysis/API/Metrics.hs |
Migrates metric resource types. |
bench/locli/locli.cabal |
Replaces resource dependencies. |
bench/locli/CHANGELOG.md |
Documents resource migration. |
bench/cardano-timeseries-io/src/Cardano/Timeseries/Component/Trace.hs |
Migrates timeseries formatting. |
bench/cardano-timeseries-io/src/Cardano/Timeseries/Component.hs |
Migrates component tracing. |
bench/cardano-timeseries-io/src/Cardano/Timeseries/AsText.hs |
Migrates text utility. |
bench/cardano-timeseries-io/CHANGELOG.md |
Documents timeseries migration. |
bench/cardano-timeseries-io/cardano-timeseries-io.cabal |
Replaces tracing/resource dependencies. |
bench/cardano-timeseries-io/app/Cardano/Timeseries/Common.hs |
Migrates resource-statistics reader. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| -- temporary! Please read the section in CONTRIBUTING about updating dependencies. | ||
|
|
||
|
|
||
| -- TEMPORARY, for review only: remove both stanzas before merging. |
| , hermod-tracing-api:public | ||
| , containers | ||
| , data-default-class | ||
| , containers | ||
| , contra-tracer | ||
| , hermod-tracing-api:public |
There was a problem hiding this comment.
Fixed in 212ddfe: each dependency is listed once. That includes cardano-ledger-conway and the plain cardano-ledger-core, which were doubled already on master; cardano-ledger-core:{cardano-ledger-core, testlib} covers the latter.
|
|
||
| -- Read before applying: a malformed or missing file must leave the | ||
| -- running traces untouched. | ||
| read' <- try @Exception.SomeException $ |
There was a problem hiding this comment.
Fixed in 518750a: the three calls use tryAny from safe-exceptions, which catches only synchronous exceptions, so a shutdown propagates out of the reload instead of being reported as a configuration error. modifyMVar_ keeps the previous state when it does.
| read' <- try @Exception.SomeException $ | ||
| readConfigurationWithDefault (FromFile configFile) lastGood |
There was a problem hiding this comment.
Fixed in 66fc845: mkTracingReconfigure takes the defaults separately, and every reload reads the file against defaultCardanoConfig, as at start-up. Only the trace options are taken from it. lastGood is used only to recognise an unchanged reload and to roll back. The regression case prop_emptyOptionsRevertToDefaults reloads with Startup raised to Info, then with {"TraceOptions":{}}, and checks that the Info message is hidden again; it fails with the old read against lastGood.
The network's MetaTrace instances for Mux.BearerTrace (TraceEmitDeltaQ), DNSTrace and TraceChurnMode now give a severity for every namespace, and TraceChurnMode has the inner name ChurnMode. So these tracers are declared like all others: documented, and part of the configuration consistency check. `undocumented` has no user left and goes. ChurnMode is declared with the prefix Net.PeerSelection; with the new inner name its runtime namespace stays Net.PeerSelection.ChurnMode. newNamespaces.txt gains the DNS and Bearer namespaces and ChurnMode. The ouroboros-network pin in cabal.project moves to 8f1f41d6b, which carries those instance changes (and CleanExit at Info).
hermod-tracing's jutaro/api-1.1 was rebased to fold in its review fixes; 6692779 is no longer on the branch. 7195daf is its head.
Every SIGHUP applied the trace options, even when the file had not changed: each trace was reset and reconfigured, so messages traced in between were dropped, and the reflection and the effective configuration were reported again. The reload now compares the options it read with the ones in force and, if they are equal, leaves the traces alone and ends with the new Startup.TraceConfigUnchanged (Notice). Comparing tcOptions suffices: the file is read with the running configuration as the default, so every other field comes from it. The MVar now records whether the traces are known to run with the last good configuration. After a failed apply whose restore failed too they are not, and the next reload applies even unchanged options; otherwise "degraded until the next SIGHUP" could only be repaired by a restart.
Updated tracing implementation to use 'hermod-tracing' and added new DataPoint types for node information. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
containers, contra-tracer, data-default-class, hermod-tracing-api:public,
cardano-ledger-conway and cardano-ledger-core were each listed twice; the
plain cardano-ledger-core is covered by cardano-ledger-core:{cardano-ledger-core,
testlib}.
The reload caught SomeException while reading, applying and restoring the configuration, so a ThreadKilled or another asynchronous exception arriving meanwhile was reported as a configuration error and swallowed, and the SIGHUP handler went on reconfiguring the traces. tryAny from safe-exceptions catches only synchronous exceptions; modifyMVar_ keeps the previous state when one propagates.
A reload read the configuration file against the configuration in force, and hermod takes the whole TraceOptions map from the default when the file's is empty. So removing the last override and sending SIGHUP kept that override in force instead of going back to the node's defaults. The file is now read against the same defaults as at start-up, passed to mkTracingReconfigure as defaultCardanoConfig. Only the trace options are taken from it; the rest stays as it runs, since a reload cannot change it. The last good configuration is used only to recognise an unchanged reload and to roll back.
b469aca to
66fc845
Compare



Moves this repository from
trace-dispatcherandcontra-tracerontohermod-tracing, which is
trace-dispatcherextracted, renamed and splitinto
hermod-tracing-api(instances only),hermod-tracing-core,hermod-tracing-prometheusandhermod-trace-resources.trace-dispatcherstays in the tree for now; nothing in this repositorydepends on it any more.
What a reviewer should know first
contra-tracer does not go away, and is not meant to. It stays wherever a
type crosses the boundary with a library that kept it in its public API —
ouroboros-network,ouroboros-consensusandcardano-rpc. SoCardano.Node.Tracing'sTracersrecord is deliberately mixed: a field takescontra-tracer's carrier exactly when its message type or its consumer belongs
to one of those libraries (
chainDBTracer,churnModeTracer,rpcTracer, theKES-agent signatures in
Run.hs, and the four library-owned tracer records),while the node's own tracers are hermod traces. Keeping the node's own tracers
on hermod is what lets
runPrometheusSimple,startResourceTracerandstartLedgerMetricsTracerwork unchanged.Existing configurations keep working.
hermod-tracing-corestill reads theTraceOptions*layout, deprecated but with the same meaning and precedence.Log output, namespaces, metrics and forwarding are unchanged, with two visible
exceptions in content: the generated documentation's title, and the effective
configuration logged at start-up now includes the
PeriodicTracerssettingsthat
trace-dispatcherleft out.Reading order
The series is one commit per package, each with its final dependency set, so
every commit is a complete migration of what it names:
NodeInfo/NodeStartupInfointotrace-forwardtrace-forward,cardano-timeseries-io,cardano-tracer,cardano-node, then the remaining sixtoContraTracerbridgesnewNamespaces.txt, which had drifted for reasons unrelated to this work — the commit message says whichbuildNodeTracers, with the documentation and the consistency check derived from that declarationCommits 12–14 are where the review value is. Everything before them is
mechanical, and the first commit of the series records the exact substitutions
used, so the diff can be checked against a rule rather than read line by line.
The two source-repository-package stanzas
cabal.projectcarries two temporary pins, both of which must be removedbefore this merges:
hermod-tracing-api 1.1.0, hermod-tracing-core 1.1.0, hermod-tracing-prometheus 1.1.0 hermod-tracing#21.
LogFormattinginstances againsthermod-tracing-apirather thantrace-dispatcher. The flipped packages keep their released version numbers,so no version bound can ask for that build; without the pin the node library
does not compile.
Nothing else is pinned.
ouroboros-consensus,cardano-api,cardano-rpc,cardano-cli,ekg-forwardanddmq-nodeall resolve from CHaP.Verification
GHC 9.6.7,
cabal build allclean andcardano-node-testpassing (40 checks),against the pinned set above and CHaP for everything else. Two of those checks
are worth naming, since they are what would catch a mistake in commits 12–14:
namespace inventory matches documented tracers, andbench/trace-schemas/newNamespaces.txt is current.