Skip to content

Move the repository from trace-dispatcher onto hermod-tracing - #6730

Draft
jutaro wants to merge 24 commits into
masterfrom
jutaro/hermod-tracing-stage2-11.1.1
Draft

jutaro wants to merge 24 commits into
masterfrom
jutaro/hermod-tracing-stage2-11.1.1

Conversation

@jutaro

@jutaro jutaro commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Moves this repository from trace-dispatcher and contra-tracer onto
hermod-tracing, which is trace-dispatcher extracted, renamed and split
into hermod-tracing-api (instances only), hermod-tracing-core,
hermod-tracing-prometheus and hermod-trace-resources.

trace-dispatcher stays in the tree for now; nothing in this repository
depends on it any more.

What a reviewer should know first

contra-tracer does not go away, and is not meant to. It stays wherever a
type crosses the boundary with a library that kept it in its public API —
ouroboros-network, ouroboros-consensus and cardano-rpc. So
Cardano.Node.Tracing's Tracers record is deliberately mixed: a field takes
contra-tracer's carrier exactly when its message type or its consumer belongs
to one of those libraries (chainDBTracer, churnModeTracer, rpcTracer, the
KES-agent signatures in Run.hs, and the four library-owned tracer records),
while the node's own tracers are hermod traces. Keeping the node's own tracers
on hermod is what lets runPrometheusSimple, startResourceTracer and
startLedgerMetricsTracer work unchanged.

Existing configurations keep working. hermod-tracing-core still reads the
TraceOptions* layout, deprecated but with the same meaning and precedence.
Log output, namespaces, metrics and forwarding are unchanged, with two visible
exceptions in content: the generated documentation's title, and the effective
configuration logged at start-up now includes the PeriodicTracers settings
that trace-dispatcher left out.

Reading order

The series is one commit per package, each with its final dependency set, so
every commit is a complete migration of what it names:

commits what they do
1–3 preparation: narrow the facade's surface, move NodeInfo/NodeStartupInfo into trace-forward
4–8 one commit per package: trace-forward, cardano-timeseries-io, cardano-tracer, cardano-node, then the remaining six
9–10 nix profiling, and dropping the last toContraTracer bridges
11 newNamespaces.txt, which had drifted for reasons unrelated to this work — the commit message says which
12–14 the tracer construction rework: every trace declared once in buildNodeTracers, with the documentation and the consistency check derived from that declaration
15 reloading the trace options on SIGHUP

Commits 12–14 are where the review value is. Everything before them is
mechanical, and the first commit of the series records the exact substitutions
used, so the diff can be checked against a rule rather than read line by line.

The two source-repository-package stanzas

cabal.project carries two temporary pins, both of which must be removed
before this merges
:

Nothing else is pinned. ouroboros-consensus, cardano-api, cardano-rpc,
cardano-cli, ekg-forward and dmq-node all resolve from CHaP.

Verification

GHC 9.6.7, cabal build all clean and cardano-node-test passing (40 checks),
against the pinned set above and CHaP for everything else. Two of those checks
are worth naming, since they are what would catch a mistake in commits 12–14:
namespace inventory matches documented tracers, and
bench/trace-schemas/newNamespaces.txt is current.

jutaro added 16 commits October 2, 2026 10:23
Preparation for moving to hermod-tracing, valid on contra-tracer as it is:

- FilePermissions test: build the capturing tracer with mkTracer instead of
  the Tracer constructor and Control.Tracer.Arrow's emit.
- Tracing/API.hs: the disabled forwarding traces are mempty instead of
  `Trace nullTracer`, which relied on trace-dispatcher's Trace constructor
  wrapping a contra-tracer Tracer.
- Import Tracer abstractly where its constructor is not used
  (Cardano.Node.Tracing, tx-generator's NodeToNode).
- Drop the "contra-tracer" and "trace-dispatcher" package qualifiers, which no
  longer disambiguate anything, and the PackageImports pragmas that only served
  them (16 files).

No behaviour change. stylish-haskell 0.14.4.0 on the touched files without
CPP; in cardano-tracer's Notifications/Timer.hs that also sorts the imports.
NodeInfo and NodeStartupInfo are the DataPoints cardano-node forwards at
start-up and cardano-tracer decodes. They were defined in trace-dispatcher,
but they are cardano-node's own payload types, and hermod-tracing (which
replaces trace-dispatcher) does not have them.

trace-forward is the one package both cardano-node and cardano-tracer depend
on, so the two types move there verbatim, as Trace.Forward.Types.NodeInfo and
Trace.Forward.Types.NodeStartupInfo; trace-forward gains a `time` dependency.
The JSON instances are derived as before, so the DataPoints are encoded
exactly as they were, and old and new nodes and tracers keep understanding
each other. cardano-node's MetaTrace instances for them stay in
Cardano.Node.Tracing.{NodeInfo,NodeStartupInfo}; Cardano.Node.Startup
re-exports the new modules as it did the old ones.

Valid with trace-dispatcher as it is, whose copies are simply no longer used.
hermod-tracing, which replaces trace-dispatcher, reads the host name it puts
into every log line from HERMOD_TRACING_LOGGING_HOSTNAME;
TRACE_DISPATCHER_LOGGING_HOSTNAME is only a deprecated fallback there. The
workbench relies on it to give each node of a local cluster its own "host",
which locli uses to tell the forger from the observers. Exporting both names
keeps that working before and after the switch, and lets the old name go
later.
The trace-dispatcher types trace-forward uses (TraceObject, DataPoint, the
forwarding options) come from hermod-tracing-core, which is trace-dispatcher
renamed and split.

Its protocol tracers stay on contra-tracer: acceptorTracer and forwarderTracer
are tracers over Ouroboros.Network.Driver's TraceSendRecv and are handed
straight to runPeer, and Forwarding.hs needs contra-tracer's nullTracer for
ouroboros-network's HandshakeArguments. hermod-tracing-api is a dependency of
neither stanza: only hermod-tracing-core is used, and -Werror=unused-packages
rejects the rest.

This commit starts the switch of the repository; together with the following
ones up to "submit-api, chairman, testnet, ...", it moves every package. Each
of them builds its own packages against the hermod-based ouroboros-network,
ouroboros-consensus, ekg-forward, cardano-api and cardano-cli, but the whole
tree builds again only after that last one, so they are best landed together.

The mechanical part of this and the following commits is
hermod-tracing's scripts/contra-tracer-to-hermod.sh (Control.Tracer ->
Hermod.Tracing.API.Tracer, contra-tracer -> hermod-tracing-api:public) plus
these substitutions:

  Cardano.Logging.Resources[.Types]   -> Hermod.Tracing.Resources[.Types]
  Cardano.Logging.Prometheus.*        -> Hermod.Tracing.Prometheus.*
  Cardano.Logging.{DocuGenerator,Utils,Formatter,Configuration,
    ConfigurationParser,Consistency,FrequencyLimiter,Tracer.*}
                                      -> Hermod.Tracing.<same>
  Cardano.Logging.TraceDispatcherMessage -> Hermod.Tracing.HermodTracingMessage
  Cardano.Logging.Types, Cardano.Logging.Trace, Cardano.Logging
                                      -> Hermod.Tracing (the umbrella; the new
                                         home of Types/Trace is not for user code)
  mkCardanoTracer(')                  -> mkHermodTracer(')
  TraceDispatcherMessage              -> HermodTracingMessage
  PrometheusM                         -> LabelSetM
  tcNodeName                          -> tcApplicationName
  CounterM n Nothing                  -> CounterM n CounterIncrement

By hand here: Forwarding.hs imports the five types it needs from the
Hermod.Tracing umbrella explicitly (the umbrella is much larger than
Cardano.Logging.Types was, and an open import would shadow its local
mkTracer); the cabal entries; the README sentence about who feeds the library.

The tracers in the forwarder/acceptor configurations keep contra-tracer's
carrier, so nothing breaks for their users.
The same move as the rest of the repository (see the trace-forward commit):
Cardano.Logging -> Hermod.Tracing (hermod-tracing-core) and
Cardano.Logging.Resources -> Hermod.Tracing.Resources (hermod-trace-resources),
whose API is identical. cardano-tracer depends on this package for its
timeseries traces, so it moves first.

Breaking for users of the library: TimeseriesTrace's LogFormatting / MetaTrace
instances are hermod's now. The traces emitted and the resource statistics read
are unchanged.
cardano-tracer receives what trace-forward and ekg-forward hand it, and both
now speak hermod: TraceObject, DataPoint and the forwarding options come from
hermod-tracing-core, the Prometheus exposition lives in
hermod-tracing-prometheus and the resource statistics in
hermod-trace-resources. The forwarding protocol tracers stay on contra-tracer,
because trace-forward hands them straight to ouroboros-network's runPeer.

Mechanical as described in the trace-forward commit, plus by hand:
- MetaTrace: machineFormatter is pure in hermod, so it is applied before
  filterSeverityFromConfig instead of being bound; the removed contramapM' is
  replaced by mkTrace, which, like it, prints messages and ignores control
  messages; withNames/withSeverity are imported from
  Hermod.Tracing.Trace.Annotations, which the umbrella does not re-export.
- ReForwarder: the separate DataPoint import is redundant behind the umbrella.
- bench: drop `hiding (LocalSocket)`, a name hermod no longer exports.
- cabal: hermod-tracing-core ^>= 1.0.1, the release that reads the forwarder
  options (also in cardano-tracer's own hasForwarding) and the host-name
  variable as trace-dispatcher did.
- docs: name ApplicationName / MetricsPrefix, and say that the TraceOption*
  spellings are still read in the deprecated TraceOptions layout.

The configuration format, the logs written and the metrics served are
unchanged; old and new nodes and tracers interoperate (the TraceObject and
DataPoint encodings are the same).
The node's tracing is built on hermod-tracing-core (trace-dispatcher renamed
and split), hermod-tracing-prometheus and hermod-trace-resources.

ouroboros-network's and ouroboros-consensus' public libraries hold
contra-tracer Tracers in their tracer records, so Cardano.Node.Tracing's
Tracers record is mixed rather than uniformly hermod: a field takes
contra-tracer's carrier exactly when its message type or its consumer belongs
to one of those libraries -- chainDBTracer, churnModeTracer, rpcTracer, the KES
agent signatures in Run.hs and the four library-owned tracer records -- and the
node's own tracers are hermod traces.

Mechanical as described in the trace-forward commit. By hand:
- Tracers.hs keeps its mkTracer (traceWith tr) wrappers. The plain mkTracer
  comes from the Hermod.Tracing umbrella, which re-exports contra-tracer's, and
  builds the library-facing fields; the node's own fields take the facade's,
  qualified as HT. Behaviour is unchanged: nothing sends control messages
  through the tracers the libraries hold. (The wrappers go when tracer
  construction is reworked.)
- CounterM n (Just i) -> CounterM n (CounterAdd (fromIntegral i)).
- The resource and ledger-metrics periods are read from PeriodicTracers
  ("resources", "ledgerMetrics") with the same fallbacks as before, clamped to
  Int; DefaultTraceConfig no longer needs to discard a library default.
- hermod-tracing-prometheus and cardano-rpc still take contra-tracer tracers:
  bridged with toContraTracer (Tracing/API.hs, Run.hs).
- contramapMCond is pure in hermod (BlockReplayProgress).
- Documentation imports HermodTracingMessage, which the umbrella does not
  re-export.
- README: the dependency diagram names hermod-tracing.

Existing configurations keep working unchanged: hermod-tracing-core 1.0.1
still reads the TraceOptions* layout (deprecated), with the same meaning and
precedence. Log output, namespaces, metrics and forwarding are the same as
before, with two visible exceptions in content: the generated trace
documentation's title ("Tracing Documentation"), and the effective
configuration logged at start-up (Reflection.TracerInfoConfig), which now
includes the PeriodicTracers settings that trace-dispatcher left out.
…ermod-tracing

The remaining packages follow cardano-node onto hermod-tracing and the hermod
tracer facade (mechanical part as described in the trace-forward commit):
- cardano-submit-api: Cardano.Logging -> Hermod.Tracing; CounterM takes a
  CounterAction. Its configuration keeps working (deprecated TraceOptions).
- cardano-node-chairman: Control.Tracer -> facade.
- cardano-testnet: Control.Tracer -> facade, except at the ouroboros-network
  boundary, where the ping client of cardano-diffusion keeps contra-tracer. Breaking for library users:
  Testnet.ChainWatchdog's chainStallWatchdog and stderrTracer use hermod
  traces. Generated node configurations are unchanged (they use the deprecated
  layout, as the comment there now says).
- tx-generator: forwardFormatter/machineFormatter are pure in hermod; the
  annotation combinators come from Hermod.Tracing.Trace.Annotations; the
  remaining mkTracer (traceWith ...) wrapper takes the umbrella's mkTracer, and
  NodeToNode.hs keeps contra-tracer's Tracer for the ouroboros-network
  boundary.
- locli: Cardano.Logging.Resources.Types -> Hermod.Tracing.Resources.Types.
- trace-schema-gen: scan hermod's sources (api, core, prometheus,
  trace-resources) instead of trace-dispatcher's.

With this commit the whole repository builds again.
The profiled build variant lists the tracing packages to build with
-fprof-auto; trace-resources is now hermod-trace-resources.
hermod-tracing-prometheus 1.1 and cardano-rpc now take hermod traces, so the
node passes its tracers to runPrometheusSimple(With), mkNodeKernelAccess and
runRpcServer directly. The three toContraTracer bridges and both
Hermod.Tracing.API.ContraTracer imports go.

This removes the bridges, not the dependency. contra-tracer is still named
directly wherever a type crosses the boundary with a library that kept it in
its public API -- ouroboros-network, ouroboros-consensus and cardano-rpc --
which is why several packages here still depend on it.

cardano-node requires hermod-tracing-prometheus ^>= 1.1. cardano-tracer uses
only the Exposition module, which 1.1 leaves unchanged, so it accepts 1.0 and
1.1.
The checked-in list had drifted from what `cardano-node trace-documentation`
produces, for reasons that have nothing to do with the hermod migration.

The Startup namespaces it gains (BlockForgingUpdateError, the DiffusionInit
ones, NetworkConfigUpdateInfo/Warning, RpcUnsupportedBlockType) were already in
the node's sources at the branch point and merely missing from the file. The
ChainDB rename IgnoreBlockOlderThanK -> IgnoreBlockOlderThanImmTip, the removal
of the LMDB backing-store namespaces and the Peras, PerasVoteDb and KESAgent
additions follow the ouroboros-consensus version this branch builds against.

Regenerated with
bench/trace-schemas/scripts/schema-gen/RegenerateTraceSchemas.sh. None of this
comes from the migration commits that precede it, which is why it stands on
its own rather than being folded into one of them.
The node declares its tracers three times by hand: the runtime construction
in Tracers.hs, the documentation in Documentation.hs and the consistency check
in Consistency.hs. Only the first is compiler-checked, and the other two have
drifted from it (Forge.Stats is documented as Forge.ThreadStats and not
checked; Net.DNS is checked as Net.DNSResolver; the KES-info and
peer-selection-counter prefixes differ; several tracers are in one list and
not the others).

This commit makes the runtime construction the single declaration. The new
module Cardano.Node.Tracing.Registry provides the constructors: newTrace
builds a hermod trace and records it (prefix, trace, how it is listed), so the
same declaration can later drive the documentation and the consistency check.
buildNodeTracers in Tracers.hs is that declaration: one `newTrace registry
["A", "B"]` per tracer, with the element types inferred from the library
records as before, so no type annotations are needed. mkDispatchTracers
builds, then configures every recorded entry, then performs the start-up
messages as before. Its unused NodeKernelData parameter goes.

Entries that only exist for the documentation (Reflection, the BlockFetch
client metrics) are declared with docOnly and are not built at runtime.
Entries whose MetaTrace instances in ouroboros-network would make the
documentation or the check warn (ChurnMode, DNS, the Mux bearer traces) are
declared as undocumented or runtime-only; the comments name the deficiency.

The two hooks (ChainDB's AddedToCurrentChain limiter, the forging statistics)
are applied to the constructed trace instead of inside mkHermodTracer', which
had applied them separately to the message and the metrics path. Each now
has one state instead of two identical ones; the output is the same.

The runtime consistency check now reads the recorded entries, so Tracers.hs
no longer imports Consistency. Documentation.hs and Consistency.hs are
unchanged in this commit; they follow in the next one.
…k from it

The documentation and the configuration consistency check no longer keep
their own lists of the node's tracers. Both now run buildNodeTracers, with
backends that only answer documentation requests, and read the registry it
fills: the documentation documents every recorded trace, the check takes
the recorded prefixes and the MetaTrace instances' namespaces. The 74
type-annotated reconstructions in Documentation.hs and the 77 annotated
namespace blocks in Consistency.hs go, as does the Proxy trick that fixed
the block type; one type application does that now.

Both therefore follow the runtime names, which changes the documented and
checked namespaces where the lists had drifted:
- Forge.StateInfo and Forge.ThreadStats.ForgingStats become
  Forge.StateInfo.StateInfo and Forge.Stats.ForgingStats, the names the
  node emits (the schema override moves with the first);
- Net.PeerSelection.Counters.Counters becomes Net.PeerSelection.Counters;
- the check knows Net.DNS instead of Net.DNSResolver, and NodeInfo,
  NodeStartupInfo and Reflection, which it rejected;
- the Mux channel and the handshake tracers are documented;
- Net.AcceptPolicy and Net.PeerSelection.Responder, which no tracer emits,
  are dropped from both.
The metrics help text is unchanged.

The consistency test's allowlists shrink to the three tracers declared
undocumented, and a new golden test keeps bench/trace-schemas/newNamespaces.txt
equal to what trace-documentation generates. The test fixtures use the
current names.
Net.ErrorPolicy, Net.AcceptPolicy and Net.PeerSelection.Responder name no
tracer the node has; with the consistency check now derived from the
tracers the node declares, they would be reported as illegal namespaces at
start-up. Net.DNSResolver is the check's old spelling of Net.DNS; in the
workbench it had the severity of its parent key, so the rename changes
nothing. The changelog entry covers the previous two commits as well.
Hermod has always advertised hot reconfiguration -- configureTracers' haddock
says "Call this function at initialisation, and later for reconfiguration" --
but the node never used it: changing a severity meant a restart. Everything
needed was already in place after the registry work; the only thing missing
was that the registry, the entries and the TraceConfig were all local to
mkDispatchTracers and discarded on return, so nothing survived for a later
SIGHUP to use.

Registry: the apply-and-report step that mkDispatchTracers had inlined becomes
ApplyTraceConfig, built by `applier` next to configureAll, so the start-up
pass and the reload pass are the same code and cannot drift apart. It has two
entry points: applyTraceConfig reports the reflection, the consistency
warnings and the effective configuration; restoreTraceConfig reports only the
effective configuration, for putting the previous one back after a failure,
where the rest is noise. Each pass builds a fresh ConfigReflection, because
traceTracerInfo empties the one it reports.

New Cardano.Node.Tracing.Reconfigure holds the reload itself. An MVar
TraceConfig is both the mutex and the last configuration known to apply
cleanly, which makes "store it only on success" the same step as releasing the
lock. The order matters: trace the start marker under the configuration still
in force, then read -- so a malformed or missing file never perturbs a running
trace -- then apply, and on failure restore before reporting, because a
half-applied pass can leave the startup tracer itself discarding everything.
try @someexception and not a narrower type: this path throws ParseException
(malformed YAML and a missing file alike), AesonException, and a bare error for
more than four "HermodTracing" redirects.

readConfigurationWithDefault with the running configuration as the default is
what makes this the trace-options part and not the whole configuration:
mergeWithDefault replaces tcOptions wholesale and takes the forwarder,
application name, metrics prefix, periodic tracers and PrometheusSimple from
the default. An operator who edits only TraceOptions changes only the options.

Run.hs: initTraceDispatcher returns the handle alongside the tracers, and it
is threaded to both SIGHUP handlers -- the early one and the one
installSigHUPHandler reinstalls once the node kernel exists (the non-UNIX stub
grows from 14 to 15 wildcards). Reconfiguring runs first in both, so the
handler's own later messages are emitted under the new settings; unlike block
forging there is no "not effective until consensus is up" caveat, and DB
replay is exactly when an operator wants to change ChainDB severity. The
handler's tracing job is that single call with no logic in it, so what cannot
be tested through a real signal is also trivial.

Reporting is three Startup constructors -- TraceConfigUpdate and
TraceConfigUpdated at Notice, TraceConfigUpdateError at Error -- modelled on
NetworkConfigUpdate/RpcConfigUpdateError. No per-field "needs a restart"
detection; the fields that cannot be reloaded are listed in the ChangeLog
instead, along with the non-atomic drop window and the pointer to hermod's
Reflection.TracerInfoConfig, which traceEffectiveConfiguration emits
unconditionally and so is the authoritative record of what is in force.
bench/trace-schemas/newNamespaces.txt is regenerated, not hand-edited.

Test/Cardano/Tracing/NewTracing/Reconfigure.hs drives the node's own registry
over a capturing stdout backend, so it covers every declared trace: that
A -> B -> A changes the output and is reversible rather than latching (the
property hermod's own runScriptMultithreadedWithReconfig cannot assert, its
oracle being pinned to the pre-reconfiguration config); that invalid YAML, a
bad enum and a deleted file are each a complete no-op and each reported (the
deleted-file case is why readConfiguration' must not be used here -- it
silently substitutes a minimal config); and that two concurrent reloads
serialise. Nothing anywhere covered two concurrent reconfigurations, which is
exactly the SIGHUP hazard: with the mutex removed the second pass throws
HermodConfigException "Trace not reset before reconfiguration" and the node
goes deaf, and the property catches it.

Verified: `cabal build all -j` green; `cardano-node-test` 30/30 (from
cardano-node/, including the namespace-list golden, which the regenerated
newNamespaces.txt keeps passing); stylish-haskell 0.14.4.0 clean on every
touched non-CPP module.

Verified on a real node (mainnet config, empty db, three real SIGHUPs;
wt/logs/smoke/n19-sighup):

  root Notice -> Debug      Startup.TraceConfigUpdate, Startup.TraceConfigUpdated
                            and Reflection.TracerInfoConfig once each, then
                            131210 new Debug-severity lines where there had
                            been none.
  malformed YAML            Startup.TraceConfigUpdateError once ("YAML parse
                            exception at line 1, column 2"), no Updated, Debug
                            still flowing, node alive: a complete no-op.
  root Debug -> Notice      Updated once, then no Debug line at all; the four
                            remaining Info lines are ChainDB, which the mainnet
                            config pins to Info.

The workbench cluster could not be used: `make shell` in this worktree fails in
haskell.nix's plan-to-nix with "unknown package: hermod-tracing-prometheus",
because nothing is published to CHaP yet. The single-node harness above
exercises the same code -- a real signal, the real handler, the real file.
…builds

Lets CI build this branch before CHaP has hermod-tracing-api 1.1.0,
hermod-tracing-core 1.0.1 and hermod-tracing-prometheus 1.1.0, which are
IntersectMBO/hermod-tracing#21.

ouroboros-network is pinned for a different reason: its tracing sublibraries
provide their LogFormatting instances against hermod-tracing-api rather than
trace-dispatcher. The flipped packages keep their released version numbers, so
no version bound can ask for that build. Without the pin the node library
fails to compile with "Could not deduce LogFormatting (Mux.WithBearer
(ConnectionId RemoteAddress) Mux.Trace)" and seven more of the same shape.

Nothing else needs pinning. ouroboros-consensus, cardano-api, cardano-rpc,
cardano-cli, ekg-forward and dmq-node all resolve from CHaP: cardano-node
builds and its test suite passes against exactly this set.

Both stanzas must be removed before this merges.
@jutaro
jutaro force-pushed the jutaro/hermod-tracing-stage2-11.1.1 branch from c9c0cf1 to 52cd97e Compare October 9, 2026 08:23
@jutaro
jutaro requested a balanced review from Copilot October 9, 2026 08:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Runtime reload defaults are incorrectly latched, asynchronous cancellation is swallowed, and the explicitly temporary dependency pins remain present.

5 open findings
What changed in this PR

Migrates repository tracing from trace-dispatcher to hermod-tracing, centralizes node tracer registration, and adds SIGHUP tracing reconfiguration.

Changes:

  • Replaces tracing/resource dependencies and imports across packages.
  • Introduces a unified node trace registry for runtime configuration, documentation, and namespace checks.
  • Adds serialized runtime trace-option reloads and related tests.
File Description
trace-forward/​trace-forward.cabal Updates tracing dependencies and exports node data types.
trace-forward/​src/​Trace/​Forward/​Utils/​TraceObject.hs Uses hermod trace-object utilities.
trace-forward/​src/​Trace/​Forward/​Utils/​DataPoint.hs Uses hermod data-point APIs.
trace-forward/​src/​Trace/​Forward/​Types/​NodeStartupInfo.hs Adds shared startup information type.
trace-forward/​src/​Trace/​Forward/​Types/​NodeInfo.hs Adds shared node information type.
trace-forward/​src/​Trace/​Forward/​Protocol/​TraceObject/​Type.hs Migrates TraceObject import.
trace-forward/​src/​Trace/​Forward/​Protocol/​DataPoint/​Type.hs Migrates data-point name import.
trace-forward/​src/​Trace/​Forward/​Forwarding.hs Migrates forwarding options and utilities.
trace-forward/​src/​Trace/​Forward/​Configuration/​TraceObject.hs Reorders trace-object configuration imports.
trace-forward/​src/​Trace/​Forward/​Configuration/​DataPoint.hs Reorders data-point configuration imports.
trace-forward/​README.md Updates tracing architecture description.
trace-forward/​CHANGELOG.md Documents migration and shared types.
README.md Updates the dependency diagram.
nix/​workbench/​service/​tracing.nix Updates network trace namespaces.
nix/​workbench/​service/​nodes.nix Adds hermod logging hostname.
nix/​nixos/​cardano-tracer-service.nix Updates metrics-prefix documentation.
nix/​haskell.nix Profiles the renamed resource package.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​Utils.hs Migrates test network types.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​Restart/​Tests.hs Migrates restart tests.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​Queue/​Tests.hs Migrates queue tests.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​Logs/​Tests.hs Migrates log tests.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​ForwardingStressTest/​Types.hs Migrates stress-test tracing types.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​ForwardingStressTest/​Script.hs Migrates stress-test tracer setup.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​ForwardingStressTest/​Messages.hs Migrates stress-test formatting.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​ForwardingStressTest/​Config.hs Migrates stress-test configuration.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​Forwarder.hs Migrates forwarder test helpers.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​DataPoint/​Tests.hs Migrates data-point tests.
cardano-tracer/​test/​Cardano/​Tracer/​Test/​Acceptor.hs Migrates acceptor tests.
cardano-tracer/​test/​cardano-tracer-test-ext.hs Uses mkHermodTracer.
cardano-tracer/​src/​Cardano/​Tracer/​Utils.hs Uses shared NodeInfo.
cardano-tracer/​src/​Cardano/​Tracer/​Run.hs Migrates resource tracing.
cardano-tracer/​src/​Cardano/​Tracer/​MetaTrace.hs Reworks hermod meta tracing.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​State/​TraceObjects.hs Migrates trace-object state handling.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​ReForwarder.hs Migrates re-forwarding.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Notifications/​Utils.hs Migrates notification traces.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Notifications/​Types.hs Migrates notification severities.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Notifications/​Timer.hs Migrates timer traces.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Notifications/​Send.hs Migrates notification logging.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Notifications/​Check.hs Migrates severity checks.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Metrics/​TimeseriesServer.hs Uses shared node information types.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Metrics/​Prometheus.hs Migrates Prometheus exposition.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Logs/​TraceObjects.hs Migrates trace-object logging.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Logs/​Journal/​Systemd.hs Migrates systemd journal tracing.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Logs/​Journal/​NoSystemd.hs Migrates no-systemd stub types.
cardano-tracer/​src/​Cardano/​Tracer/​Handlers/​Logs/​File.hs Migrates file trace objects.
cardano-tracer/​src/​Cardano/​Tracer/​Environment.hs Migrates tracer environment types.
cardano-tracer/​src/​Cardano/​Tracer/​Configuration.hs Migrates tracing configuration types.
cardano-tracer/​src/​Cardano/​Tracer/​CLI.hs Migrates CLI tracing imports.
cardano-tracer/​src/​Cardano/​Tracer/​Acceptors/​Server.hs Migrates server acceptor tracing.
cardano-tracer/​src/​Cardano/​Tracer/​Acceptors/​Run.hs Migrates acceptor runtime utilities.
cardano-tracer/​src/​Cardano/​Tracer/​Acceptors/​Client.hs Migrates client acceptor tracing.
cardano-tracer/​docs/​cardano-tracer.md Updates application-name documentation.
cardano-tracer/​demo/​ssh/​forwarder.hs Migrates demo network types.
cardano-tracer/​CHANGELOG.md Documents tracer migration.
cardano-tracer/​cardano-tracer.cabal Replaces tracing dependencies.
cardano-tracer/​bench/​cardano-tracer-bench.hs Migrates benchmark tracing.
cardano-testnet/​src/​Testnet/​Ping.hs Reorders tracer import.
cardano-testnet/​src/​Testnet/​Defaults.hs Documents deprecated configuration layout.
cardano-testnet/​src/​Testnet/​ChainWatchdog.hs Migrates watchdog tracers.
cardano-testnet/​cardano-testnet.cabal Adds hermod API dependency.
cardano-testnet/​.changes/​20261001_120000_juergen.nicklisch_hermod_tracing.yml Records testnet API migration.
cardano-submit-api/​src/​Cardano/​TxSubmit/​Web.hs Migrates web tracing.
cardano-submit-api/​src/​Cardano/​TxSubmit/​Util.hs Migrates exception logging.
cardano-submit-api/​src/​Cardano/​TxSubmit/​Tracing/​TraceSubmitApi.hs Migrates formatting and counter actions.
cardano-submit-api/​src/​Cardano/​TxSubmit/​Rest/​Web.hs Migrates REST tracing.
cardano-submit-api/​src/​Cardano/​TxSubmit/​Metrics.hs Migrates metrics tracing.
cardano-submit-api/​src/​Cardano/​TxSubmit.hs Uses hermod tracer construction.
cardano-submit-api/​cardano-submit-api.cabal Replaces dispatcher dependency.
cardano-submit-api/​.changes/​20261001_120000_juergen.nicklisch_hermod_tracing.yml Records submit API migration.
cardano-node/​test/​Test/​Cardano/​Tracing/​NewTracing/​Reconfigure.hs Adds runtime reconfiguration tests.
cardano-node/​test/​Test/​Cardano/​Tracing/​NewTracing/​data/​goodConfig.yaml Updates valid namespace fixture.
cardano-node/​test/​Test/​Cardano/​Tracing/​NewTracing/​data/​badConfig.yaml Updates invalid namespace fixture.
cardano-node/​test/​Test/​Cardano/​Tracing/​NewTracing/​Consistency.hs Derives consistency checks from registry output.
cardano-node/​test/​Test/​Cardano/​Node/​FilePermissions.hs Migrates capturing tracer.
cardano-node/​test/​cardano-node-test.hs Registers reconfiguration tests.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​Startup.hs Adds reload events and migrates metrics.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​Shutdown.hs Migrates shutdown formatting.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​Rpc.hs Migrates RPC tracing and counters.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​Resources.hs Migrates resource tracer APIs.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​NodeVersion.hs Migrates version metrics.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​NodeToNode.hs Migrates node-to-node formatting.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​NodeToClient.hs Migrates node-to-client formatting.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​LedgerMetrics.hs Migrates periodic ledger metrics.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​KESInfo.hs Migrates KES tracing.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​ForgingStats.hs Migrates forging statistics.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​ConsensusStartupException.hs Migrates startup exception formatting.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​Consensus.hs Migrates consensus metrics and tracers.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​ChainDB.hs Migrates ChainDB metrics.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers/​BlockReplayProgress.hs Adapts replay tracer construction.
cardano-node/​src/​Cardano/​Node/​Tracing/​Tracers.hs Centralizes all tracer declarations.
cardano-node/​src/​Cardano/​Node/​Tracing/​StateRep.hs Migrates node-state tracing.
cardano-node/​src/​Cardano/​Node/​Tracing/​Render.hs Migrates rendering classes.
cardano-node/​src/​Cardano/​Node/​Tracing/​Registry.hs Adds unified tracer registry and applier.
cardano-node/​src/​Cardano/​Node/​Tracing/​Reconfigure.hs Adds serialized SIGHUP trace reloads.
cardano-node/​src/​Cardano/​Node/​Tracing/​NodeStartupInfo.hs Uses shared startup information type.
cardano-node/​src/​Cardano/​Node/​Tracing/​NodeInfo.hs Uses shared node information type.
cardano-node/​src/​Cardano/​Node/​Tracing/​Formatting.hs Migrates formatting classes.
cardano-node/​src/​Cardano/​Node/​Tracing/​Era/​Shelley.hs Migrates Shelley tracing instances.
cardano-node/​src/​Cardano/​Node/​Tracing/​Era/​HardFork.hs Migrates hard-fork tracing instances.
cardano-node/​src/​Cardano/​Node/​Tracing/​Era/​Byron.hs Migrates Byron tracing instances.
cardano-node/​src/​Cardano/​Node/​Tracing/​DefaultTraceConfig.hs Migrates default trace configuration.
cardano-node/​src/​Cardano/​Node/​Tracing/​Consistency.hs Replaces handwritten namespace inventory.
cardano-node/​src/​Cardano/​Node/​Tracing/​API.hs Integrates hermod setup and reload handle.
cardano-node/​src/​Cardano/​Node/​Tracing.hs Splits hermod and contra tracer fields.
cardano-node/​src/​Cardano/​Node/​TraceConstraints.hs Migrates formatting constraint.
cardano-node/​src/​Cardano/​Node/​Startup.hs Adds reload events and shared data types.
cardano-node/​src/​Cardano/​Node/​Run.hs Invokes trace reloads on SIGHUP.
cardano-node/​src/​Cardano/​Node/​Parsers.hs Migrates parser tracing types.
cardano-node/​src/​Cardano/​Node/​Handlers/​Shutdown.hs Migrates shutdown handler tracer.
cardano-node/​src/​Cardano/​Node/​Configuration/​TopologyP2P.hs Migrates topology tracer type.
cardano-node/​src/​Cardano/​Node/​Configuration/​POM.hs Migrates configuration tracing types.
cardano-node/​ChangeLog.md Documents migration, registry, and reloads.
cardano-node/​cardano-node.cabal Adds hermod dependencies and new modules.
cardano-node-chairman/​cardano-node-chairman.cabal Replaces contra dependency.
cardano-node-chairman/​app/​Cardano/​Chairman/​Commands/​Run.hs Migrates command tracers.
cardano-node-chairman/​app/​Cardano/​Chairman.hs Migrates chairman tracers.
cardano-node-chairman/​.changes/​20261001_120000_juergen.nicklisch_hermod_tracing.yml Records chairman migration.
cabal.project Temporarily pins migration dependencies.
bench/​tx-generator/​tx-generator.cabal Adds hermod dependencies.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​Tracer.hs Migrates generator tracer setup.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​Script/​Submission.hs Migrates submission tracing.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​Script/​Env.hs Migrates script environment tracing.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​Script/​Core.hs Migrates core tracing.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​LogTypes.hs Migrates logging instances.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​GeneratorTx/​SubmissionClient.hs Migrates submission client tracing.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​GeneratorTx.hs Migrates generator tracing.
bench/​tx-generator/​src/​Cardano/​Benchmarking/​Command.hs Migrates benchmark control tracing.
bench/​tx-generator/​CHANGELOG.md Documents tx-generator migration.
bench/​trace-schemas/​scripts/​schema-gen/​GhciSchemaGen.hs Updates schema source roots.
bench/​trace-schemas/​overrides/​messages/​Forge/​StateInfo/​StateInfo.schema.override.json Updates namespace explanation.
bench/​trace-schemas/​newNamespaces.txt Regenerates namespace inventory.
bench/​locli/​src/​Cardano/​Unlog/​Resources.hs Migrates resource types.
bench/​locli/​src/​Cardano/​Unlog/​LogObjectDB.hs Migrates stored resource types.
bench/​locli/​src/​Cardano/​Unlog/​LogObject.hs Migrates log resource types.
bench/​locli/​src/​Cardano/​Analysis/​API/​Types.hs Migrates analysis resource types.
bench/​locli/​src/​Cardano/​Analysis/​API/​Metrics.hs Migrates metric resource types.
bench/​locli/​locli.cabal Replaces resource dependencies.
bench/​locli/​CHANGELOG.md Documents resource migration.
bench/​cardano-timeseries-io/​src/​Cardano/​Timeseries/​Component/​Trace.hs Migrates timeseries formatting.
bench/​cardano-timeseries-io/​src/​Cardano/​Timeseries/​Component.hs Migrates component tracing.
bench/​cardano-timeseries-io/​src/​Cardano/​Timeseries/​AsText.hs Migrates text utility.
bench/​cardano-timeseries-io/​CHANGELOG.md Documents timeseries migration.
bench/​cardano-timeseries-io/​cardano-timeseries-io.cabal Replaces tracing/resource dependencies.
bench/​cardano-timeseries-io/​app/​Cardano/​Timeseries/​Common.hs Migrates resource-statistics reader.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cabal.project
-- temporary! Please read the section in CONTRIBUTING about updating dependencies.


-- TEMPORARY, for review only: remove both stanzas before merging.
Comment thread cardano-testnet/cardano-testnet.cabal Outdated
Comment on lines +64 to +68
, hermod-tracing-api:public
, containers
, data-default-class
, containers
, contra-tracer
, hermod-tracing-api:public

@jutaro jutaro Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 212ddfe: each dependency is listed once. That includes cardano-ledger-conway and the plain cardano-ledger-core, which were doubled already on master; cardano-ledger-core:{cardano-ledger-core, testlib} covers the latter.


-- Read before applying: a malformed or missing file must leave the
-- running traces untouched.
read' <- try @Exception.SomeException $

@jutaro jutaro Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 518750a: the three calls use tryAny from safe-exceptions, which catches only synchronous exceptions, so a shutdown propagates out of the reload instead of being reported as a configuration error. modifyMVar_ keeps the previous state when it does.

Comment on lines +72 to +73
read' <- try @Exception.SomeException $
readConfigurationWithDefault (FromFile configFile) lastGood

@jutaro jutaro Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 66fc845: mkTracingReconfigure takes the defaults separately, and every reload reads the file against defaultCardanoConfig, as at start-up. Only the trace options are taken from it. lastGood is used only to recognise an unchanged reload and to roll back. The regression case prop_emptyOptionsRevertToDefaults reloads with Startup raised to Info, then with {"TraceOptions":{}}, and checks that the Info message is hidden again; it fails with the old read against lastGood.

Comment thread trace-forward/CHANGELOG.md Resolved
jutaro and others added 7 commits October 12, 2026 11:58
The network's MetaTrace instances for Mux.BearerTrace (TraceEmitDeltaQ),
DNSTrace and TraceChurnMode now give a severity for every namespace, and
TraceChurnMode has the inner name ChurnMode. So these tracers are
declared like all others: documented, and part of the configuration
consistency check. `undocumented` has no user left and goes.

ChurnMode is declared with the prefix Net.PeerSelection; with the new
inner name its runtime namespace stays Net.PeerSelection.ChurnMode.

newNamespaces.txt gains the DNS and Bearer namespaces and ChurnMode.

The ouroboros-network pin in cabal.project moves to 8f1f41d6b, which
carries those instance changes (and CleanExit at Info).
hermod-tracing's jutaro/api-1.1 was rebased to fold in its review
fixes; 6692779 is no longer on the branch. 7195daf is its head.
Every SIGHUP applied the trace options, even when the file had not
changed: each trace was reset and reconfigured, so messages traced in
between were dropped, and the reflection and the effective configuration
were reported again.

The reload now compares the options it read with the ones in force and,
if they are equal, leaves the traces alone and ends with the new
Startup.TraceConfigUnchanged (Notice). Comparing tcOptions suffices: the
file is read with the running configuration as the default, so every
other field comes from it.

The MVar now records whether the traces are known to run with the last
good configuration. After a failed apply whose restore failed too they
are not, and the next reload applies even unchanged options; otherwise
"degraded until the next SIGHUP" could only be repaired by a restart.
Updated tracing implementation to use 'hermod-tracing' and added new DataPoint types for node information.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
containers, contra-tracer, data-default-class, hermod-tracing-api:public,
cardano-ledger-conway and cardano-ledger-core were each listed twice; the
plain cardano-ledger-core is covered by cardano-ledger-core:{cardano-ledger-core,
testlib}.
The reload caught SomeException while reading, applying and restoring the
configuration, so a ThreadKilled or another asynchronous exception arriving
meanwhile was reported as a configuration error and swallowed, and the
SIGHUP handler went on reconfiguring the traces. tryAny from safe-exceptions
catches only synchronous exceptions; modifyMVar_ keeps the previous state
when one propagates.
A reload read the configuration file against the configuration in force,
and hermod takes the whole TraceOptions map from the default when the file's
is empty. So removing the last override and sending SIGHUP kept that override
in force instead of going back to the node's defaults.

The file is now read against the same defaults as at start-up, passed to
mkTracingReconfigure as defaultCardanoConfig. Only the trace options are taken
from it; the rest stays as it runs, since a reload cannot change it. The last
good configuration is used only to recognise an unchanged reload and to roll
back.
@jutaro
jutaro force-pushed the jutaro/hermod-tracing-stage2-11.1.1 branch from b469aca to 66fc845 Compare October 12, 2026 12:39
@mgmeier mgmeier added the WIP Work In Progress (cannot be merged yet) label Oct 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WIP Work In Progress (cannot be merged yet)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants