Skip to content

Conflicting help texts for node_cpu_frequency_hertz #3689

Open

Description

@jjegg01

Hi,

we noticed an error message every time the Node exporter metrics endpoint is hit on our HPC cluster when enabling --collector.cpu.info:

level=ERROR source=http.go:175 msg="error gathering metrics: 192 error(s) occurred:
* [from Gatherer #2] collected metric node_cpu_frequency_hertz label:{name:"cpu" value:"0"} gauge:{value:2.4e+09} has help "Current CPU thread frequency in hertz." but should have "CPU frequency in hertz from /proc/cpuinfo."
[repeating for all other cores]

Weirdly enough, this only seems to happen on nodes with a relatively high amount of cores (192 in the case above), but not for our smaller nodes. Also, the metric node_cpu_frequency_hertz is exported just fine with Current CPU thread frequency in hertz as the help text. My Go knowledge is basic at best, but it seems a bit strange to me that cpufreq_common.go and cpu_linux.go seem to define a Desc with the same FQName, but with different help texts:

cpuFrequencyHz: prometheus.NewDesc(
prometheus.BuildFQName(namespace, cpuCollectorSubsystem, "frequency_hertz"),
"CPU frequency in hertz from /proc/cpuinfo.",
[]string{"package", "core", "cpu"}, nil,
),

cpuFreqHertzDesc = prometheus.NewDesc(
prometheus.BuildFQName(namespace, cpuCollectorSubsystem, "frequency_hertz"),
"Current CPU thread frequency in hertz.",
[]string{"cpu"}, nil,
)

The error indeed disappears when the two help texts are the same, but I don't know if that only fixes the symptom or if a better fix would be to avoid the FQName duplication in the first place.

Details about test environment

  • Tested with both the EPEL version of node-exporter for Rocky Linux 9 and a fresh build against the master branch:
node_exporter, version 1.11.1 (branch: tarball, revision: 1.el9)
  build user:
  build date:       20260407
  go version:       go1.25.8 (Red Hat 1.25.8-1.el9_7)
  platform:         linux/amd64
  tags:             rpm_crashtraceback,libtrust_openssl,netgo,osusergo,static_build

node_exporter, version  (branch: , revision: b9b6820db7e125a4666280cedb2c3abcd45f9327-modified)
  build user:
  build date:
  go version:       go1.25.7
  platform:         linux/amd64
  tags:             unknown
  • The error message appears on a dual-socket AMD EPYC 9654 node with 192 cores, but not on a dual-socket Intel Xeon Gold 6140 node with 36 cores.
  • Full commandline of node-exporter (the port was changed to avoid interfering with the normal monitoring):
prometheus-node-exporter --web.listen-address=127.0.0.1:8100 --collector.cpu.info

Activity

  1. SuperQ commented on Jul 24, 2026

    @SuperQ
    Member

    Sorry about introducing this bug. At the time of reviewing #3318 I didn't realize that we had a metric name colllison between cpu and cpufreq collectors. This is a bit of an odd situation because we don't usually re-use the "subsystem" names between different collectors.

    As @rtreffer said in #3756 there is some debate about which of the two metrics should take precedence.

    So now we have two competing solutions.

    My personal preference is to do the simple revert proposed in #3749. This restores the previous behavior which isn't optimal but is the least behavior changing.

    As a followup I think we should do the "correctness" thing and fix the metric names coming from cpufreq to be node_cpufreq_....

    But this is a breaking metric name change, which means we will need to put it behind a feature flag. Which is going to make the cpufreq code more complicated. But long-term this solves the collision issues.

  2. jjegg01 commented on Jul 24, 2026

    @jjegg01
    Author

    Wouldn't the revert be a breaking change as well? If you run with default settings plus --collector.cpu.info enabled and cpufreq is not exporting the current frequency (e.g., in a cloud environment), the revert would make node_cpu_frequency_hertz not be exported at all anymore.

    Coincidently, that situation is exactly what we have on our system: The older parts of our HPC cluster runs on Skylake CPUs, which are seemingly not fully supported by cpufreq (at least the cpuinfo_cur_freq in sysfs is missing). And since we run node-exporter as /usr/bin/prometheus-node-exporter --collector.cpu.info, the metric indeed vanishes if I invert the logic in cpu_linux.go. While it would be kind of funny if my issue report about a bit of log spam ends up causing some actual breakage on our system, I think I prefer the current state to that 🙂

    Edit: I just realized that this is probably the real reason why we only see the log spam on the nodes with higher core counts. It just happens to be that all of those nodes are AMD and seem to be better supported by cpufreq.

  3. SuperQ commented on Jul 24, 2026

    @SuperQ
    Member

    We just added #3712 that may be useful for hardware platforms that don't support cur.

  4. rtreffer commented on Aug 24, 2026

    @rtreffer
    Contributor

    @SuperQ I gave this discussion a shot in #3795
    @jjegg01 you would be able to set --collector.cpufreq.enable-cpufreq-prefix to get both metrics exporter under different subsystems.

  5. SuperQ commented on Aug 26, 2026

    @SuperQ
    Member

    Thanks @rtreffer, that's exactly the fix I was thinking about. If there's no other objections soon I'll go with it.

  6. jjegg01 commented on Aug 27, 2026

    @jjegg01
    Author

    @rtreffer Thanks for the work on this! Your fix does indeed fix the duplicate metric issue and thus the log spam.

    However, as I feared, this does also make the node_cpu_frequency_hertz metric vanish on machines that do not have /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq and --collector.cpu.info set unless you enable that new flag. Before, this metric would fall back to the data in /proc/cpuinfo. So overall I would consider this a breaking change. I don't mind switching to the new naming scheme, but anyone unaware of this discussion might run into unexpected problems when updating. Perhaps it is preferable to just add the new flag and leave the bug as is (i.e. don't negate the cpuFreqEnabled conditional) until the new naming scheme becomes the default?

  7. SuperQ commented on Aug 28, 2026

    @SuperQ
    Member

    I much prefer reverting to the original behavior. Yes, it's not ideal, but I don't think we can really fix it without making yet another behavior change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions