Repository navigation
Description
Hi,
we noticed an error message every time the Node exporter metrics endpoint is hit on our HPC cluster when enabling --collector.cpu.info:
level=ERROR source=http.go:175 msg="error gathering metrics: 192 error(s) occurred:
* [from Gatherer #2] collected metric node_cpu_frequency_hertz label:{name:"cpu" value:"0"} gauge:{value:2.4e+09} has help "Current CPU thread frequency in hertz." but should have "CPU frequency in hertz from /proc/cpuinfo."
[repeating for all other cores]
Weirdly enough, this only seems to happen on nodes with a relatively high amount of cores (192 in the case above), but not for our smaller nodes. Also, the metric node_cpu_frequency_hertz is exported just fine with Current CPU thread frequency in hertz as the help text. My Go knowledge is basic at best, but it seems a bit strange to me that cpufreq_common.go and cpu_linux.go seem to define a Desc with the same FQName, but with different help texts:
node_exporter/collector/cpu_linux.go
Lines 102 to 106 in b9b6820
| cpuFrequencyHz: prometheus.NewDesc( | |
| prometheus.BuildFQName(namespace, cpuCollectorSubsystem, "frequency_hertz"), | |
| "CPU frequency in hertz from /proc/cpuinfo.", | |
| []string{"package", "core", "cpu"}, nil, | |
| ), |
node_exporter/collector/cpufreq_common.go
Lines 23 to 27 in b9b6820
| cpuFreqHertzDesc = prometheus.NewDesc( | |
| prometheus.BuildFQName(namespace, cpuCollectorSubsystem, "frequency_hertz"), | |
| "Current CPU thread frequency in hertz.", | |
| []string{"cpu"}, nil, | |
| ) |
The error indeed disappears when the two help texts are the same, but I don't know if that only fixes the symptom or if a better fix would be to avoid the FQName duplication in the first place.
Details about test environment
- Tested with both the EPEL version of node-exporter for Rocky Linux 9 and a fresh build against the master branch:
node_exporter, version 1.11.1 (branch: tarball, revision: 1.el9)
build user:
build date: 20260407
go version: go1.25.8 (Red Hat 1.25.8-1.el9_7)
platform: linux/amd64
tags: rpm_crashtraceback,libtrust_openssl,netgo,osusergo,static_build
node_exporter, version (branch: , revision: b9b6820db7e125a4666280cedb2c3abcd45f9327-modified)
build user:
build date:
go version: go1.25.7
platform: linux/amd64
tags: unknown
- The error message appears on a dual-socket AMD EPYC 9654 node with 192 cores, but not on a dual-socket Intel Xeon Gold 6140 node with 36 cores.
- Full commandline of node-exporter (the port was changed to avoid interfering with the normal monitoring):
prometheus-node-exporter --web.listen-address=127.0.0.1:8100 --collector.cpu.info
Activity
SuperQ commentedon Jul 24, 2026
Sorry about introducing this bug. At the time of reviewing #3318 I didn't realize that we had a metric name colllison between
cpuandcpufreqcollectors. This is a bit of an odd situation because we don't usually re-use the "subsystem" names between different collectors.As @rtreffer said in #3756 there is some debate about which of the two metrics should take precedence.
So now we have two competing solutions.
My personal preference is to do the simple revert proposed in #3749. This restores the previous behavior which isn't optimal but is the least behavior changing.
As a followup I think we should do the "correctness" thing and fix the metric names coming from
cpufreqto benode_cpufreq_....But this is a breaking metric name change, which means we will need to put it behind a feature flag. Which is going to make the cpufreq code more complicated. But long-term this solves the collision issues.
jjegg01 commentedon Jul 24, 2026
Wouldn't the revert be a breaking change as well? If you run with default settings plus
--collector.cpu.infoenabled and cpufreq is not exporting the current frequency (e.g., in a cloud environment), the revert would makenode_cpu_frequency_hertznot be exported at all anymore.Coincidently, that situation is exactly what we have on our system: The older parts of our HPC cluster runs on Skylake CPUs, which are seemingly not fully supported by cpufreq (at least the
cpuinfo_cur_freqin sysfs is missing). And since we run node-exporter as/usr/bin/prometheus-node-exporter --collector.cpu.info, the metric indeed vanishes if I invert the logic in cpu_linux.go. While it would be kind of funny if my issue report about a bit of log spam ends up causing some actual breakage on our system, I think I prefer the current state to that 🙂Edit: I just realized that this is probably the real reason why we only see the log spam on the nodes with higher core counts. It just happens to be that all of those nodes are AMD and seem to be better supported by cpufreq.
SuperQ commentedon Jul 24, 2026
We just added #3712 that may be useful for hardware platforms that don't support
cur.rtreffer commentedon Aug 24, 2026
@SuperQ I gave this discussion a shot in #3795
@jjegg01 you would be able to set
--collector.cpufreq.enable-cpufreq-prefixto get both metrics exporter under different subsystems.SuperQ commentedon Aug 26, 2026
Thanks @rtreffer, that's exactly the fix I was thinking about. If there's no other objections soon I'll go with it.
jjegg01 commentedon Aug 27, 2026
@rtreffer Thanks for the work on this! Your fix does indeed fix the duplicate metric issue and thus the log spam.
However, as I feared, this does also make the
node_cpu_frequency_hertzmetric vanish on machines that do not have/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freqand--collector.cpu.infoset unless you enable that new flag. Before, this metric would fall back to the data in/proc/cpuinfo. So overall I would consider this a breaking change. I don't mind switching to the new naming scheme, but anyone unaware of this discussion might run into unexpected problems when updating. Perhaps it is preferable to just add the new flag and leave the bug as is (i.e. don't negate thecpuFreqEnabledconditional) until the new naming scheme becomes the default?SuperQ commentedon Aug 28, 2026
I much prefer reverting to the original behavior. Yes, it's not ideal, but I don't think we can really fix it without making yet another behavior change.