Skip to content

node_filesystem_readonly reports writable filesystem as read-only after upgrade to 1.12.0/1.12.1 #3755

Open

Description

@Merlijnv

node_filesystem_readonly reports writable filesystem as read-only after upgrade to 1.12.0/1.12.1

Bug report

After upgrading node_exporter from 1.11.x to 1.12.0 or 1.12.1, I am seeing multiple false positives for node_filesystem_readonly.

The most concerning example is /var/data, which is clearly mounted read-write on the host and accepts write operations, yet node_exporter reports it as read-only.

Environment

Docker Compose

node-exporter:
  image: prom/node-exporter:${NODE_EXPORTER_VERSION}
  user: "${NODE_EXPORTER_UID}:${NODE_EXPORTER_GID}"
  container_name: node-exporter
  restart: unless-stopped
  network_mode: host
  command:
    - --path.rootfs=/host
    - --web.listen-address=:9100
    - --collector.textfile.directory=/textfile_collector
  volumes:
    - /:/host:ro,rslave
    - ${NODE_EXPORTER_TEXTFILE_DIR:?NODE_EXPORTER_TEXTFILE_DIR must be set in .env}:/textfile_collector:ro

Observed metrics

Among others, I am seeing the following metric:

node_filesystem_readonly{
  device="/dev/mapper/data--vg0-data--lv--0",
  fstype="ext4",
  mountpoint="/var/data"
} 1

There are also several entries with device_error="permission denied" and device_error="no such file or directory".

Additional verification

The filesystem is mounted as read-write on the host.

findmnt

findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /var/data

Output:

/dev/mapper/data--vg0-data--lv--0 /var/data ext4 rw,relatime

mount

mount | grep ' /var/data '

Output:

/dev/mapper/data--vg0-data--lv--0 on /var/data type ext4 (rw,relatime)

The mount options clearly show rw.

Write test

I can successfully write to the filesystem:

touch /var/data/node_exporter_rw_test
echo test > /var/data/node_exporter_rw_test
cat /var/data/node_exporter_rw_test
rm -f /var/data/node_exporter_rw_test

Output:

test

All operations complete successfully.

Expected behavior

node_filesystem_readonly{mountpoint="/var/data"} 0

Actual behavior

node_filesystem_readonly{mountpoint="/var/data"} 1

Version comparison

Version Result
1.11.x Works as expected
1.12.0 Reports /var/data as read-only
1.12.1 Reports /var/data as read-only

Related issue

I found issue #2506:

#2506

However, that issue appears to be related to the host root filesystem being bind-mounted read-only into the container, causing node_exporter to observe the filesystem as read-only from inside the container. 1

My situation appears different:

  • /var/data is mounted as rw on the host.
  • Write operations succeed.
  • The issue only appears after upgrading from 1.11.x to 1.12.x.
  • Multiple filesystems are now being reported as read-only or returning device errors.

Has anything changed in the filesystem collector between 1.11.x and 1.12.x that could affect how read-only status is determined for bind-mounted root filesystems in containers?

I am happy to provide additional debugging information if needed.

Activity

  1. Merlijnv commented on Aug 28, 2026

    @Merlijnv
    Author

    @locker95 what is the status on this issue?

  2. Merlijnv commented on Sep 15, 2026

    @Merlijnv
    Author

    @discordianfish @SuperQ I am waiting for the pullrequest to be merged is there an estimate when this wil be fixed?

  3. Merlijnv commented on Oct 1, 2026

    @Merlijnv
    Author

    @discordianfish @SuperQ Still waiting for this pr to be merged. Can't upgrade to newest version until it is fixed

  4. muhammad-tariq-zaman commented on Nov 16, 2026

    @muhammad-tariq-zaman

    Thanks for the detailed report — this matches the false positive described in #3758. With the setup in your compose file (- /:/host:ro,rslave together with --path.rootfs=/host), every host mount carries a per-mount ro flag inside the exporter's own mount namespace, even though the filesystem is writable on the host.

    One way to confirm this is the same false positive, rather than a genuinely read-only filesystem, is to look at the mount entry from inside the container in /proc/self/mountinfo: for the affected mount, the per-mount options (field 6) will show ro, while the super options (the last field) should show rw if the host filesystem is actually writable. A truly read-only filesystem would show ro in the super options instead.

    Until #3758 is reviewed and merged, the workaround documented there is to run the container with --pid=host, so the exporter reads the host's view from /proc/1/mountinfo instead of its own namespace.

  5. Merlijnv commented on Nov 16, 2026

    @Merlijnv
    Author

    Oh thanks for the workaround will try it out tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions