OpenShift Data Foundation must-gather is huge (50+G)
Environment
- Red Hat OpenShift Data Foundation
- 4.20
Issue
-
The OpenShift Data Foundation must-gather is larger than 50 gibibyte
$ du -h --max-depth=0 odf-must-gather 73G odf-must-gather -
Using the
--since=or--since-time=flag for the OpenShift Data Foundation must-gather does not decrease the size to a reasonable amount -
The
ceph-client.ceph-exporter.loglog file is larger than 1 gigabyte$ tree -sh ./odf-must-gather/ | grep -E '[0-9]{3}M]|[0-9]G]' │ │ │ ├── [4.6G] ceph-client.ceph-exporter.log │ │ │ ├── [237M] ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.log │ │ │ ├── [674M] ceph-volume.log │ │ │ ├── [4.2G] ceph-client.ceph-exporter.log │ │ │ ├── [555M] ceph-volume.log │ │ │ ├── [4.5G] ceph-client.ceph-exporter.log │ │ │ ├── [510M] ceph-volume.log │ │ │ ├── [4.2G] ceph-client.ceph-exporter.log │ │ │ ├── [420M] ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.log │ │ │ ├── [788M] ceph-volume.log │ │ │ ├── [4.1G] ceph-client.ceph-exporter.log │ │ │ ├── [679M] ceph-volume.log │ │ │ ├── [4.4G] ceph-client.ceph-exporter.log │ │ │ ├── [297M] ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.log │ │ │ ├── [830M] ceph-volume.log │ │ │ ├── [4.3G] ceph-client.ceph-exporter.log │ │ │ ├── [699M] ceph-volume.log │ │ │ ├── [3.6G] ceph-client.ceph-exporter.log │ │ │ ├── [165M] ceph-osd.35.log │ │ │ ├── [197M] ceph-osd.5.log │ │ │ ├── [166M] ceph-volume.log │ │ │ ├── [4.4G] ceph-client.ceph-exporter.log │ │ │ ├── [904M] ceph-volume.log │ │ │ ├── [4.2G] ceph-client.ceph-exporter.log │ │ │ ├── [477M] ceph-volume.log │ │ │ ├── [4.6G] ceph-client.ceph-exporter.log │ │ │ ├── [258M] ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.log │ │ │ ├── [475M] ceph-volume.log │ │ │ ├── [4.5G] ceph-client.ceph-exporter.log │ │ │ ├── [517M] ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.log │ │ │ ├── [680M] ceph-volume.log │ │ │ ├── [4.3G] ceph-client.ceph-exporter.log │ │ │ ├── [828M] ceph-volume.log
Resolution
Solution
- This issue has been reported to Red Hat engineering. It is being tracked in Bug This content is not included.DFBUGS-5523. For more information, please open a This content is not included.new support case on the Red Hat Customer Portal referring to this solution.
Workaround
-
A workaround can be applied by identifying, deleting the
/var/log/ceph/ceph-client.ceph-exporter.logand restarting therook-ceph-exporterpods which can then be followed up with capturing amust-gatheras per the documentation.$ oc -n openshift-storage get pods -l=app=rook-ceph-exporter -o custom-columns=:metadata.name --no-headers | while read -r pod; do echo "Will remove the following file from $pod"; oc -n openshift-storage exec $pod -c ceph-exporter -- ls -lah /var/log/ceph/ceph-client.ceph-exporter.log; done Will remove the following file from rook-ceph-exporter-worker-1-gqqrx-c87cb46b4dgkv7 -rw-r--r--. 1 ceph ceph 34M Feb 10 09:52 /var/log/ceph/ceph-client.ceph-exporter.log Will remove the following file from rook-ceph-exporter-worker-1-s58vm-7f479d9b9mpkdb -rw-r--r--. 1 ceph ceph 34M Feb 10 09:52 /var/log/ceph/ceph-client.ceph-exporter.log Will remove the following file from rook-ceph-exporter-worker-1-tvtcs-6fbb8b5f7mc4b2 -rw-r--r--. 1 ceph ceph 32M Feb 10 09:52 /var/log/ceph/ceph-client.ceph-exporter.log # The following command will delete the log file and remove the pods $ oc -n openshift-storage get pods -l=app=rook-ceph-exporter -o custom-columns=:metadata.name --no-headers | while read -r pod; do echo "Removing ceph-exporter logs from $pod and deleting pod afterwards"; oc -n openshift-storage exec $pod -c ceph-exporter -- rm /var/log/ceph/ceph-client.ceph-exporter.log; oc -n openshift-storage delete pod $pod; done Removing ceph-exporter logs from rook-ceph-exporter-worker-1-gqqrx-c87cb46b44dspw and deleting pod afterwards pod "rook-ceph-exporter-worker-1-gqqrx-c87cb46b44dspw" deleted from openshift-storage namespace Removing ceph-exporter logs from rook-ceph-exporter-worker-1-s58vm-7f479d9b97h48n and deleting pod afterwards pod "rook-ceph-exporter-worker-1-s58vm-7f479d9b97h48n" deleted from openshift-storage namespace Removing ceph-exporter logs from rook-ceph-exporter-worker-1-tvtcs-6fbb8b5f7h2lbp and deleting pod afterwards pod "rook-ceph-exporter-worker-1-tvtcs-6fbb8b5f7h2lbp" deleted from openshift-storage namespace $ oc -n openshift-storage get pods -l=app=rook-ceph-exporter -o custom-columns=:metadata.name --no-headers | while read -r pod; do echo "Checking for ceph-exporter logs on $pod"; oc -n openshift-storage exec $pod -c ceph-exporter -- ls -lah /var/log/ceph/ceph-client.ceph-exporter.log; done Checking for ceph-exporter logs on rook-ceph-exporter-worker-1-gqqrx-c87cb46b44dspw -rw-r--r--. 1 root root 8.3K Feb 10 10:04 /var/log/ceph/ceph-client.ceph-exporter.log Checking for ceph-exporter logs on rook-ceph-exporter-worker-1-s58vm-7f479d9b97h48n -rw-r--r--. 1 root root 8.3K Feb 10 10:04 /var/log/ceph/ceph-client.ceph-exporter.log Checking for ceph-exporter logs on rook-ceph-exporter-worker-1-tvtcs-6fbb8b5f7h2lbp -rw-r--r--. 1 root root 9.2K Feb 10 10:04 /var/log/ceph/ceph-client.ceph-exporter.log
Root Cause
-
The ceph exporter daemon runs on each node, for the purpose of collecting prometheus metrics that in the past had all been collected by the ceph mgr.
-
The ceph-exporter logs, which are part of
oc -n openshift-storage get pods -l=app=rook-ceph-exporterpods, are not rotated early enough and thus can reach gigabytes of data. -
The ceph-exporter logs will contain duplicated content in terms of
pid path is empty; process metrics won't be fetched for: ...logs. This can be verified by counting duplicated lines in the pods with the following example:$ oc -n openshift-storage exec -it rook-ceph-exporter-worker-0.<remaining-podname> -c ceph-exporter -- awk '{ $1=""; ; $2=""; $3=""; print $0 }' /var/log/ceph/ceph-client.ceph-exporter.log | sed 's/^ //' | sort | uniq -c | sort -nr 89448 pid path is empty; process metrics won't be fetched for: ceph-mon.c 89430 pid path is empty; process metrics won't be fetched for: ceph-osd.2 78934 pid path is empty; process metrics won't be fetched for: ceph-mds.ocs-storagecluster-cephfilesystem-a 15984 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.3.xxxx 3 ceph version 19.2.1-292.el9cp (ba02d589f9356be88303b8e8ec2790f12300f3b5) squid (stable), process ceph-exporter, pid 1 3 HTTP server running on 0.0.0.0:9926 2 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2 Collector shutdown initiated, timer canceled 2 Ceph exporter web server stopped 2 Ceph exporter stopped 2 *** Got signal Terminated *** 2 pid path is empty; process metrics won't be fetched for: ceph-mds.ocs-storagecluster-cephfilesystem-a pid path is empty; process metrics won't be fetched for: ceph-osd.2on.c pid path is empty; process metrics won't be fetched for: ceph-mon.cds.ocs-storagecluster-cephfilesystem-a pid path is empty; process metrics won't be fetched for: ceph-mon.cds.ocs-storagecluster-cephfilesystem-a 1 pid path is empty; process metrics won't be fetched for: cceph-mds.ocs-storagecluster-cephfilesystem-a
Diagnostic Steps
Option 1: Creating a must-gather
-
Create a OpenShift Data Foundation must-gather based on the provided official documentation
-
Confirm that the must-gather has reached an unreasonable size
du -h ./odf-must-gather/ --max-depth=0 71G ./odf-must-gather/
Option 2: Confirm large ceph-client.ceph-exporter.log files sizes before capturing a must-gather
- Identify the
rook-ceph-exporterpods:
$ oc -n openshift-storage get pods -l=app=rook-ceph-exporter
-
Confirm that the
ceph-client.ceph-exporter.logis several gibibytes large:$ oc -n openshift-storage exec -it rook-ceph-exporter-worker-0.<remaining-podname> -c ceph-exporter -- ls -lah /var/log/ceph | grep -E '[0-9]{3}M|[0-9]G' total 5.6G -rw-r--r--. 1 ceph ceph 4.2G Feb 5 16:47 ceph-client.ceph-exporter.log -rw-r--r--. 1 ceph ceph 420M Jun 12 2025 ceph-client.rgw.ocs.storagecluster.cephobjectstore.a.log -rw-r--r--. 1 ceph ceph 788M Feb 5 08:25 ceph-volume.log
Notes for completeness:
-
The GNU
ls -handdu -hcommand does, per default, output units as power of2Units are K,M,G,T,P,E,Z,Y,R,Q (powers of 1024) or KB,MB,... (powers of 1000)) -
The GNU
tree -hcommand does, per default, output as powers of10Print the size of each file but in a more human readable way, e.g. appending a size letter for kilobytes (K), megabytes (M), gigabytes (G), terrabytes (T), petabytes (P) and exabytes (E).
SBR
Product(s)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.