The size of the must-gather generated by RHOCP4 is very large when MetalLB is installed

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • MetalLB operator

Issue

  • Why is the size of the must-gather generated by RHOCP4 very large?
  • Why are the frr logs from MetalLB collected by the must-gather very large?

Resolution

  • To exclude the collection of full frr logs from the generated must-gather the following command can be used:

    oc adm must-gather -- "sed -i 's#\"frr.log\"##g' /usr/bin/gather_metallb; /usr/bin/gather"

  • In case the size of the must-gather is still too large it is possible to also exclude the collection of rotated pod logs, this would be the resulting command:

    oc adm must-gather -- "sed -i 's#--rotated-pod-logs##g' /usr/bin/gather; sed -i 's#\"frr.log\"##g' /usr/bin/gather_metallb; /usr/bin/gather"

Root Cause

The must-gather image collects both a trimmed down version of the frr logs and a full one that can be huge in size, for an initial troubleshooting the trimmed down version should be enough since it only contains the latest lines of the full log. In case it is needed it is still possible to get the full logs from the cluster.

Diagnostic Steps

  • Verify the size of the generated must-gather, it should not exceed 30GB for a 10-node cluster.

    $ du -sh must-gather.local.0123456789012345678
51G     must-gather.local.0123456789012345678

  • Verify the size of the collected frr logs, if it is over 200MB (each) it will have a significant impact on the size of the must-gather, especially if the cluster has a large amount of nodes:

    $ find must-gather.local.0123456789012345678/ -name '*frr.log' -exec ls -alh {} \;
-rw-rw-rw-+ 1 yank yank 4.8G Jan 1 10:09 must-gather.local.0123456789012345678/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-2eddfe6b31b4275d681e16a21c8d0db983f7fe6a86f6cf309b8c1a06c03ba63d/namespaces/metallb-system/pods/speaker-6vspl/frr/frr/logs/frr.log
-rw-rw-rw-+ 1 yank yank 3.4G Jan 1 10:09 must-gather.local.0123456789012345678/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-2eddfe6b31b4275d681e16a21c8d0db983f7fe6a86f6cf309b8c1a06c03ba63d/namespaces/metallb-system/pods/speaker-gsrsw/frr/frr/logs/frr.log
-rw-rw-rw-+ 1 yank yank 4.4G Jan 1 10:10 must-gather.local.0123456789012345678/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-2eddfe6b31b4275d681e16a21c8d0db983f7fe6a86f6cf309b8c1a06c03ba63d/namespaces/metallb-system/pods/speaker-kpb9k/frr/frr/logs/frr.log
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.