High CPU consumption when running oc adm must-gather

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.13.x

Issue

  • CPU usage on node hosting must-gather pod is reaching 100% and potentially causing it going not ready.
  • The must-gather command does not have a direct, simple flag to set resource limits.

Resolution

  • Use a dedicated must-gather namespace to run must-gather pods and apply resource limits to it.

  • Create must-gather dedicated namespace.

    oc create ns must-gather-ns

  • Add cluster-admin cluster role to the default service account on the dedicated namespace.

    oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:must-gather-ns:default

  • Create and apply a LimitRange to the namespace.

    oc create -n must-gather-ns -f - <<EOF
apiVersion: v1
kind: LimitRange
metadata:
  name: must-gather-limits
spec:
  limits:
  - type: Container
    default:
      cpu: 500m
      memory: 2Gi
    defaultRequest:
      cpu: 250m
      memory: 512Mi
EOF

  • Run oc adm must-gather using --run-namespace option.

    oc adm must-gather --run-namespace=must-gather-ns

NOTE

The memory usage of the must-gather process can vary depending on the plugins used, the number of nodes in the cluster, and other parameters. Ensure that your must-gather process is not terminated due to an Out-Of-Memory (OOM) condition.

SBR
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.