Default values for the garbage collection eviction thresholds on OpenShift

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Kubelet
• Eviction

Issue

• How to know the default values for the garbage collection related parameters of the kubelet on OCP 4?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

IMPORTANT NOTE: the default setting values of eviction thresholds could be different by OpenShift version and Machine Config Operator version.

The default values of each parameter, described in Content from kubernetes.io is not included.Kubelet Configuration at Kubernetes upstream documentation, are as follows (refer to the documentation for other parameters):

• evictionSoft and evictionSoftGracePeriod are nil, which means nothing, by default.

• evictionPressureTransitionPeriod is 5m0s by default (only "ns", "us", "ms", "s", "m", "h" as the time suffix unit are allowed).

• evictionHard, detailed explanation in Content from kubernetes.io is not included.hard eviction thresholds, and for the source code refer to Content from github.com is not included.defaults_linux.go for OpenShift 4.20.:

  • memory.available < 100Mi (Linux nodes)
  • memory.available < 500Mi (Windows nodes)
  • nodefs.available < 10%
  • imagefs.available < 15%
  • nodefs.inodesFree < 5% (Linux nodes)
  • imagefs.inodesFree < 5% (Linux nodes)

Refer to configuring garbage collection for containers and images in OpenShift 4 and the documentation for freeing node resources using garbage collection for additional information, which includes the following important information:

Note: For evictionHard you must specify all of these parameters. If you do not specify all parameters, only the specified parameters are applied and the garbage collection will not function properly.

Root Cause

The default values of each parameter is described in Kubernetes upstream documentation.

Diagnostic Steps

• Check the default values within the kubelet service initialization journal logs. It Is needed to access to the node to check:

  $ oc get nodes
  [...]
  $ oc debug node/[node_name]
  [...]
  sh-4.4# chroot /host bash
  # journalctl -u kubelet.service
  ... FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
  ... FLAG: --eviction-pressure-transition-period="5m0s"
  ... FLAG: --eviction-soft=""
  ... FLAG: --eviction-soft-grace-period=""
  

• It is also possible to check the Kubelet config this way (change the ${node_name} accordingly):

  $ oc get --raw /api/v1/nodes/${node_name}/proxy/configz | jq
  

  For example, for checking the evictionHard for all the nodes in the cluster:

  $ for NODE in $(oc get nodes -o jsonpath='{.items[*].metadata.name}') ; do echo "=== Node: ${NODE} ===" ; oc get --raw /api/v1/nodes/${NODE}/proxy/configz | jq '.kubeletconfig.evictionHard' ; done
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.