Configuring garbage collection for containers and images in OpenShift 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Kubelet

Issue

• How to configure the garbage collection for containers and images in OCP 4?
• How to automatically remove containers and unused images in OpenShift 4 nodes?
• How to apply garbage collection configuration changes to a Machine Pool using a KubeletConfig custom resource (CR) in OpenShift Container Platform?
• How does the Machine Config Operator (MCO) enforce KubeletConfig changes across nodes?
• How to review default/configured Kubelet configuration and parameters of garbage collection in OpenShift.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

The garbage collection for freeing up node resources is enabled by default in OpenShift 4. The default configuration is explained in default values for the garbage collection eviction thresholds on OpenShift. Apart of the default configuration, it is also possible to manually configure the garbage collection with any of the combinations below:

1. Soft eviction for containers.
2. Hard eviction for containers.
3. Eviction for unused images.

The explanation of the variables for the eviction is as follows:

memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available := node.stats.fs.available	
nodefs.inodesFree := node.stats.fs.inodesFree	
imagefs.available := node.stats.runtime.imagefs.available	
imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree

Refer to freeing node resources using garbage collection for additional information about configuring the garbage collection eviction thresholds. To apply garbage collection configuration changes to a Machine Pool, create a KubeletConfig custom resource (CR), which is applied by the Machine Config Operator in OpenShift Container Platform to enforce the changes across nodes, as explained in configuring garbage collection for containers and images . The following note from the documentation when configuring the evictionHard is important:

Note: For evictionHard you must specify all of these parameters. If you do not specify all parameters, only the specified parameters are applied and the garbage collection will not function properly.

For more details on configuring Kubelet parameters using KubeletConfig, refer to the official Red Hat documentation for the KubeletConfig. To review Kubelet configuration and eviction parameters, refer to the Kubernetes upstream documentation for the Content from kubernetes.io is not included.KubeletConfiguration .

Root Cause

The garbage collection for freeing up node resources is enabled by default in OpenShift 4, and the information for manually configuring it is already in the OpenShift documentation.

Diagnostic Steps

• Check default/configured values of garbage collection:

  $ oc get --raw /api/v1/nodes/<node_name>/proxy/configz | jq     # Replace <node_name> with the target node name. 
  

  For example, for checking the evictionHard for all the nodes in the cluster:

  $ for NODE in $(oc get nodes -o jsonpath='{.items[*].metadata.name}') ; do echo "=== Node: ${NODE} ===" ; oc get --raw /api/v1/nodes/${NODE}/proxy/configz | jq '.kubeletconfig.evictionHard' ; done
  

• Check if the garbage collection is configured in a KubeletConfig resource:

  $ oc get kubeletconfig
  [...]
  $ oc get kubeletconfig [kubeletconfig_name] -o yaml
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.