What are the default system-reserved and eviction thresholds in OpenShift?

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Kubelet

Issue

• What is the default setting of system reserved resources in OpenShift 4?
• What is the default setting for the eviction thresholds in OpenShift 4?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

IMPORTANT NOTE: the default setting values of system reserved resources and eviction thresholds could be different by OpenShift version and Machine Config Operator version.

Default system reserved resources configuration

Reserved resource setting includes --kube-reserved and --system-reserved in OpenShift. The default values in OpenShift 4 are:

• For the kube-reserved: None.

• For the system-reserved: default parameters are specified and can be confirmed on Content from github.com is not included.machine-config-operator templates for 4.20 (and in a running cluster, following the steps in the Diagnostic Steps section). At the time of writing this, the default values for OpenShift 4.20 are:

  memory: 1Gi
  cpu: 500m
  ephemeral-storage: 1Gi
  

Refer to which amount of CPU and memory are recommended to reserve for the system in OpenShift 4 nodes? for recommendations.

Default eviction thresholds configuration

Eviction resource limit setting include --eviction-hard and --eviction-soft in OpenShift. Refer to the default values for the garbage collection eviction thresholds on OpenShift

Root Cause

The default reservation is expected to be sufficient for low resource configurations, but it could be not enough in some cases, specially when the resource count of the nodes is high. For additional recommendations, refer to which amount of CPU and memory are recommended to reserve for the system in OpenShift 4 nodes?

The system-reserved is only used for the processes running from /system.slice/, and in OpenShift 4 they are only node critical processes such as CRI-O, Kubelet, etc. It is possible to see the entire list of processes included in the reservation as described in the Diagnostic Steps section.

Diagnostic Steps

• Check the /etc/kubernetes/kubelet.conf file in the nodes:

  $ oc get nodes
  [...]
  $ oc debug node/[node-name]
  [...]
  # chroot /host bash
  # cat /etc/kubernetes/kubelet.conf
  

• Look through the output and find the systemReserved values:

  [...]
  systemReserved:                                                                                                                            
    cpu: 500m                                                                                                          
    memory: 1Gi
    ephemeral-storage: 1Gi
  [...]
  

• It is also possible to check the Kubelet config this way (change the ${node_name} accordingly):

  $ oc get --raw /api/v1/nodes/${node_name}/proxy/configz | jq
  

  For example, for checking the systemReserved for all the nodes in the cluster:

  $ for NODE in $(oc get nodes -o jsonpath='{.items[*].metadata.name}') ; do echo "=== Node: ${NODE} ===" ; oc get --raw /api/v1/nodes/${NODE}/proxy/configz | jq '.kubeletconfig.systemReserved' ; done
  

• Check the processes included in the reservation:

  $ oc debug node/[node_name]
  [...]
  
  sh-4.4# chroot /host bash
  # systemd-cgls /system.slice/
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.