How to configure log level verbosity for Kubelet in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat Openshift Container Platform (RHOCP)
    • 4
  • Kubelet

Issue

  • In order to troubleshoot some issues with the nodes, it is recommended to establish verbosity of log level for kubelet depending on the issue to be tracked.
  • It is needed to understand log level verbosity of Kubelet.

Resolution

Starting with OpenShift 4.12, it is possible to modify the kubelet log level for individual nodes without requiring reboot as explained in modifying the kubelet as a one-time scenario by creating a config file in /etc/systemd/system/kubelet.service.d/.

For persistent kubelet log level configuration in all nodes, refer to persistent kubelet log level configuration.

Note: increasing the logLevel uses more IO, CPU, and resources on all the machines within the pool, so it is recommended to back to the default configuration after the required logs are collected.

Kubelet log level values

As a general use case, it is recommended to use 0-4 as debug-level logs, and 5-8 as trace-level logs. The default kubelet log level is set as 2.

Refer to kubelet log verbosity descriptions for detailed information.

Root Cause

Starting with OpenShift 4.7, it is possible to configure the log level for kubelet via the KubeletConfig resource. Starting with OpenShift 4.12, it is possible to modify the log level for kubelet for a single node via config file in /etc/systemd/system/kubelet.service.d/.

Diagnostic Steps

  • Information about the spec.logLevel for the KubeletConfig resource:

    $ oc explain kubeletconfig.spec.logLevel
KIND:     KubeletConfig
VERSION:  machineconfiguration.openshift.io/v1

FIELD:    logLevel <integer>

DESCRIPTION:
     logLevel defines the log level of the Kubelet

  • Once the log level is configured properly, kubelet logs can be gathered as per below examples:

        ##For generic roles:
    $ oc adm node-logs --role master -u kubelet
    $ oc adm node-logs --role worker -u kubelet

    ##For specific nodes:
    $ oc adm node-logs <node-name> -u kubelet

    ##Or inside a node:
    $ journalctl -b -f -u kubelet.service

  • It is also possible to gather the kubelet logs of all nodes directly:

    $ for n in $(oc get node --no-headers | awk '{print $1}'); do oc adm node-logs $n | gzip > $n.log.gz; done

  • Additional information for collecting kubelet logs in gathering kubelet logs.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.