Scheduling resources

Red Hat OpenShift Logging 6.4

Scheduling log forwarding and LokiStack resources

Red Hat OpenShift Documentation Team

Abstract

This document explains how to schedule OpenShift Logging resources using node selectors, taints, and affinity and anti-affinity settings.

Chapter 1. Scheduling logging resources

You can schedule logging resources by defining node selectors, taints and tolerations, and affinity and anti-affinity configurations.

1.1. About pod scheduling controls

Learn how node selectors, taints and tolerations, and affinity rules control where pods run in the cluster.

Pod scheduling controls decide where the scheduler places pods in the cluster. You can use node selectors, taints and tolerations, and affinity rules to influence scheduling decisions.

Node selector: A node selector specifies a map of key-value pairs that are defined using custom labels on nodes and selectors specified in pods. For the pod to be eligible to run on a node, the pod must have the same key-value pair as the label on the node.
Taints and toleration: Taints and tolerations control which pods can be scheduled on nodes. A taint prevents pods from being scheduled unless the pod defines a matching toleration.
Affinity and anti-affinity: Affinity and anti-affinity rules constrain pod scheduling based on labels. Pod affinity attracts pods to nodes with specific labeled pods. Pod anti-affinity prevents pods from running with certain other pods.

Important

If you configure nodeSelector and nodeAffinity fields, the conditions of both fields must be met for the pod to be scheduled onto a candidate node.

1.2. Scheduling use cases for logging components

Different deployment scenarios require different scheduling approaches. Use this guide to determine which scheduling mechanism to use for your logging infrastructure.

The following table describes common use cases and the scheduling mechanisms to apply:

Table 1.1. Scheduling mechanisms by use case

Use case	Node selectors	Taints and tolerations	Affinity rules	Resource limits
Schedule logging on infrastructure nodes	Required	Optional	Not required	Optional
Dedicate nodes exclusively to logging	Required	Required	Not required	Optional
Distribute logging across availability zones	Not required	Not required	Required	Not required
Tune logging performance and resource usage	Not required	Not required	Not required	Required

1.2.1. Infrastructure nodes

When you have dedicated infrastructure nodes labeled with node-role.kubernetes.io/infra, use node selectors to schedule logging components on those nodes. This separates logging workloads from application workloads, which optimizes costs and maintains clear operational boundaries.

To prevent non-logging workloads from using infrastructure nodes, apply taints to the infrastructure nodes and configure tolerations on logging pods. This ensures that infrastructure node resources are reserved exclusively for logging.

1.2.2. High availability across zones

In multi-zone clusters, use pod anti-affinity rules to distribute LokiStack components across availability zones. This maintains logging availability during zone failures and meets business continuity requirements.

For example, configure anti-affinity to prevent multiple ingester pods from running in the same zone. If one zone fails, the remaining zones continue to process logs.

1.2.3. Performance tuning

When you experience high log volume or performance issues, adjust CPU and memory resource limits for collector pods. Increasing resource limits allows collectors to handle higher throughput, while setting appropriate limits prevents logging from consuming excessive node resources.

Monitor collector resource usage and adjust limits based on actual consumption and node capacity.

1.2.4. Verification

After configuring scheduling rules, verify that pods are running on the expected nodes:

For collectors, use the oc get pods command with the --selector and -o wide flags to view pod placement.
For LokiStack components, check the pod status and node assignment for each component type.

If pods are not scheduled as expected, check node labels, taints, and pod tolerations. Verify that the scheduling configuration matches your cluster’s node configuration.

1.3. Configuring resources and scheduling for logging collectors

Administrators can change the resources and scheduling of the collector by configuring the collector field in a ClusterLogForwarder custom resource (CR).

Prerequisites

You have administrator permissions.
You have installed Red Hat OpenShift Logging Operator.
You have created a ClusterLogForwarder CR.

Procedure

Update the ClusterLogForwarder CR to configure scheduling and resources.

The following example schedules collectors on infrastructure nodes:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collector:
    nodeSelector:
      node-role.kubernetes.io/infra: ""
# ...

The following example schedules collectors on dedicated infrastructure nodes with taints:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collector:
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    tolerations:
    - key: node-role.kubernetes.io/infra
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/infra
      operator: Exists
      effect: NoExecute
# ...

The following example shows all available scheduling and resource fields:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collector:
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 1Gi
    tolerations:
    - key: node-role.kubernetes.io/infra
      operator: Exists
      effect: NoSchedule
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - preference:
            matchExpressions:
            - key: node-role.kubernetes.io/infra
              operator: Exists
          weight: 1
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                - collector
            topologyKey: topology.kubernetes.io/zone
          weight: 100
# ...

Apply the ClusterLogForwarder CR by running the following command:
```
$ oc apply -f <filename>.yaml
```

1.4. Viewing logging collector pods

You can view the logging collector pods and the corresponding nodes that they are running on.

Procedure

Run the following command in a project to view the logging collector pods and their details:

$ oc get pods --selector component=collector -o wide -n <project_name>

You get an output similar to the following example:

NAME           READY  STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
collector-8d69v  1/1    Running   0          134m    10.130.2.30   master1.example.com   <none>           <none>
collector-bd225  1/1    Running   0          134m    10.131.1.11   master2.example.com   <none>           <none>
collector-cvrzs  1/1    Running   0          134m    10.130.0.21   master3.example.com   <none>           <none>
collector-gpqg2  1/1    Running   0          134m    10.128.2.27   worker1.example.com   <none>           <none>
collector-l9j7j  1/1    Running   0          134m    10.129.2.31   worker2.example.com   <none>           <none>

1.5. Loki pod placement

You can control which nodes the Loki pods run on, and prevent other workloads from using those nodes, by using tolerations or node selectors on the pods.

You can apply tolerations to the log store pods with the LokiStack custom resource (CR) and apply taints to a node with the node specification. A taint on a node is a key:value pair that instructs the node to repel all pods that do not allow the taint. Using a specific key:value pair that is not on other pods ensures that only the log store pods can run on that node.

The following example displays LokiStack with node selectors:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
# ...
  template:
    compactor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    distributor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    gateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    indexGateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    ingester:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    querier:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    queryFrontend:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    ruler:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
# ...

where:

compactor: Specifies the component pod type that applies to the node selector.
node-role.kubernetes.io/infra: Specifies the pods that are moved to nodes containing the defined label.

In the earlier example configuration, all Loki pods are moved to nodes containing the node-role.kubernetes.io/infra: "" label.

Note

Apply the same nodeSelector and tolerations configuration to all LokiStack components: compactor, distributor, gateway, indexGateway, ingester, querier, queryFrontend, and ruler.

For multi-zone high availability deployments, LokiStack also provides the Zone field under ReplicationSpec to configure topology spread constraints across availability zones. This ensures that Loki components are distributed across zones for fault tolerance.

The following example displays LokiStack CR with node selectors and tolerations for dedicated infrastructure nodes with taints. The configuration pattern is shown for three components and applies to all components:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
# ...
  template:
    compactor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    distributor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    gateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    # ... repeat for indexGateway, ingester, querier, queryFrontend, ruler
# ...

To configure the nodeSelector and tolerations fields of the LokiStack (CR), you can use the oc explain command to view the description and fields for a particular resource:

$ oc explain lokistack.spec.template

You get an output similar to the following example:

KIND:     LokiStack
VERSION:  loki.grafana.com/v1

RESOURCE: template <Object>

DESCRIPTION:
     Template defines the resource/limits/tolerations/nodeselectors per
     component

FIELDS:
   compactor	<Object>
     Compactor defines the compaction component spec.

   distributor	<Object>
     Distributor defines the distributor component spec.
...

For more detailed information, you can add a specific field:

$ oc explain lokistack.spec.template.compactor

You get an output similar to the following example:

KIND:     LokiStack
VERSION:  loki.grafana.com/v1

RESOURCE: compactor <Object>

DESCRIPTION:
     Compactor defines the compaction component spec.

FIELDS:
   nodeSelector	<map[string]string>
     NodeSelector defines the labels required by a node to schedule the
     component onto it.
...

1.6. Troubleshooting logging pod scheduling

If logging pods are not scheduled on the expected nodes or remain in a pending state, verify the node labels, taints, and pod scheduling configuration.

Prerequisites

You have administrator permissions.
You have installed the Red Hat OpenShift Logging Operator or Loki Operator.

Procedure

Check the pod status to identify scheduling issues:
```
$ oc get pods -n openshift-logging -o wide
```
Pods that cannot be scheduled display a Pending status.
Describe the pod to view scheduling events:
```
$ oc describe pod <pod-name> -n openshift-logging
```
Review the Events section for messages such as:
- 0/X nodes are available: X node(s) didn’t match Pod’s node affinity/selector - Indicates missing or incorrect node labels. Resolved in step 3.
- 0/X nodes are available: X node(s) had untolerated taint - Indicates missing tolerations in the pod specification. Resolved in step 4.
- 0/X nodes are available: Insufficient cpu, Insufficient memory - Indicates resource constraints on available nodes. Requires increasing node resources or reducing pod resource requests.
Verify that target nodes have the required labels:
```
$ oc get nodes --show-labels
```
Confirm that nodes intended for logging have the labels specified in the nodeSelector configuration.
If using taints and tolerations, verify node taints:
```
$ oc describe node <node-name>
```
Review the Taints section and confirm that logging pods have matching tolerations configured.
Verify the pod’s scheduling configuration:
For collector pods, check the ClusterLogForwarder custom resource:
```
$ oc get clusterlogforwarder <name> -n <namespace> -o yaml
```
For LokiStack pods, check the LokiStack custom resource:
```
$ oc get lokistack logging-loki -n openshift-logging -o yaml
```
Correct any mismatches between the pod configuration and node labels or taints:
- If node labels are missing, add them:
```
$ oc label node <node-name> <key>=<value>
```
- If the pod’s nodeSelector has a typing error, update the custom resource with the correct label.
- If a taint is missing from the pod’s tolerations, add it to the custom resource.
After making corrections, verify that the pods are scheduled:
```
$ oc get pods -n openshift-logging -o wide
```
Pods should move to Running status on the expected nodes.

Verification

Confirm that logging pods are running on the intended nodes by checking the NODE column in the pod list.

1.7. Additional resources

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.