Scheduling resources

Red Hat OpenShift Logging 6.4

Scheduling log forwarding and LokiStack resources

Red Hat OpenShift Documentation Team

Abstract

This document explains how to schedule OpenShift Logging resources using node selectors, taints, and affinity and anti-affinity settings.

Chapter 1. Scheduling logging resources

You can schedule logging resources by defining node selectors, taints and tolerations, and affinity and anti-affinity configurations.

1.1. About pod scheduling controls

Learn how node selectors, taints and tolerations, and affinity rules control where pods run in the cluster.

Pod scheduling controls decide where the scheduler places pods in the cluster. You can use node selectors, taints and tolerations, and affinity rules to influence scheduling decisions.

Node selector
A node selector specifies a map of key-value pairs that are defined using custom labels on nodes and selectors specified in pods. For the pod to be eligible to run on a node, the pod must have the same key-value pair as the label on the node.
Taints and toleration
Taints and tolerations control which pods can be scheduled on nodes. A taint prevents pods from being scheduled unless the pod defines a matching toleration.
Affinity and anti-affinity
Affinity and anti-affinity rules constrain pod scheduling based on labels. Pod affinity attracts pods to nodes with specific labeled pods. Pod anti-affinity prevents pods from running with certain other pods.
Important

If you configure nodeSelector and nodeAffinity fields, the conditions of both fields must be met for the pod to be scheduled onto a candidate node.

1.2. Configuring resources and scheduling for logging collectors

Administrators can change the resources and scheduling of the collector by configuring the collector field in a ClusterLogForwarder custom resource (CR).

Prerequisites

  • You have administrator permissions.
  • You have installed Red Hat OpenShift Logging Operator.
  • You have created a ClusterLogForwarder CR.

Procedure

  1. Update the ClusterLogForwarder CR:

    The following example displays ClusterLogForwarder CR YAML: 

    apiVersion:  observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <name>
  namespace: <namespace>
spec:
  collector:
    nodeSelector:
      collector: needed
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 1Gi
    tolerations:
    - key: "logging"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 6000
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - preference:
            matchExpressions:
            - key: label-1
              operator: Exists
          weight: 1
      podAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: test
                operator: In
                values:
                 - value1
            topologyKey: kubernetes.io/hostname
          weight: 50
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: run
              operator: In
              values:
              - test
          namespaceSelector: {}
          topologyKey: kubernetes.io/hostname
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
           labelSelector:
             matchExpressions:
             - key: security
               operator: In
               values:
               - S2
           topologyKey: topology.kubernetes.io/zone
          weight: 100
# ...

  2. Apply the ClusterLogForwarder CR by running the following command: 

    $ oc apply -f <filename>.yaml

1.3. Viewing logging collector pods

You can view the logging collector pods and the corresponding nodes that they are running on.

Procedure

  • Run the following command in a project to view the logging collector pods and their details: 

    $ oc get pods --selector component=collector -o wide -n <project_name>

    You get an output similar to the following example: 

    NAME           READY  STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
collector-8d69v  1/1    Running   0          134m    10.130.2.30   master1.example.com   <none>           <none>
collector-bd225  1/1    Running   0          134m    10.131.1.11   master2.example.com   <none>           <none>
collector-cvrzs  1/1    Running   0          134m    10.130.0.21   master3.example.com   <none>           <none>
collector-gpqg2  1/1    Running   0          134m    10.128.2.27   worker1.example.com   <none>           <none>
collector-l9j7j  1/1    Running   0          134m    10.129.2.31   worker2.example.com   <none>           <none>

1.4. Loki pod placement

You can control which nodes the Loki pods run on, and prevent other workloads from using those nodes, by using tolerations or node selectors on the pods.

You can apply tolerations to the log store pods with the LokiStack custom resource (CR) and apply taints to a node with the node specification. A taint on a node is a key:value pair that instructs the node to repel all pods that do not allow the taint. Using a specific key:value pair that is not on other pods ensures that only the log store pods can run on that node.

The following example displays LokiStack with node selectors: 

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
# ...
  template:
    compactor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    distributor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    gateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    indexGateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    ingester:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    querier:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    queryFrontend:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    ruler:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
# ...

where:

  • compactor: Specifies the component pod type that applies to the node selector.
  • node-role.kubernetes.io/infra: Specifies the pods that are moved to nodes containing the defined label.

In the earlier example configuration, all Loki pods are moved to nodes containing the node-role.kubernetes.io/infra: "" label.

The following example displays LokiStack CR with node selectors and tolerations: 

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
# ...
  template:
    compactor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    distributor:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    indexGateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    ingester:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    querier:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    queryFrontend:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    ruler:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
    gateway:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        value: reserved
      - effect: NoExecute
        key: node-role.kubernetes.io/infra
        value: reserved
# ...

To configure the nodeSelector and tolerations fields of the LokiStack (CR), you can use the oc explain command to view the description and fields for a particular resource: 

$ oc explain lokistack.spec.template

You get an output similar to the following example: 

KIND:     LokiStack
VERSION:  loki.grafana.com/v1

RESOURCE: template <Object>

DESCRIPTION:
     Template defines the resource/limits/tolerations/nodeselectors per
     component

FIELDS:
   compactor	<Object>
     Compactor defines the compaction component spec.

   distributor	<Object>
     Distributor defines the distributor component spec.
...

For more detailed information, you can add a specific field: 

$ oc explain lokistack.spec.template.compactor

You get an output similar to the following example: 

KIND:     LokiStack
VERSION:  loki.grafana.com/v1

RESOURCE: compactor <Object>

DESCRIPTION:
     Compactor defines the compaction component spec.

FIELDS:
   nodeSelector	<map[string]string>
     NodeSelector defines the labels required by a node to schedule the
     component onto it.
...

1.5. Additional resources

Legal Notice

Copyright © Red Hat.
Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.
The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.
All other trademarks are the property of their respective owners.