About OpenShift logging

Red Hat OpenShift Logging 6.6

Introduction to OpenShift logging.

Red Hat OpenShift Documentation Team

Abstract

This document provides an overview of OpenShift Logging features, and also includes release notes and support information.

Chapter 1. Red Hat OpenShift Logging overview

The ClusterLogForwarder custom resource (CR) is the central configuration point for log collection and forwarding.

1.1. Inputs and outputs

Inputs specify the sources of logs to forward. Outputs define the destinations where logs are sent.

Logging provides the following built-in input types to select logs from different parts of your cluster:

  • application
  • receiver
  • infrastructure
  • audit

You can also define custom inputs based on namespaces or pod labels to fine-tune log selection.

Each output type has its own set of configuration options, allowing you to customize the behavior and authentication settings.

1.2. Receiver input type

The receiver input type enables the Logging system to accept logs from external sources. It supports two formats for receiving logs: http and syslog.

The ReceiverSpec field defines the configuration for a receiver input.

1.3. Pipelines and filters

Pipelines determine the flow of logs from inputs to outputs. A pipeline consists of one or more input refs, output refs, and optional filter refs. You can use filters to transform or drop log messages within a pipeline. The order of filters matters because they are applied sequentially, and earlier filters can prevent log messages from reaching later stages.

1.4. Operator behavior

The Cluster Logging Operator manages the deployment and configuration of the collector based on the managementState field of the ClusterLogForwarder resource:

  • When set to Managed (default), the Operator actively manages the logging resources to match the configuration defined in the spec.
  • When set to Unmanaged, the Operator does not take any action, allowing you to manually manage the logging components.

1.5. Validation

Logging automatically validates configurations and applies sensible defaults to prevent errors.

The ClusterLogForwarder resource enforces validation checks on required fields, dependencies between fields, and the format of input values. Default values are provided for certain fields, which reduces the need for explicit configuration in common scenarios.

Chapter 2. Cluster logging support

Ensure cluster stability and full support by using only official logging configurations.

Configuration paradigms might change across OpenShift Container Platform releases, and such cases can only be handled gracefully if all configuration possibilities are controlled. If you use configurations other than those described in this documentation, your changes will be overwritten, because Operators are designed to reconcile any differences.

Note

If you must perform configurations not described in the OpenShift Container Platform documentation, you must set your Red Hat OpenShift Logging Operator to Unmanaged. An unmanaged logging instance is not supported and does not receive updates until you return its status to Managed.

Note

Logging is provided as an installable component, with a distinct release cycle from the core OpenShift Container Platform. The Red Hat Red Hat OpenShift Logging Life Cycle Policy outlines release compatibility.

Loki is a horizontally scalable, highly available, multitenant log aggregation system offered as a GA log store for logging for Red Hat OpenShift that can be visualized with the OpenShift Observability UI. The Loki configuration provided by OpenShift Logging is a short-term log store designed to help users perform fast troubleshooting with the collected logs. For that purpose, the logging for Red Hat OpenShift configuration of Loki has short-term storage, and is optimized for very recent queries. For long-term storage or queries over a long time period, users should look to log stores external to their cluster.

Elasticsearch indexes incoming log records completely during ingestion. Loki indexes only a few fixed labels during ingestion and defers more complex parsing until after the logs have been stored. This means Loki can collect logs more quickly.

2.1. Logging capabilities and limitations

Logging for Red Hat OpenShift is an opinionated collector and normalizer of application, infrastructure, and audit logs. It is intended to be used for forwarding logs to various supported systems.

Logging is not:

  • Security Information and Event Monitoring (SIEM) compliant
  • A "bring your own" (BYO) log collector configuration
  • Historical or long term log retention or storage
  • A guaranteed log sink
  • Secure storage - audit logs are not stored by default

2.2. Supported API custom resource definitions

The following table describes the supported Logging APIs.

Table 2.1. Logging API support states

CustomResourceDefinition (CRD)ApiVersionSupport state

LokiStack

lokistack.loki.grafana.com/v1

Supported from 5.5

RulerConfig

rulerconfig.loki.grafana/v1

Supported from 5.7

AlertingRule

alertingrule.loki.grafana/v1

Supported from 5.7

RecordingRule

recordingrule.loki.grafana/v1

Supported from 5.7

LogFileMetricExporter

LogFileMetricExporter.logging.openshift.io/v1alpha1

Supported from 5.8

ClusterLogForwarder

clusterlogforwarder.observability.openshift.io/v1

Supported from 6.0

2.3. Unsupported configurations

You must set the Red Hat OpenShift Logging Operator to the Unmanaged state to modify the following components:

  • The collector configuration file
  • The collector daemonset

Explicitly unsupported cases include:

  • Configuring the logging collector using environment variables. You cannot use environment variables to modify the log collector.
  • Configuring how the log collector normalizes logs. You cannot modify default log normalization.

2.4. Support policy for unmanaged Operators

The management state of an Operator determines whether an Operator is actively managing the resources for its related component in the cluster as designed. If an Operator is set to an unmanaged state, it does not respond to changes in configuration nor does it receive updates.

Although useful for non-production or debugging, Operators in an unmanaged state are unsupported, and the cluster administrator assumes full responsibility for configuration and upgrades.

An Operator can be set to an unmanaged state by using the following methods:

  • Individual Operator configuration

    Individual Operators have a managementState parameter in their configuration. This can be accessed in different ways, depending on the Operator. For example, the Red Hat OpenShift Logging Operator accomplishes this by modifying a custom resource (CR) that it manages, while the Cluster Samples Operator uses a cluster-wide configuration resource.

    Changing the managementState parameter to Unmanaged means that the Operator is not actively managing its resources and will take no action related to the related component. Some Operators might not support this management state as it might damage the cluster and require manual recovery.

    Warning

    Changing individual Operators to the Unmanaged state renders that particular component and functionality unsupported. Reported issues must be reproduced in Managed state for support to proceed.

  • Cluster Version Operator (CVO) overrides

    The spec.overrides parameter can be added to the CVO’s configuration to allow administrators to provide a list of overrides to the CVO’s behavior for a component. Setting the spec.overrides[].unmanaged parameter to true for a component blocks cluster upgrades and alerts the administrator after a CVO override has been set:

    Example output

    Disabling ownership via cluster version overrides prevents upgrades. Please remove overrides before continuing.

    Warning

    Setting a CVO override puts the entire cluster in an unsupported state. Reported issues must be reproduced after removing any overrides for support to proceed.

2.5. Collecting logging data for Red Hat Support

When opening a support case, provide debugging information about your cluster to Red Hat Support.

You can use the must-gather tool to collect diagnostic information for project-level resources, cluster-level resources, and each of the logging components. For prompt support, supply diagnostic information for both OpenShift Container Platform and logging.

2.5.1. About the must-gather tool

The oc adm must-gather CLI command collects the information from your cluster that is most likely needed for debugging issues.

For your logging, must-gather collects the following information:

  • Project-level resources, including pods, config maps, service accounts, roles, role bindings, and events at the project level
  • Cluster-level resources, including nodes, roles, and role bindings at the cluster level
  • OpenShift Logging resources in the openshift-logging and openshift-operators-redhat namespaces, including health status for the log collector, the log store, and the log visualizer

When you run oc adm must-gather, a new pod is created on the cluster. The data is collected on that pod and saved in a new directory that starts with must-gather.local. This directory is created in the current working directory.

2.5.2. Collecting logging data

You can use the oc adm must-gather CLI command to collect information about logging.

Procedure

  1. Navigate to the directory where you want to store the must-gather information.

  2. Run the oc adm must-gather command against the logging image: 

    $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

    The must-gather tool creates a new directory that starts with must-gather.local within the current directory. For example: must-gather.local.4157245944708210408.

  3. Create a compressed file from the must-gather directory that was just created. For example, on a computer that uses a Linux operating system, run the following command: 

    $ tar -cvaf must-gather.tar.gz must-gather.local.4157245944708210408
  4. Attach the compressed file to your support case on the Red Hat Customer Portal.

2.6. Additional resources

Chapter 3. Visualization for logging

Enhance the observability capabilities of the Red Hat OpenShift Logging web console by installing and managing UI plugins.

3.1. Additional resources

Legal Notice

Copyright © Red Hat.
Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.
The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.
All other trademarks are the property of their respective owners.