CollectorNodeDown

Prometheus could not scrape vector <instance> for more than 10m.

Vector is reporting that Prometheus could not scrape a specific Vector instance.

Critical

DiskBufferUsage

Collectors potentially consuming too much node disk, <value>

Collectors are consuming too much node disk on the host.

Warning

CollectorHigh403ForbiddenResponseRate

High rate of "HTTP 403 Forbidden" responses detected for collector <instance> in namespace <namespace> for output <label>. The rate of 403 responses is <rate> over the last 2 minutes, persisting for more than 5 minutes. This could indicate an authorization issue.

At least 10% of sent requests responded with "HTTP 403 Forbidden" for collector "<intance>" in namespace <namespace> for the output "<output>".

Critical

LokiIngesterFlushFailureRateCritical

Loki ingester {{ $labels.pod }} in the namespace {{ $labels.namespace }} has a critical flush failure rate of {{ $value | humanizePercentage }} over the last 5 minutes. This requires immediate attention as data is not being flushed to the storage. Validate if the storage configuration is still valid and if the storage is still reachable. Current failure rate: {{ $value | humanizePercentage }} Threshold: 20%

One or more Loki ingesters are failing to flush at least 20% of their chunks to backend storage over a 5-minute period. This indicates issues with storage connectivity, authentication, or storage capacity that require immediate intervention.

critical

LokiRequestErrors

{{ $labels.job }} {{ $labels.route }} is experiencing <value>% errors.

At least 10% of requests result in 5xx server errors.

critical

LokiStackWriteRequestErrors

<value>% of write requests from {{ $labels.job }} in <namespace> are returned with server errors.

At least 10% of write requests to the lokistack-gateway result in 5xx server errors.

critical

LokiStackReadRequestErrors

<value>% of query requests from {{ $labels.job }} in <namespace> are returned with server errors.

At least 10% of query requests to the lokistack-gateway result in 5xx server errors.

critical

LokiRequestPanics

{{ $labels.job }} is experiencing an increase of <value> panics.

A panic was triggered.

critical

LokiRequestLatency

{{ $labels.job }} {{ $labels.route }} is experiencing <value>s 99th percentile latency.

The 99th percentile is experiencing latency higher than 1 second.

critical

LokiTenantRateLimit

{{ $labels.job }} {{ $labels.route }} is experiencing 429 errors.

At least 10% of requests are received the rate limit error code.

warning

LokiStorageSlowWrite

The storage path is experiencing slow write response rates.

The storage path is experiencing slow read response rates.

warning

LokiWritePathHighLoad

The write path is experiencing high load.

The write path is experiencing high load causing backpressure storage flushing.

warning

LokiReadPathHighLoad

The read path is experiencing high load.

The read path has a high volume of queries, causing longer response times.

warning

LokiDiscardedSamplesWarning

Loki in namespace "<namespace>" is discarding samples in the "<tenant>" tenant during ingestion. Samples are discarded because of "<reason>" at a rate of <value> samples per second.

Loki is discarding samples during ingestion because they fail validation.

warning

LokistackComponentsNotReadyWarning

The LokiStack "{{ $labels.stack_name }}" in namespace "{{ $labels.namespace }}" has components that are not ready.

The LokiStack resource is reporting that some components have not reached the Ready state. This might be related to OpenShift resources such as pods or deployments, configuration, or external dependencies.

warning

LokistackSchemaUpgradesRequired

The LokiStack "{{ $labels.stack_name }}" in namespace "<namespace>" is using a storage schema configuration that does not contain the latest schema version. It is recommended to update the schema configuration to update the schema version to the latest

One or more of the deployed LokiStacks has an outdated storage schema configuration.

warning