Configuring NooBaa Prometheus Alert Rules - OpenShift Data Foundation 4.22 (Developer Preview)

Updated

Overview

NooBaa ships a PrometheusRule custom resource that contains recording rules and alert definitions. This CR is automatically created by the NooBaa operator during system setup.

The CR is named <noobaa-name>-prometheus-rules in the NooBaa namespace (e.g. noobaa-prometheus-rules for a default installation).

Important: ODF Does Not Reconcile Prometheus Rules

Neither the NooBaa operator nor ODF reconciles the PrometheusRule spec after its initial creation. The operator only ensures the CR exists — it does not enforce or overwrite its contents on subsequent reconciliation cycles.

This means:

  • You can directly edit the PrometheusRule CR to customize alerts and recording rules.
  • Your changes will persist across operator reconciliations and pod restarts.
  • If the CR is deleted, the operator will recreate it with the default rules on the next reconciliation cycle.
  1. List the PrometheusRule CR.

    Verify the `PrometheusRule` exists in your namespace:
    
    
    ```
    $ oc -n  get prometheusrules
    ```
    
    
    Expected output:
    
NAME                        AGE
noobaa-prometheus-rules     1d
  1. View the Current Rules.

    Inspect the full CR to understand the existing rules before making changes:
    
    
    ```
    $ oc -n  get prometheusrule -prometheus-rules -o yaml
    ```
    
    
    The `PrometheusRule` CR contains several rule groups organized by function. Knowing these group names can help you target specific sections when editing the YAML:
    
Rule GroupTypeDescription
noobaa-telemeter.rulesRecordingAggregated metrics for unhealthy buckets, bucket counts, object counts, accounts, and usage
noobaa-odf.rulesRecordingODF integration metrics — health status, capacity, IOPS, throughput
noobaa-replication.rulesRecordingReplication write sizes, counts, errors, and cycle totals
replication-alert.rulesAlertFires when a replication target bucket is unreachable for more than 5 minutes
bucket-state-alert.rulesAlertFires on bucket error states, size/quantity quota warnings, and backing store low capacity
resource-state-alert.rulesAlertFires when a backing store or namespace resource enters error state
system-capacity-alert.rulesAlertFires at 85%, 95%, and 100% system capacity thresholds
noobaa-db-alert.rulesAlertFires when the NooBaa database PVC reaches 80% or exceeds 90% capacity
  1. Edit the PrometheusRule CR.

    Use `oc edit` to modify the rules directly:
    
    
    ```
    $ oc -n  edit prometheusrule -prometheus-rules
    ```
    
    
    *For example*: 
    
    Change an alert threshold — raise the system capacity warning from 85% to 90% and fire the bucket error alert only after 15 minutes instead of 5:
    
  # Before
  - alert: NooBaaSystemCapacityWarning85
    expr: |
      NooBaa_system_capacity > 85
    for: 5m
  # After
  - alert: NooBaaSystemCapacityWarning90  # Optionally rename alert for clarity
    expr: |
      NooBaa_system_capacity > 90
    for: 15m
  1. Verify the Changes

    Confirm the CR was updated correctly:
    
    
    ```
    $ oc -n  get prometheusrule -prometheus-rules -o yaml
    ```
    

Restoring Default Rules

If you want to reset the rules back to defaults:

  • Delete the PrometheusRule CR:

     ```
     $ oc -n <namespace> delete prometheusrule <noobaa-name>-prometheus-rules
     ```
    
  • The NooBaa operator will automatically recreate it with the default rules on the next reconciliation cycle (typically within seconds).

Key points

  • Edits are safe — The NooBaa operator and ODF will not overwrite your changes.
  • Deletion triggers recreation — Deleting the CR restores defaults automatically.
  • Labels matter — The CR uses labels prometheus: k8s and role: alert-rules. Your Prometheus instance must have matching ruleSelector labels to discover these rules.
Article Type