Configuring NooBaa Prometheus Alert Rules - OpenShift Data Foundation 4.22 (Developer Preview)
Overview
NooBaa ships a PrometheusRule custom resource that contains recording rules and alert definitions. This CR is automatically created by the NooBaa operator during system setup.
The CR is named <noobaa-name>-prometheus-rules in the NooBaa namespace (e.g. noobaa-prometheus-rules for a default installation).
Important: ODF Does Not Reconcile Prometheus Rules
Neither the NooBaa operator nor ODF reconciles the PrometheusRule spec after its initial creation. The operator only ensures the CR exists — it does not enforce or overwrite its contents on subsequent reconciliation cycles.
This means:
- You can directly edit the
PrometheusRuleCR to customize alerts and recording rules. - Your changes will persist across operator reconciliations and pod restarts.
- If the CR is deleted, the operator will recreate it with the default rules on the next reconciliation cycle.
-
List the PrometheusRule CR.
Verify the `PrometheusRule` exists in your namespace: ``` $ oc -n get prometheusrules ``` Expected output:
NAME AGE
noobaa-prometheus-rules 1d
-
View the Current Rules.
Inspect the full CR to understand the existing rules before making changes: ``` $ oc -n get prometheusrule -prometheus-rules -o yaml ``` The `PrometheusRule` CR contains several rule groups organized by function. Knowing these group names can help you target specific sections when editing the YAML:
| Rule Group | Type | Description |
|---|---|---|
noobaa-telemeter.rules | Recording | Aggregated metrics for unhealthy buckets, bucket counts, object counts, accounts, and usage |
noobaa-odf.rules | Recording | ODF integration metrics — health status, capacity, IOPS, throughput |
noobaa-replication.rules | Recording | Replication write sizes, counts, errors, and cycle totals |
replication-alert.rules | Alert | Fires when a replication target bucket is unreachable for more than 5 minutes |
bucket-state-alert.rules | Alert | Fires on bucket error states, size/quantity quota warnings, and backing store low capacity |
resource-state-alert.rules | Alert | Fires when a backing store or namespace resource enters error state |
system-capacity-alert.rules | Alert | Fires at 85%, 95%, and 100% system capacity thresholds |
noobaa-db-alert.rules | Alert | Fires when the NooBaa database PVC reaches 80% or exceeds 90% capacity |
-
Edit the PrometheusRule CR.
Use `oc edit` to modify the rules directly: ``` $ oc -n edit prometheusrule -prometheus-rules ``` *For example*: Change an alert threshold — raise the system capacity warning from 85% to 90% and fire the bucket error alert only after 15 minutes instead of 5:
# Before
- alert: NooBaaSystemCapacityWarning85
expr: |
NooBaa_system_capacity > 85
for: 5m
# After
- alert: NooBaaSystemCapacityWarning90 # Optionally rename alert for clarity
expr: |
NooBaa_system_capacity > 90
for: 15m
-
Verify the Changes
Confirm the CR was updated correctly: ``` $ oc -n get prometheusrule -prometheus-rules -o yaml ```
Restoring Default Rules
If you want to reset the rules back to defaults:
-
Delete the
PrometheusRuleCR:``` $ oc -n <namespace> delete prometheusrule <noobaa-name>-prometheus-rules ``` -
The NooBaa operator will automatically recreate it with the default rules on the next reconciliation cycle (typically within seconds).
Key points
- Edits are safe — The NooBaa operator and ODF will not overwrite your changes.
- Deletion triggers recreation — Deleting the CR restores defaults automatically.
- Labels matter — The CR uses labels prometheus: k8s and role: alert-rules. Your Prometheus instance must have matching ruleSelector labels to discover these rules.