Prometheus storage size impacted by high cardinality of Vector metrics in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Dedicated (RHOSD)
    • 4
  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Prometheus
  • Red Hat OpenShift Logging
  • Vector

Issue

  • It's observed high cardinality coming from metrics vector_component_received_events_count_bucket and vector_buffer_send_duration_seconds_bucket analysis

  • After analyzing the Prometheus tsdb, it's observed that the metrics with higher cardinality are coming from Vector: vector_*

    Highest cardinality metric names:
727740 vector_component_received_events_count_bucket
719960 vector_buffer_send_duration_seconds_bucket
166536 etcd_request_duration_seconds_bucket
150240 vector_source_lag_time_seconds_bucket
137720 vector_adaptive_concurrency_reached_limit_bucket
137720 vector_adaptive_concurrency_averaged_rtt_bucket
137720 vector_adaptive_concurrency_past_rtt_mean_bucket
137720 vector_adaptive_concurrency_observed_rtt_bucket
137720 vector_adaptive_concurrency_limit_bucket
137720 vector_adaptive_concurrency_in_flight_bucket
137720 vector_adaptive_concurrency_back_pressure_bucket

Resolution

The option for reducing the high cardinality should be to reduce the number of inputs, outputs and pipelines.

It was created the feature request RFE This content is not included.OBSDA-1341 for reducing the Vector cardinality. If this feature is required, please open a This content is not included.new support case on the Red Hat Customer Portal referring to this solution.

Root Cause

Setting a high number of inputs, outputs or pipelines, it will generate that some of the metrics generated by Vector will have a high cardinality.

Let's review the component_id present in the vector_component_received_events_count_bucket and let's check it visually, for it, review the screenshot attached in the Red Hat Knowledge Article "How to use vector tap and vector top for troubleshooting in RHOCP 4", and in the "vector top" section is present an screenshot where the first column will be the Components ID (label component_id from the metric vector_component_received_events_count_bucket) and each component_id has Events In, Events Out, Bytes In and Bytes Out.

When it's created a simple input , outputs, filters in the clusterLogForwarder instance, it's created different components_id as part of the pipelines as for every input is at least required to get the metadata from the logs and doing some transformations: filter, remap, etc that it will finish having a high cardinality in the component_id and with it, a high number of samples collected for the metric vector_component_received_events_count_bucket.

Diagnostic Steps

  1. Analyze the Prometheus tsdb for getting the labels and metrics with highest cardinality

    $ oc exec -n openshift-monitoring prometheus-k8s-0 -c prometheus -- /bin/sh -c "promtool tsdb analyze /prometheus"
...
Most common label pairs:
3642847 endpoint=metrics
3166018 namespace=openshift-logging
3116791 container=collector
3105098 app_kubernetes_io_component=collector
3105098 app_kubernetes_io_part_of=cluster-logging
3105098 app_kubernetes_io_name=vector
2850648 app_kubernetes_io_instance=app-logforward-customers
2850576 job=app-logforward-customers
2850573 service=app-logforward-customers
...
Highest cardinality metric names:
727740 vector_component_received_events_count_bucket
719960 vector_buffer_send_duration_seconds_bucket
150240 vector_source_lag_time_seconds_bucket
137720 vector_adaptive_concurrency_reached_limit_bucket
137720 vector_adaptive_concurrency_averaged_rtt_bucket
137720 vector_adaptive_concurrency_past_rtt_mean_bucket
137720 vector_adaptive_concurrency_observed_rtt_bucket
137720 vector_adaptive_concurrency_limit_bucket
137720 vector_adaptive_concurrency_in_flight_bucket
137720 vector_adaptive_concurrency_back_pressure_bucket

  2. Go to the OpenShift Console > Observe > Metrics and run the PromQL count(sum by (component_id)(rate(vector_component_received_events_count_bucket[1m]))) to get the count of different values that the component_id can take for the metric vector_component_received_events_count_bucket:

    count(sum by (component_id)(rate(vector_component_received_events_count_bucket[1m])))
484

  3. Run the previous PromQL adding the filter to get the count of the inputs_*

    count(sum by (component_id)(rate(vector_component_received_events_count_bucket{component_id=~"input_.*"}[1m])))
188

  4. When analyzed the Prometheus tsdb was found that one of the labels with more pairs was app_kubernetes_io_instance=app-logforward-customers. The value of this label is the name of the clusterLogForwarder instance. Let's use it as filter to get the count of the vector_component_received_events_count_bucket samples and by component_id:

    count(sum by (component_id)(rate(vector_component_received_events_count_bucket{app_kubernetes_io_instance="app-logforward-customers"}[1m])))
453

count(rate(vector_component_received_events_count_bucket{app_kubernetes_io_instance="app-logforward-customers"}[1m]))
679500

  5. Get the clusterLogForwarder instance to verify that the instance app-logforward-customers to review the number of inputs, outputs and pipelines set:

    $ oc get clusterlogforwarder app-logforward-customers -n openshift-logging -o yaml

  6. Same Diagnostic steps can be done for the metric vector_buffer_send_duration_seconds_bucket

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.