Service Mesh Visualization and Distributed Tracing for SAP Edge Integration Cell on Red Hat OpenShift

Updated

Part of: SAP Edge Integration Cell on Red Hat OpenShift

Table of Contents

Overview

This guide describes how to set up service mesh visualization and distributed tracing for SAP Edge Integration Cell (EIC) on Red Hat OpenShift. It covers three complementary capabilities:

  • Distributed Tracing (Tempo + OpenTelemetry) -- Captures request traces across EIC services and Istio sidecars, stored in a TempoStack instance and viewable through Kiali and the OpenShift console.
  • Kiali -- Provides a dedicated console for visualizing service mesh topology, traffic flows, configuration validation, and distributed traces.
  • OpenShift Service Mesh Console (OSSMC) Plugin -- Embeds Kiali features (Traffic Graph, Istio Config, Mesh overview) directly into the OpenShift web console.

This guide complements the other two observability articles in this series:

Together, these three guides provide a complete observability stack: metrics, logs, and traces with mesh visualization.

Resource planning: This setup deploys several additional components on your cluster: TempoStack (ingester, compactor, querier, and gateway pods), an OpenTelemetry Collector, and a Kiali instance. TempoStack also requires S3-compatible object storage (e.g., Amazon S3, MinIO, or OpenShift Data Foundation) that must be provisioned separately. Higher trace sampling rates increase CPU, memory, and storage consumption proportionally. Plan your cluster capacity accordingly -- especially on smaller or resource-constrained clusters. See the Tempo Operator documentation for TempoStack resource requirements.

Already have tracing in place? If your organization already operates a tracing platform (e.g., Jaeger, Zipkin, Datadog APM, or similar), you can skip the TempoStack deployment and configure only the OpenTelemetry Collector to forward traces to your existing backend. The Istio tracing configuration in Step 3 applies regardless of the backend.

Intended Audience: This document is for Red Hat OpenShift Cluster Administrators who have already deployed EIC on OpenShift with Service Mesh 3.x and want to add distributed tracing and mesh visualization.

Prerequisites

Before beginning this setup, ensure you have completed the following requirements:

Required Access and Credentials

  • Cluster-admin privileges on the target OpenShift cluster
  • Authenticated oc CLI session to your OpenShift cluster

Existing EIC Deployment

Object Storage

TempoStack requires S3-compatible object storage for trace data. Supported backends include:

  • Amazon S3
  • MinIO
  • Azure Blob Storage
  • Google Cloud Storage
  • Red Hat OpenShift Data Foundation (ODF)

Have your object storage endpoint, bucket name, and credentials ready before proceeding.

Required Operators

Install the following operators from OperatorHub (Operators -> OperatorHub) before proceeding:

OperatorRoleWhat it provides
Red Hat Tempo OperatorTrace storageTempoStack -- stores and indexes distributed traces
Red Hat build of OpenTelemetryTrace collectionOpenTelemetryCollector -- receives, processes, and forwards trace data
Kiali Operator provided by Red HatMesh visualizationKiali console for traffic graphs, configuration validation, and trace analysis

Warning: Do not install the Community versions of these operators. Only the Red Hat-provided versions are supported.

Step 1: Deploy TempoStack

TempoStack provides the trace storage backend. It uses the Grafana Tempo project to store and index distributed traces, with a Jaeger-compatible query frontend for trace exploration.

Create the Tempo namespace and storage secret

oc new-project tempo

oc create secret generic tempo-storage-s3 -n tempo \
  --from-literal=bucket="tempo-traces" \
  --from-literal=endpoint="https://<your-s3-endpoint>" \
  --from-literal=access_key_id="<your-access-key>" \
  --from-literal=access_key_secret="<your-secret-key>"

Note: Replace all <placeholder> values with your actual object storage details. If using MinIO, the endpoint is typically http://minio.minio.svc:9000. For ODF, see the This content is not included.ODF documentation for the NooBaa S3 endpoint.

Deploy the TempoStack CR

oc apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: sample
  namespace: tempo
spec:
  managementState: Managed
  tenants:
    mode: openshift
    authentication:
    - tenantName: dev
      tenantId: dev
  template:
    gateway:
      enabled: true
    queryFrontend:
      jaegerQuery:
        enabled: true
  storage:
    secret:
      name: tempo-storage-s3
      type: s3
EOF

Configuration notes:

  • tenants.mode: openshift -- Uses OpenShift's built-in authentication and authorization for multi-tenant trace access. The authentication block must explicitly list at least one tenant (dev), or the OPA authorization sidecar will fail to start with missing tenant mappings.
  • gateway.enabled: true -- Deploys a gateway that handles authentication via OpenShift OAuth and provides a Route for trace query access.
  • jaegerQuery.enabled: true -- Enables the Jaeger-compatible query frontend behind the gateway, which Kiali uses to retrieve traces (do not set jaegerQuery.ingress.type when the gateway is enabled -- the two options are mutually exclusive).

Wait for all TempoStack pods to reach Running state:

oc get pods -n tempo -l app.kubernetes.io/instance=sample --watch

Configure trace access RBAC

Grant authenticated users read access to traces, and create a write role for the OpenTelemetry Collector:

oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-reader
rules:
- apiGroups:
  - tempo.grafana.com
  resources:
  - dev
  resourceNames:
  - traces
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-reader
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:authenticated
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-write
rules:
- apiGroups:
  - tempo.grafana.com
  resources:
  - dev
  resourceNames:
  - traces
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-write
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-write
subjects:
- kind: ServiceAccount
  name: otel-collector
  namespace: <istio-namespace>
EOF

Note: The resources field (dev) is the tenant name. In openshift tenancy mode, the default tenant name is dev. The reader role is granted to all authenticated users; the writer role is granted specifically to the otel-collector ServiceAccount that will be created in the next step.

Step 2: Deploy the OpenTelemetry Collector

The OpenTelemetry Collector receives trace data from Istio sidecars via OTLP (gRPC) and forwards it to the TempoStack distributor.

oc apply -f - <<EOF
kind: OpenTelemetryCollector
apiVersion: opentelemetry.io/v1beta1
metadata:
  name: otel
  namespace: <istio-namespace>
spec:
  observability:
    metrics: {}
  deploymentUpdateStrategy: {}
  config:
    exporters:
      otlp:
        endpoint: tempo-sample-gateway.tempo.svc.cluster.local:8090
        tls:
          insecure: false
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: dev
    extensions:
      bearertokenauth:
        filename: /var/run/secrets/kubernetes.io/serviceaccount/token
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http: {}
    service:
      extensions:
      - bearertokenauth
      pipelines:
        traces:
          exporters:
          - otlp
          receivers:
          - otlp
EOF

Configuration notes:

  • endpoint -- Points to the TempoStack gateway service. The gateway handles authentication and multi-tenant routing.
  • X-Scope-OrgID: dev -- Identifies the tenant for trace storage. Must match the tenant name used in the RBAC configuration.
  • bearertokenauth -- Uses the collector's ServiceAccount token to authenticate with the Tempo gateway.
  • tls.ca_file -- Trusts the OpenShift service CA certificate for TLS connections to the gateway.

Verify the collector is running:

oc get pods -n <istio-namespace> -l app.kubernetes.io/name=otel-collector

Disable mTLS for the collector (required when using STRICT mTLS)

The OpenTelemetry Collector runs without an Istio sidecar. If your mesh namespaces enforce STRICT mTLS via PeerAuthentication, sidecars will attempt mTLS when sending traces to the collector and fail because the collector cannot terminate mTLS. Create a DestinationRule to disable mTLS for the collector service:

oc apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: otel-collector-disable-mtls
  namespace: <istio-namespace>
spec:
  host: otel-collector.<istio-namespace>.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE
EOF

Note: This DestinationRule is applied once in the collector's namespace and takes effect mesh-wide -- you do not need to create one per namespace. Without it, traces will appear for istio-ingressgateway (which sends traces internally, not through the mTLS pipeline) but not for application workloads like policyengine or edc. The symptom is silent: TCP connections to the collector succeed but all gRPC trace requests fail (rq_error in Envoy stats, rq_success: 0).

Step 3: Enable Tracing in the Istio Control Plane

Add the extensionProviders block to your existing Istio CR to register the OpenTelemetry Collector as a tracing provider. This modifies the Istio CR deployed in Setting Up a Restricted-Access OpenShift Cluster for SAP ELM with Service Mesh 3.x.

Patch the Istio CR:

oc patch istio <your-istio-cr-name> --type merge -p '
spec:
  values:
    meshConfig:
      extensionProviders:
        - name: otel-tracing
          opentelemetry:
            port: 4317
            service: otel-collector.<istio-namespace>.svc.cluster.local
'

Note: Replace <your-istio-cr-name> with the name of your Istio CR (e.g., default or sap-mesh-istio) and <istio-namespace> with the namespace where the OpenTelemetry Collector is deployed (the same namespace as your Istio control plane). The extensionProviders field registers the collector as an available tracing backend for Istio sidecars. If your Istio CR already has other extensionProviders entries, use oc edit instead to append to the existing list rather than replacing it.

Create the Telemetry resource

The Telemetry resource activates the tracing provider and sets the sampling rate:

oc apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: otel-tracing
  namespace: <istio-namespace>
spec:
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 100
EOF

Note: A randomSamplingPercentage of 100 captures all traces, which is useful for initial setup and verification. For production environments, lower this value (e.g., 1 or 10) to reduce trace volume and storage costs. You can also set it to default to use Istio's built-in sampling rate.

Step 4: Install and Configure Kiali

Kiali is the management console for Red Hat OpenShift Service Mesh. It provides traffic topology visualization, configuration validation, health monitoring, and integration with distributed tracing.

Install the Kiali Operator

  1. Log in to the OpenShift web console.
  2. Navigate to Operators -> OperatorHub.
  3. Type Kiali into the filter box to find the Kiali Operator provided by Red Hat.
  4. Click Install, select the stable Update Channel, and accept the default settings.

Warning: Do not install the Community version of the Kiali Operator. The Community version is not supported.

Verify Installation:

oc get csv -n openshift-operators | grep kiali

You should see the operator with a Succeeded phase.

Grant Kiali monitoring access

Create a ClusterRoleBinding so Kiali's service account can query the platform Thanos Querier for metrics:

oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kiali-monitoring-rbac
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-monitoring-view
subjects:
- kind: ServiceAccount
  name: kiali-service-account
  namespace: <istio-namespace>
EOF

Create the Kiali CR

This configuration integrates Kiali with the platform Thanos Querier for metrics and with the TempoStack for distributed tracing:

oc apply -f - <<EOF
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: <istio-namespace>
spec:
  external_services:
    prometheus:
      auth:
        ca_file: /kiali-cabundle/service-ca.crt
        insecure_skip_verify: false
        type: bearer
        use_kiali_token: true
      thanos_proxy:
        enabled: true
      url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
    tracing:
      enabled: true
      provider: tempo
      use_grpc: false
      internal_url: https://tempo-sample-gateway.tempo.svc.cluster.local:8080/api/traces/v1/dev/tempo
      auth:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        insecure_skip_verify: false
        type: bearer
        use_kiali_token: true
      tempo_config:
        url_format: jaeger
EOF

Configuration notes:

  • prometheus -- Connects to the platform Thanos Querier using Kiali's own ServiceAccount bearer token and the OpenShift service CA for TLS. The thanos_proxy setting enables efficient metric aggregation.
  • tracing.internal_url -- The cluster-internal URL used by the Kiali server to query traces. Uses the in-cluster Service FQDN. The URL path includes the tenant name (dev) and the /tempo suffix for the native Tempo query API.
  • tracing.provider: tempo -- Tells Kiali to use the Tempo API for trace queries.
  • tempo_config.url_format: jaeger -- Tells Kiali to use the Jaeger-compatible query format when fetching traces from the Tempo API.

Wait for the Kiali pod to be ready:

oc get pods -n <istio-namespace> -l app.kubernetes.io/name=kiali --watch

Access the Kiali UI:

echo "https://$(oc get routes -n <istio-namespace> kiali -o jsonpath='{.spec.host}')"

Open the URL in your browser. Navigate to the Traffic Graph tab to see the mesh topology, and use the Workload -> Traces tab to view distributed traces.

Step 5 (Optional): Install the OpenShift Service Mesh Console Plugin

The OpenShift Service Mesh Console (OSSMC) plugin integrates Kiali features directly into the OpenShift web console. Once installed, a Service Mesh menu appears in the OpenShift console navigation with the following views:

  • Overview -- Summary of mesh namespaces
  • Traffic Graph -- Full topology view of mesh traffic
  • Istio Config -- List of all Istio configuration resources with validation status
  • Mesh -- Multi-cluster mesh view (if applicable)

Warning: The OSSMC plugin supports only one Kiali instance, regardless of its project access scope.

Install the OSSMC Plugin

oc apply -f - <<EOF
apiVersion: kiali.io/v1alpha1
kind: OSSMConsole
metadata:
  name: ossmconsole
  namespace: <istio-namespace>
spec:
  kiali:
    serviceName: kiali
    serviceNamespace: <istio-namespace>
EOF

Wait for the plugin to reconcile:

oc get ossmconsole ossmconsole -n <istio-namespace> -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

You should see Successful=True. After applying, refresh the OpenShift console. The Service Mesh category will appear in the left navigation menu.

Verification Checklist

Use this checklist to confirm the tracing and visualization stack is operational:

TempoStack:

# TempoStack pods are running
oc get pods -n tempo -l app.kubernetes.io/instance=tempo-sample

# Gateway route exists
oc get routes -n tempo

OpenTelemetry Collector:

# Collector pod is running
oc get pods -n <istio-namespace> -l app.kubernetes.io/name=otel-collector

# DestinationRule exists (required for STRICT mTLS environments)
oc get destinationrule -n <istio-namespace> otel-collector-disable-mtls

Istio Tracing Configuration:

# Verify extensionProviders includes the tracing provider
oc get istio <your-istio-cr-name> -o jsonpath='{.spec.values.meshConfig.extensionProviders}'

# Verify the Telemetry resource exists
oc get telemetry otel-tracing -n <istio-namespace>

Kiali:

# Kiali pod is running
oc get pods -n <istio-namespace> -l app.kubernetes.io/name=kiali

# Kiali route exists
oc get routes -n <istio-namespace> kiali

# Kiali operator should show Succeeded
oc get csv -n openshift-operators | grep kiali

OSSMC Plugin (if installed):

# OSSMC plugin is reconciled
oc get ossmconsole ossmconsole -n <istio-namespace> -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

End-to-end trace verification:

  1. Generate some traffic to the EIC endpoints (or wait for normal application traffic).
  2. Open the Kiali UI and navigate to a workload in the edge-icell namespace.
  3. Click the Traces tab -- you should see distributed traces showing request flows across EIC services and their Istio sidecars.
  4. Alternatively, install the OSSMC plugin (Step 5) to view traces directly in the OpenShift web console under Service Mesh -> Workloads -> Traces.
Article Type