Loki Ingester 0/1 fails to Connect to Outdated IPs in Gossip Ring in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5
  • 6
• LokiStack
• Loki

Issue

• Lokistack operator fails to start logging-loki-ingester pod due to connection timeout to an outdated IP in the gossip ring.

• IP addresses in the gossip ring endpoint list that are no longer in use causing the issue.

• The logs indicate an error connecting to the outdated IP address: WriteTo failed" addr=<IP>:7946 err="dial tcp <IP>:7946: i/o timeout".

• The IP addresses are not present in the podnetwork or in the list of addresses for the logging-loki-gossip-ring endpoint.

• Loki Ingester in 0/1, even, when not having an issue with the Loki storage as described in the Red Hat Knowledge Article "Loki ingesters 0/1 in RHOCP 4"

• Loki Ingester pod throws the error:

  msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 10.x.x.x:9095 past heartbeat timeout"
  

Resolution

Red Hat investigated this issue in bug report:

If this issue still occurs in the environment after updating, open a support case in the Red Hat Customer Portal referring to this solution.

Workaround

Note: the variable LOGGING_LOKI_DISTRIBUTOR_HTTP_PORT_3100_TCP_ADDR could be different depending on the environment as it has two parts:

• Variable part: LokiStack CR name in upper case. In this example LOGGING_LOKI
• Fixed part: _DISTRIBUTOR_HTTP_PORT_3100_TCP_ADDR

1. Set environment variables

$ cr="logging-loki"      ### LokiStack CR name, change is the name is different 
$ ns="openshift-logging" ### change if Loki runs in a different namespace

2. Enter in a Loki Distributor pod

$ oc -n ${ns} rsh deployment/${cr}-distributor

3. Get the "Unhealthy" Loki Ingester members

Content from ${loki} is not included.https://${loki}:3100/ring%20/

4. Forget the "Unhealthy" Loki Ingester members

$ curl -k --cert /var/run/tls/http/server/tls.crt --key /var/run/tls/http/server/tls.key --noproxy "${LOKI}" https://${LOKI}:3100/ring -X POST --data-raw 'forget=<UNHEALTHY_POD_FROM_EARLIER_COMMAND>'

5. Restart the Loki Ingester pods

$ oc delete pods -l app.kubernetes.io/component=ingester -n [namespace_name]

Root Cause

Loki ingesters that got into an Unhealthy state due to networking issues stayed in that state even when the network recovered.

Diagnostic Steps

1. Set environment variables:

$ cr="logging-loki"      ### LokiStack CR name, change is the name is different 
$ ns="openshift-logging" ### change if Loki runs in a different namespace

2. Verify that the Loki Ingester pod is 0/1

   $ oc get pods -l app.kubernetes.io/component=ingester -n ${ns}
   NAME                   READY   STATUS    RESTARTS   AGE
   lokistack-ingester-0   0/1     Running   0          8d
   

3. Check the logs of the Loki Ingester pod in 0/1. Observe that it indicates that found an existing instance(s) with a problem in the ring:

   $ oc logs lokistack-ingester-0 -n openshift-logging |grep "found an existing instance(s) with a problem in the ring"|tail -1
   2026-07-18T16:12:23.416664742Z level=warn ts=2026-07-18T16:12:23.384385174Z caller=lifecycler.go:295 component=ingester msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 10.128.2.21:9095 past heartbeat timeout"
   

4. Verify that the Loki Ingester being 0/1 is "UNHEALTHY" in the Loki ring. In the next example, the logging-loki-ingester-0isUNHEALTHY`

   $ oc -n ${ns} rsh deployment/${cr}-distributor
   $ LOKI=$(env |grep DISTRIBUTOR_HTTP_PORT_3100_TCP_ADDR|cut -d"=" -f2)
   $ curl -k --cert /var/run/tls/http/server/tls.crt --key /var/run/tls/http/server/tls.key --noproxy "${LOKI}" https://${LOKI}:3100/ring -H "Accept: application/json"     
   

5. Verify that the Loki Gossip ring contains IP addresses that not longer exists

   $ oc describe endpoints ${cr}-gossip-ring -n [namespace_name]
   [...]
   Subsets:
     Addresses:          10.128.1.44,10.128.1.46,10.129.0.73,10.129.0.76,10.129.0.77,10.129.0.78,10.129.0.79,10.130.1.10,10.130.1.11,10.130.1.12,10.130.1.13
     NotReadyAddresses:  10.128.1.45,10.129.0.80
     Ports:
       Name         Port  Protocol
       ----         ----  --------
       gossip-ring  7946  TCP
   

6. Verify that not hitting the issue described in the Red Hat Article Knowledge Base "Loki ingesters 0/1 in RHOCP 4"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.