Reducing log loss during node scaledown by ensuring Vector terminates last.

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging(RHOL)
  • 5
  • 6
• Vector

Issue

• Reducing log loss during node scaledown by ensuring Vector terminates last.
• During node shutdown/scaledown, the Vector pod terminates before other workload pods, leading to log loss. Ideally, the Vector pod should be the last to terminate, ensuring that workload pod logs are captured and forwarded before shutdown.

Resolution

• Red Hat Logging stack is not a guaranteed log sink, feature request This content is not included.OBSDA-1206 is raised to review if log loss caused by the Vector's early during node scaledown/shutdown event can be reduced.
• For more information please open a new support case with Red Hat Support.
• In order to allow the collector to persist already collected log-entries until they are shipped to the target, validate if the setting clusterlogforwarder.spec.outputs.[type].tuning.deliveryMode is set to atLeastOnce, which enables buffering on disk. Otherwise this buffer will be in memory and lost during a reboot of the node or restart of the collector.

Diagnostic Steps

Steps to Validate:

1> Deployed a log-generator application that continuously writes logs to both Persistent Volume and STDOUT.

2> Initiated a shutdown of the node hosting this log-generator pod.

3> Compare the logs:

Example:
PV recorded up to line 95999
Loki recorded only up to line 93627
This confirms a loss of 2372 log lines during shutdown.

4> Verify deliveryMode is unset or set to AtMostOnce

% oc get clusterlogforwarder -n openshift-logging \ 
  -o json | jq -r '.items[].spec.outputs[] | "\(.name)\t\(.type)\t\(.[.type].tuning.deliveryMode // "unset")"'
default-lokistack	lokiStack	AtMostOnce

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.