Loki ingester pods fail to flush to S3 storage with "x-amz-content-sha256 must be UNSIGNED-PAYLOAD" error after upgrade to Logging 6.4.6 in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform
    • 4
  • Red Hat Loki Operator
    • 6.4.6
  • NetApp ONTAP S3
  • S3-compatible object storage

Issue

  • Loki ingester pods fail to flush data chunks to S3-compatible object storage after upgrading Red Hat OpenShift Logging and Loki Operator to version v6.4.6.

  • The LokiIngesterFlushFailureRateCritical alert is firing in the OpenShift console.

  • Loki ingester logs contain the following error message:

    level=error ts=2026-07-02T06:39:52.189509452Z caller=flush.go:262 component=ingester loop=30 org_id=infrastructure msg="failed to flush" retries=5 err="failed to flush chunks: store put chunk: operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value."

  • Loki Compactor fails to put the objects with the error message:

    2026-07-02T07:06:05.701053175Z level=error ts=2026-07-02T07:06:05.627605124Z caller=delete_requests_db_boltdb.go:94 msg="failed to upload delete requests file" err="operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value."

Resolution

This issue has been reported to Red Hat engineering. It is being tracked in Bug Content from redhat.atlassian.net is not included.LOG-9632. For more information, please open a This content is not included.new support case on the Red Hat Customer Portal referring to this solution.

Workaround 1

  1. Downgrade the Red Hat Loki Operator to v6.4.5 version following the Red Hat Article How to downgrade Red Hat OpenShift Logging and Loki Operator in RHOCP 4 or to the working version before upgrading to v6.4.6. The previous version used can be obtained running the command:

    $ oc get csv -n openshift-logging|egrep "loki|NAME"
NAME                     DISPLAY                     VERSION   REPLACES                 PHASE
loki-operator.v6.4.6     Loki Operator               6.4.6     loki-operator.v6.4.5     Succeeded

  2. Set the installPlanApproval to Manual to avoid the automatic upgrade.

Note: it is not required to downgrade the Red Hat Logging Operator. It should only be downgraded the Red Hat Loki Operator

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

In the upstream pull request Content from github.com is not included.21848 was introduced a regression on S3-compatible storage that not support the STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER value in the x-amz-content-sha256 header:

			ChecksumAlgorithm: types.ChecksumAlgorithmSha256,

This Loki upstream version is used in the Red Hat Loki Operator v6.4.6.

Diagnostic Steps

  1. Verify that the Loki operator is in the version affected listed in the Environment section

    $ oc get csv -n openshift-logging |grep -i loki

  2. Set the environment variables. Adjust the LokiStack CR name and namespace from the one to be verified

    $ cr="logging-loki"
$ ns="openshift-logging"

  3. Confirm that the Loki Ingesters have the error api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD:

    $ oc logs ${cr}-ingester-0 n ${ns}|grep "x-amz-content-sha256 must be UNSIGNED-PAYLOAD"|tail -1
2026-07-02T07:09:16.704133420Z level=error ts=2026-07-02T07:09:16.645805962Z caller=flush.go:262 component=ingester loop=1 org_id=infrastructure msg="failed to flush" retries=2 err="failed to flush chunks: store put chunk: operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value., num_chunks: 1, labels: {k8s_container_name=\"kube-scheduler-cert-syncer\", k8s_namespace_name=\"openshift-kube-scheduler\", k8s_node_name=\"worker-example-com\", k8s_pod_name=\"openshift-kube-scheduler-worker-example-com\", kubernetes_container_name=\"kube-scheduler-cert-syncer\", kubernetes_host=\"worker-example-com\", kubernetes_namespace_name=\"openshift-kube-scheduler\", kubernetes_pod_name=\"openshift-kube-scheduler-worker-example-com\", log_type=\"infrastructure\", openshift_log_type=\"infrastructure\"}"
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.