noobaa-db-pg-cluster now synchronizes successfully without WAL segment errors

The noobaa-db-pg-cluster database cluster now synchronizes properly and avoids synchronization loops. Previously, the cluster would stop synchronizing and log an error indicating that a requested Write-Ahead Logging (WAL) segment had already been removed. To resolve this and increase replication resiliency, the configuration parameters wal_keep_size, wal_level, wal_sender_timeout, wal_receiver_timeout, and failoverDelay have been updated for NooBaa’s PostgreSQL database usage.

(This content is not included.DFBUGS-6943)

Intermittent S3 upload failures (HTTP 500) to NooBaa buckets have been resolved

Uploading files to NooBaa buckets via JFrog Artifactory no longer fails intermittently with HTTP 500 / InternalError. Previously, concurrent PUT uploads to the same bucket and key could cause a database unique constraint conflict when attempting to update version records simultaneously, causing the second upload to fail. This issue has been resolved by implementing internal retry logic within the upload completion flow.

(This content is not included.DFBUGS-6548)

NooBaa database cleaner environment variables are no longer overridden during reconciliation

Environment variables configured for the NooBaa database cleaner (DB_CLEANER) now properly persist and take effect. Previously, the noobaa-operator would overwrite custom environment variables on the core pod during its reconciliation loop, causing user-defined values (such as database cleaner timers) to be ignored. This issue has been resolved by ensuring the operator retains and respects existing custom configurations.

(This content is not included.DFBUGS-5052)

The PodDisruptionBudgetAtLimit alert for the noobaa-db-pg-cluster-primary PDB is now automatically silenced

A misleading warning alert is no longer triggered for single-primary database configurations. Previously, CloudNativePG (the database operator used by NooBaa) created a PodDisruptionBudget (PDB) object to protect the primary database instance from deletion. Because there is only one primary, the PDB correctly allowed zero disruptions, which inadvertently caused OpenShift to fire a misleading PodDisruptionBudgetAtLimit warning alert. This issue has been resolved by updating the NooBaa operator to automatically silence this specific alert using the Alertmanager API.

(This content is not included.DFBUGS-5294)

noobaa-core logs are no longer spammed by SignatureDoesNotMatch errors

S3 operations now succeed without signature validation failures, and the noobaa-core logs are no longer filled with error messages. Previously, a regression introduced during the migration to AWS SDK V3 caused incorrect signature calculations, which resulted in failed S3 operations and log spam. This issue has been resolved by removing applyChecksum from the requests.

(This content is not included.DFBUGS-4461)

Regional-DR is now supported in environments deployed on IBM Power

Regional-DR is now supported in OpenShift Data Foundation environments deployed on IBM Power because ACM 2.15 support has been introduced for this platform in this release. This applies to both new and upgraded deployments on IBM Power.

(This content is not included.DFBUGS-5369)

CIDR range does now persists in csiaddonsnode object when the respective node is down

The Classless Inter-Domain Routing (CIDR) information now correctly persists in the csiaddonsnode object even when a node is down. This ensures the fencing mechanism functions as intended when required to fence impacted nodes, eliminating the need to manually collect CIDR information immediately after the NetworkFenceClass object is created.

(This content is not included.DFBUGS-2948)

Remove DR option is now available for discovered apps on the Virtual machines page

The Remove DR option is now available for discovered applications listed on the Virtual machines page. You no longer need to manually add the missing label to the DRPlacementControl or patch the PROTECTED_VMS recipe parameter as a workaround.

(This content is not included.DFBUGS-2823)

DR Status is now displayed for discovered apps on the Virtual machines page

The DR Status is now correctly displayed for discovered applications listed on the Virtual machines page. Manually adding the missing label to the DRPlacementControl and patching the PROTECTED_VMS recipe parameter is no longer required.

(This content is not included.DFBUGS-2822)

Secondary PVCs are now removed when DR protection is removed for discovered apps

On the secondary cluster, CephFS PVCs linked to a workload discovered using the Discovered Applications feature are now correctly marked as VRG-owned. As a result, when the workload is disabled, these PVCs are automatically cleaned up and no longer become orphaned, eliminating the need to manually delete them.

(This content is not included.DFBUGS-2827)

VolSync data mover pods can now access persistent volumes in environments with custom SCCs

Data replication operations no longer fail due to permission restrictions in clusters that enforce custom Security Context Constraints (SCCs). Previously, VolSync data mover pods lacked the necessary privileges to access underlying persistent volumes under restrictive security policies, causing synchronization to fail. This issue has been resolved by introducing a VolSyncSpec field to the DRPlacementControl (DRPC) specification, which allows users to explicitly define a custom MoverSecurityContext and MoverServiceAccount that are successfully propagated to the underlying components.

(This content is not included.DFBUGS-4728)

Custom s3StoreProfile configuration values are no longer lost after an upgrade

Upgrading the OpenShift Data Foundation operator no longer causes custom configurations to be wiped from the Ramen configuration. Previously, the Multicluster Orchestrator (MCO) operator would reconcile the ramen-hub-operator-config ConfigMap without accounting for existing custom values, leading to the loss of the CACertificates and veleroNamespaceSecretKeyRef fields under s3StoreProfiles. This issue has been resolved by updating the MCO operator to properly respect and preserve these custom values during upgrades.

(This content is not included.DFBUGS-440)

Sync no longer stops after PVC deselection

Sync operations now continue to function correctly when a PersistentVolumeClaim (PVC) is added to or removed from a group by modifying its label. Stale protected PVC entries are now properly cleaned up from the VolumeReplicationGroup (VRG) status, eliminating the need to manually edit the VRG’s status field as a workaround.

(This content is not included.DFBUGS-4012)

OSD disks are now properly zapped and cleaned during reinstallation

Disk zapping and cleanup logic now runs successfully during reinstallation even if the initial disk metadata offsets have been modified or cleared by external automation. Previously, the cleanup process relied on a mechanism that only checked the first location for BlueStore metadata; if it was missing, the process skipped zapping the rest of the disk, leaving remaining metadata intact. This issue has been resolved by migrating the disk cleanup logic directly into Rook and removing dependencies on external zapping commands.

(This content is not included.DFBUGS-5549)

Ceph monitors can now successfully form a quorum during a network partition between data centers

A network split between data sites no longer causes Ceph commands and I/O operations to hang indefinitely. Previously, when the two main data centers lost connectivity with each other but maintained communication with the Arbiter node, a flaw in the monitor election logic triggered an endless loop of leader elections. This rendered the Ceph cluster unavailable and required administrators to manually shut down monitors in one data center and reset connection scores as a workaround. This issue has been resolved by fixing the election logic to handle asymmetric network disruptions correctly.

(This content is not included.DFBUGS-425)

UI no longer shows "Unauthorized" error and temporary blank loading screen during ODF operator installation

During the OpenShift Data Foundation operator installation, the page now correctly handles instances where the InstallPlan transiently goes missing. The UI no longer displays an unknown status, "Unauthorized" error, or a temporary blank screen with missing titles and messages.

(This content is not included.DFBUGS-3574)

OpenShift Data Foundation console no longer incorrectly displays internal clusters as external

The OpenShift Data Foundation web console now accurately identifies and displays the cluster deployment mode. Previously, the console determined whether a cluster was internal or external based solely on the presence of the externalStorage.enable field rather than its actual boolean value, occasionally causing internal clusters to be misidentified as external. This issue has been resolved by verifying the exact boolean state of the field.

(This content is not included.DFBUGS-6242)

The kube-apiserver no longer logs 404 error messages originating from OpenShift Data Foundation

The kube-apiserver no longer generates frequent HTTP 404 (Not Found) error logs related to OpenShift Data Foundation internal requests or endpoints. This issue has been resolved by correcting the underlying missing or misconfigured resource requests between OpenShift Data Foundation components and the Kubernetes API server.

(This content is not included.DFBUGS-784)

odf-blackbox-exporter deployment no longer fails when upgrading

The odf-blackbox-exporter deployment now upgrades successfully without failure. Previously, when upgrading to certain OpenShift Data Foundation versions, the deployment would fail, requiring administrators to manually delete the deployment as a workaround. This issue has been resolved.

(This content is not included.DFBUGS-6789)

Encrypted OSDs are no longer unnecessarily redeployed during a 4.19.z upgrade

Upgrading legacy clusters (originally deployed on version 4.10 or earlier) to version 4.19 no longer triggers unnecessary OSD migrations and data movement. Previously, a checking mechanism failed to account for the legacy spec.encryption.enable setting, causing already-encrypted OSDs to be treated as unencrypted and migrated sequentially. This issue has been resolved by properly respecting the legacy encryption setting.

(This content is not included.DFBUGS-6696)

External clusters no longer upgrade OpenShift Data Foundation automatically without manual approval

When upgrading OpenShift Container Platform, external OpenShift Data Foundation clusters now correctly respect manual approval settings for installation plans. Previously, the ocs-operator did not set the disableInstallPlanAutoApproval key in the ocs-client-operator-config ConfigMap for external deployments, which caused the client-operator to automatically approve installation plans unexpectedly. This issue has been resolved by ensuring the key is set across all cluster types.

(This content is not included.DFBUGS-6043)

DRCluster validation can fail after upgrading to 4.22 when configured CIDRs are not detected by CSINodeAddons

In 4.22, CIDR validation for MDR fencing has been enhanced. The CIDRs configured in DRCluster.Spec.CIDRs are now validated against the CIDRs reported by CSINodeAddons on the cluster. As a result, any CIDRs that are not detected by CSINodeAddons cause DRCluster validation to fail with the following condition:

DRClusterValidated: False — undetected CIDRs specified <cidr1>, <cidr2>

CSINodeAddons reports the node IPs that are visible to the storage cluster. If DRCluster.Spec.CIDRs was configured by collecting CIDRs from all cluster nodes before upgrading, control plane node CIDRs might be present in the configuration. Because control plane nodes are typically not part of the storage network, their IPs are not reported by CSINodeAddons and can cause validation to fail after upgrading to 4.22.

Workaround: Verify that the CIDRs configured in DRCluster.Spec.CIDRs match the CIDRs detected by the cluster, either before or after upgrading to 4.22. Post-upgrade verification is recommended because the detected CIDRs are available directly.

Before upgrading to 4.22:

UI cleanup message identifies the wrong cluster after aborting a discovered application dryRun test failover

When a dryRun test failover for a discovered application is aborted, the failoverCluster field is removed from the DRPlacementControl (DRPC) specification while the DRPC status.phase remains FailedOver. The UI derives the cleanup target cluster by finding the complement of the primary cluster, but since getPrimaryClusterName() returns undefined (due to missing spec.failoverCluster), the cluster lookup defaults to the first cluster in the DRPolicy - which may be the original primary cluster instead of the actual failover target.

As a result, the UI cleanup message incorrectly directs the user to delete application resources from the original primary cluster instead of the failover target cluster where the dryRun workload was deployed. Acting on this incorrect guidance could result in deleting the production workload.

Workaround: Ignore the cluster name displayed in the UI cleanup message. Instead, identify the failover target cluster to which the dryRun test failover was triggered and delete the application resources from that cluster.

Result: After cleaning up the workload from the correct failover target cluster, the application will remain running only on the original primary managed cluster and data replication will resume, restoring the DR state for the application.

(This content is not included.DFBUGS-7998

UI does not reflect DRPC progression updates during ApplicationSet dryRun test failover abort

When a dryRun test failover for an ApplicationSet workload is aborted, the DRPC progression correctly transitions through Cleaning Up and then Completed. However, the user interface does not recognize or display these progression updates during the abort workflow.

As a result, the UI continues to display a stale state, such as TestingFailover or FailedOver, instead of the actual DRPC progression. This can give users an inaccurate view of the dryRun abort and cleanup status.

Workaround: Verify the actual DRPC progression by using the command line instead of relying on the UI status. For example, run oc get drpc <name> -n <namespace> -o yaml and review the DRPC progression details to confirm that the abort and cleanup have completed successfully.

Result: After the dryRun abort completes and the DRPC progression reaches Completed, the workload is removed from the failover cluster and normal disaster recovery protection resumes on the primary managed cluster.

(This content is not included.DFBUGS-8070)

CephFS VolSync PVCs can get stuck during disable DR

During disable DR for CephFS workloads protected by VolSync, Ramen deletes ReplicationSource (and related replication-group) objects without first removing their owner references from the PVCs, even when the disable-DR workflow requests that PVCs be preserved (do-not-delete-pvc). Kubernetes then marks the PVCs for deletion.

As a result, PersistentVolumeClaims can remain stuck in a Terminating state instead of staying Bound. Disable DR does not complete cleanly for affected CephFS deployments, and workloads might continue running while storage is left in an inconsistent lifecycle state.

Workaround: Do not disable DR for CephFS VolSync workloads until a fix is available.

If PVCs are already stuck in a Terminating state, recovery requires manual intervention and is not suitable for customer self-service. One manual recovery method is to use kubectl patch to remove the ReplicationSource and ReplicationDestination owner references from the PVCs during the disable-DR operation.

(This content is not included.DFBUGS-8196)

Ramen fails to detect updated SSL certificates for S3 connections

After an SSL certificate rotation on the S3 service, Ramen might not automatically detect newly added SSL certificates in the hub cluster. The Ramen pod continues to use the certificates that were loaded during startup, which can cause S3 connections to newly added clusters to fail.

This issue might surface as errors such as:

failed to list objects in bucket: code: RequestError, message: send request failed

Note that this error can occur due to various connection or access issues and should not always be interpreted as an SSL certificate problem.

Workaround: Manually restart the Ramen pod on the hub cluster to trigger reinitialization of the S3 connections. This process allows Ramen to load and use the updated SSL certificates.

If you are using managed clusters:

Manually restart the Ramen pods in the managed clusters after the certificate rotation is complete.

(This content is not included.DFBUGS-2520)

CephBlockPoolRadosNamespace remains in Progressing state for custom CephBlockPools in an RDR configuration

When a custom CephBlockPool is created on a cluster configured for Regional Disaster Recovery (RDR), the automatically created CephBlockPoolRadosNamespace resource (<pool-name>-builtin-implicit) can remain in the Progressing state indefinitely. This occurs because the MirroringController in the ocs-operator cannot enable mirroring on the CephBlockPool when the peer cluster bootstrap token is not yet available.

As a result, mirroring is not enabled on the CephBlockPool and the associated CephBlockPoolRadosNamespace does not transition to a healthy state.

Workaround: If the CephBlockPoolRadosNamespace remains in the Progressing state and mirroring is not enabled on the CephBlockPool, restart the ocs-operator pod.

Result: After the ocs-operator restarts, mirroring is enabled successfully on the CephBlockPool and the CephBlockPoolRadosNamespace transitions out of the Progressing state.

(This content is not included.DFBUGS-7981)

DRPCs protect all persistent volume claims created on the same namespace

The namespaces that host multiple disaster recovery (DR) protected workloads protect all the persistent volume claims (PVCs) within the namespace for each DRPlacementControl resource in the same namespace on the hub cluster that does not specify and isolate PVCs based on the workload using its spec.pvcSelector field.

This results in PVCs that match the DRPlacementControl spec.pvcSelector across multiple workloads. Or, if the selector is missing across all workloads, replication management to potentially manage each PVC multiple times and cause data corruption or invalid operations based on individual DRPlacementControl actions.

Workaround: Label PVCs that belong to a workload uniquely, and use the selected label as the DRPlacementControl spec.pvcSelector to disambiguate which DRPlacementControl protects and manages which subset of PVCs within a namespace. It is not possible to specify the spec.pvcSelector field for the DRPlacementControl using the user interface, hence the DRPlacementControl for such applications must be deleted and created using the command line.

Result: PVCs are no longer managed by multiple DRPlacementControl resources and do not cause any operation and data inconsistencies.

(This content is not included.DFBUGS-1749)

Disabled PeerReady flag prevents changing the action to Failover

The DR controller executes full reconciliation as and when needed. When a cluster becomes inaccessible, the DR controller performs a sanity check. If the workload is already relocated, this sanity check causes the PeerReady flag associated with the workload to be disabled, and the sanity check does not complete due to the cluster being offline. As a result, the disabled PeerReady flag prevents you from changing the action to Failover.

Workaround: Use the command-line interface to change the DR action to Failover despite the disabled PeerReady flag.

(This content is not included.DFBUGS-665)

For discovered apps with CephFS, sync stop after failover

For CephFS-based workloads, synchronization of discovered applications may stop at some point after a failover or relocation. This can occur with a Permission Denied error reported in the ReplicationSource status.

Workaround:

DRPC progression completes without workload cleanup when aborting a CephFS dryRun test failover

For a CephFS discovered application that is in the Deployed state before a dryRun test failover is triggered, aborting the dryRun can cause the DRPlacementControl (DRPC) progression to transition from TestingFailover to Completed even though the workload has not been cleaned up from the failover cluster (test cluster).

This results in the CephFS discovered application continuing to run on both managed clusters simultaneously, which is not a desirable state and can leave the application’s disaster recovery state inconsistent.

Workaround: Delete the application from the failover cluster (test cluster) where the dryRun test failover was triggered.

Result: After the application is successfully removed from the failover cluster, the workload continues running only on the primary managed cluster. Data replication resumes after the cleanup is completed, and the application’s disaster recovery state is restored.

(This content is not included.DFBUGS-8115)

Unable to create new OBCs using Multicloud Object Gateway

When provisioning an NSFS bucket via ObjectBucketClaim (OBC), the default filesystem path is expected to use the bucket name. However, if path is set in OBC.Spec.AdditionalConfig, it should take precedence. This behavior is currently inconsistent, resulting in failures when creating new OBCs.

(This content is not included.DFBUGS-3817)

RHCS 9.1z1 is not compatible with ODF 4.22.0

Customers running ODF 4.22.0 in external mode should not deploy or upgrade to RHCS 9.1z1 due to an incompatibility issue. A fix will be provided in ODF 4.22.z.

OSD pods restart during add capacity

OSD pods restart after performing cluster expansion by adding capacity to the cluster. However, no impact to the cluster is observed apart from pod restarting.

(This content is not included.DFBUGS-1426)

SELinux relabelling issue with a very high number of files

When attaching volumes to pods in Red Hat OpenShift Container Platform, the pods sometimes do not start or take an excessive amount of time to start. This behavior is generic and it is tied to how SELinux relabelling is handled by Kubelet. This issue is observed with any filesystem based volumes having very high file counts. In OpenShift Data Foundation, the issue is seen when using CephFS based volumes with a very high number of files. There are multiple ways to work around this issue. Depending on your business needs you can choose one of the workarounds from the knowledgebase solution https://access.redhat.com/solutions/6221251.

(This content is not included.RFE-3327)

UI shows WaitOnUserCleanUp even when automatic cleanup is enabled

The UI incorrectly displays the WaitOnUserCleanUp status even when automatic cleanup is enabled for VMs. This occurs because the UI relies only on the phase and progression fields of the DRPlacementControl to determine cleanup behavior and does not evaluate the more granular AutoCleanup condition that explicitly indicates automatic cleanup.

Workaround: There is no manual workaround required. This state is transient and clears automatically once the progression field advances to Completed. Manual cleanup should be avoided unless the AutoCleanup condition and its corresponding reason in the DRPlacementControl or VRG status indicate otherwise.

During automatic cleanup, the UI may briefly present a misleading status, which can cause temporary confusion until the cleanup completes.

(This content is not included.DFBUGS-5824)

DRPlacementControl shows ProtectionError even after successful relocation

When a relocation completes, the DRPlacementControl may continue to display a ProtectionError status. This occurs because the Protected condition in the DRPlacementControl status incorrectly reports an Error state, even though the relocation has finished (phase: Relocated, progression: Completed).

Workaround: No direct workaround is available. Wait until retrying the NoClusterDataConflict condition is met.

The DR status in the UI remains in the ProtectionError state until the data conflict is resolved.

(This content is not included.DFBUGS-5823)