Enable Regional Disaster Recovery with CephFS Change File Tracking Based Replication in OpenShift Data Foundation 4.22 (Developer Preview)

Updated

Regional Disaster Recovery (RDR) for CephFS volumes uses VolSync to copy data to a secondary site. The traditional method works like this: every time replication runs, it rescans the entire filesystem — every file and every block — to figure out what changed. This works fine for small volumes, but as storage grows, rescanning everything becomes slow and consumes significant resources, even when only a tiny fraction of the data has actually changed.

CephFS Change File Tracking (CFT) solves this by having Ceph natively track what changes between snapshots. Instead of rescanning everything, Ceph tells replication exactly which files and which blocks were modified — so the sync process only copies what actually changed.

For environments with large CephFS volumes and tight recovery time targets, CFT-based replication can cut sync times from hours to minutes, making disaster recovery at scale practical.

Dev Preview: This feature is available as a Developer Preview in ODF 4.22. It is not intended for production use and is subject to change in future releases.

Prerequisites

  • OpenShift Container Platform 4.22 or later on both managed clusters
  • OpenShift Data Foundation (ODF) 4.22 or later with CephFS storage configured
  • Red Hat Advanced Cluster Management (ACM) Hub cluster configured
  • Regional DR (RDR) configured between primary and secondary clusters

Steps to Enable CephFS CFT-Based Replication

Step 1: Install OpenShift DR VolSync Plugin Operator

Install the OpenShift DR VolSync Plugin Operator on both the primary and secondary managed clusters. This operator replaces VolSync's default rsync-based mover with a Ceph-native mover that leverages CFT for incremental replication.

  1. Navigate to Ecosystem → Software Catalog on each managed cluster

  2. Search for OpenShift DR VolSync Plugin Operator

  3. Install with default settings (it installs into the openshift-dr-system namespace)

  4. Verify the operator is in Succeeded state:

    $ oc get csv -n openshift-dr-system | grep odr-volsync-plugin

    Example output:

    odr-volsync-plugin-operator.v4.22.0-97.stable   OpenShift DR VolSync Plugin Operator   4.22.0-97.stable                      Succeeded

  5. Verify the controller pod is running:

    $ oc get pods -n openshift-dr-system -l app.kubernetes.io/name=ceph-volsync-plugin-operator

    Example output:

    NAME                                                              READY   STATUS    RESTARTS   AGE
ceph-volsync-plugin-operator-controller-manager-5b4dd97b45zzkcn   1/1     Running   0          22h

Repeat on both managed clusters.

Step 2: Protect the application

Deploy an application using a CephFS-backed PVC on the primary managed cluster. Then, using the ACM Hub console, protect the application. The DRPC, that is created, defines the disaster recovery policy including the primary and secondary clusters, the replication schedule, and the PVC label selector.

Ensure the application's PVC uses a CephFS StorageClass (e.g., ocs-storagecluster-cephfs) and has a label that the DRPC's pvcSelector can match.

  1. Verify the DRPC is deployed:

    $ oc get drpc -n openshift-dr-ops
NAME                        AGE    PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE
dr-volsync-plugin-test      30m    cluster-1                                           Deployed

  2. Verify ReplicationSource and ReplicationDestination are created: Wait for the LAST SYNC timestamp to appear, confirming the first replication cycle has completed.

        # On the primary managed cluster (cluster-1)
    $ oc get replicationsource -n <app-namespace>
    NAME         SOURCE          LAST SYNC              DURATION          NEXT SYNC
    cephfs-pvc   vs-cephfs-pvc   2026-06-17T09:01:52Z   1m48.412060946s

    # On the secondary managed cluster (cluster-2)
    $ oc get replicationdestination -n <app-namespace>
    NAME         LAST SYNC              DURATION          NEXT SYNC
    cephfs-pvc   2026-06-17T09:01:52Z   2m47.120971867s

  3. Check mover pod on the secondary cluster:

    # On the secondary managed cluster (cluster-2)
$ oc get pods -n <app-namespace> | grep mover
volsync-rsync-tls-dst-cephfs-pvc-7k2x9   1/1     Running   0          45s

Note: At this point, the DRPC is using the default VolSync rsyncTLS-based mover. The next step enables CFT-based replication by adding the diff annotation. The outputs shown above are examples; actual names and values will differ in your environment.

Step 3: Add the CFT Diff Annotation to DRPC

To enable CFT-based replication, add the annotation drplacementcontrol.ramendr.openshift.io/enable-diff="true" to the DRPC resource on the Hub cluster. This can be done through the ACM Hub console by editing the DRPC resource annotations, or manually with the following command:

$ oc annotate drpc <drpc-name> -n <namespace> \
  drplacementcontrol.ramendr.openshift.io/enable-diff="true"

This annotation instructs the DR stack to use the OpenShift DR VolSync Plugin Operator's Ceph-native mover instead of VolSync's default rsyncTLS mover. The Ceph-native mover uses CFT to identify changed files between snapshots and block-level diffs within those files for efficient incremental transfer.

Step 4: Verify CFT-Based Replication Is Working

Wait for the next replication cycle to complete after adding the annotation. The LAST SYNC timestamp on ReplicationSource and ReplicationDestination should update to a time after the annotation was added.

  1. Verify the DRPC status:

    $ oc get drpc -n openshift-dr-ops
NAME                        AGE    PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE
dr-volsync-plugin-test      60m    cluster-1                                           Deployed

    The DRPC should remain in Deployed state.

  2. Verify ReplicationSource and ReplicationDestination: Confirm the LAST SYNC timestamp has updated since adding the annotation.

        # On the primary managed cluster (cluster-1)
    $ oc get replicationsource -n <app-namespace>
    NAME         SOURCE          LAST SYNC              DURATION        NEXT SYNC
    cephfs-pvc   vs-cephfs-pvc   2026-06-17T09:24:25Z   20.948364914s

    # On the secondary managed cluster (cluster-2)
    $ oc get replicationdestination -n <app-namespace>
    NAME         LAST SYNC              DURATION          NEXT SYNC
    cephfs-pvc   2026-06-17T09:21:27Z   1m22.591619661s

  3. Check mover pods: After adding the annotation, mover pod names change from volsync-rsync-tls-* to ceph-volsync-cephfs-*, confirming CFT-based replication is active.

    # On the secondary managed cluster (cluster-2)
$ oc get pods -n <app-namespace> | grep mover
ceph-volsync-cephfs-dst-cephfs-pvc-mh2pj   1/1     Running   0          39s

Reverting to Default VolSync Rsync Replication

To disable CFT-based replication and revert to the default VolSync rsyncTLS-based mover, remove the annotation from the DRPC:

$ oc annotate drpc <drpc-name> -n <namespace> \
  drplacementcontrol.ramendr.openshift.io/enable-diff-

After removal, the next replication cycle will use the default VolSync rsyncTLS-based mover. Mover pod names will revert from ceph-volsync-cephfs-* back to volsync-rsync-tls-*.

How Incremental Replication Works

Once CFT-based replication is enabled, all subsequent sync cycles are incremental. Unlike rsync, which must rescan the entire filesystem on every cycle, the Ceph-native mover only transfers data that actually changed between the previous and current snapshots:

  • Snap Diff: Ceph's Change File Tracking identifies exactly which files were created, modified, or deleted between two snapshots.
  • Block Diff: For modified files, only the changed byte ranges are identified and transferred, not entire files.
  • Reduced cross-cluster transfer: Only changed blocks are sent over the network between clusters, significantly reducing bandwidth consumption and replication time.
  • Lower resource usage: Mover pods consume less CPU and memory since they skip the expensive filesystem scan and full-file comparison phases.
  • Tighter RPOs: Faster sync cycles mean replication schedules can be set more aggressively, enabling recovery point objectives that were impractical with rsyncTLS-based replication on large volumes.
  • Faster failover recovery: On failover, the destination volume is incrementally synced to the last replicated snapshot instead of rsync's full filesystem scan, leading to faster application recovery on the secondary cluster.

As a result, the replication duration remains proportional to the amount of data changed rather than the total volume size, making disaster recovery scalable for large CephFS volumes.

Category
Components
Tags
Article Type