How do I backup and restore galera clusters in Red Hat OpenStack Services on Openshift

Solution Verified - Updated

Environment

Red Hat OpenStack Services on Openshift

Issue

As an operator I need to backup the contents of all the galera clusters that are in use in the Openstack control plane.

Resolution

In Red Hat OpenStack Services on OpenShift (RHOSO) 18, several galera clusters can be deployed on the OpenStack control plane, each cluster serving a dedicated set of OpenStack services. For instance, one galera cluster could process traffic for a specific nova cell, while another cluster can process traffic from all the other OpenStack services, such as keystone, cinder, etc.

As an operator, backing up the state of the galera clusters consists of taking a different backup of every galera cluster configured in your OpenStack control plane. You can run the following command to list all the galera clusters deployed in your control plane:

oc get galera --no-headers -o custom-columns=":metadata.name"

Backing up OpenStack data from a galera cluster

To backup the state of the galera cluster, you select a galera pod and back up all OpenStack databases and tables present in the mysql server running on this pod.

This procedure attempts to select a galera pod that is currently not serving traffic to limit any risks of impacting service during the backup.
The backup file is saved on your workstation in the directory in which you perform this procedure.

Procedure

  1. Use the cd command to change your current directory to the directory on your workstation where you intend to save the backup.

  2. Specify the name of the galera cluster to backup:

    DBNAME=<galera-cluster>

    Replace <galera-cluster> with the name of your galera cluster, for example, openstack.

  3. Locate the galera pod on this galera cluster that is currently serving traffic:

    SVC_ENDPOINT=$(oc get svc $DBNAME -o json | jq -r '.spec.selector["statefulset.kubernetes.io/pod-name"]')

  4. List the clustered pods that are currently ready to provide backups:

    ALL_PODS=$(oc exec -q -c galera $SVC_ENDPOINT -- bash -c "mysql -nNE -uroot -p\$DB_ROOT_PASSWORD -e \"show status like 'wsrep_incoming_addresses';\"" | tail -1 | tr ',' '\n' | sed 's/\..*//')

  5. Select the target pod that is not currently serving traffic:

    BACKUP_POD=$(echo "$ALL_PODS" | grep -v $SVC_ENDPOINT | head -1)
BACKUP_POD=${BACKUP_POD:-$DBNAME-galera-0}

  6. Retrieve the OpenStack data to backup:

    BACKUP_DBS=$(oc exec -q -c galera $BACKUP_POD -- bash -c "mysql -uroot -p\$DB_ROOT_PASSWORD -sN -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\"")

  7. Backup the state of the galera cluster on your workstation:

    BACKUPDATE=$(date -u +'%Y-%m-%d_%H-%M-%S')
for db in $BACKUP_DBS; do echo $db; oc exec -q -c galera $BACKUP_POD -- bash -c "mysqldump -uroot -p\$DB_ROOT_PASSWORD --single-transaction --databases $db" >> $DBNAME-databases-$BACKUPDATE.sql ;done

  8. If the backup is successful, a backup file is saved on your workstation in the current directory. Each backup file preserves the current timestamp, for example openstack-databases-2025-10-24_10-02-24.sql.

Restore OpenStack data in a galera cluster

To restore a backup of the OpenStack data to a running galera cluster, you must scale the galera cluster down to 1 pod and disable the database traffic when the data is being restored. When the data is restored you must enable the database traffic and scale the galera cluster back to 3 pods. Scaling the galera cluster requires updating the existing OpenStackControlPlane custom resource (CR).

NOTE: You must perform a full password rotation for every Openstack service declared in the database to restore their OpenStack credentials. See Restore OpenStack credentials in a galera cluster.

Procedure

  1. Use the cd command to change your current directory to the directory on your workstation in which you have previously saved your galera cluster backups.

  2. Specify the name of the galera cluster to restore:

    DBNAME=<galera-cluster>

    Replace <galera-cluster> with the name of your galera cluster, for example, openstack.

  3. Specify the openstack control plane resource that manages the galera cluster:

    CONTROLPLANE=<control-plane-cr>

    Replace <control-plane-cr> with the name of your existing OpenStackControlPlane CR, for example, openstack-control-plane.

  4. Specify the name of the galera backup file you want to restore:

    BACKUP_FILE=openstack-databases-2025-10-24_10-02-24.sql

  5. Scale the galera cluster to 1 pod:

    oc patch oscp $CONTROLPLANE --type merge  -p '{"spec":{"galera":{"templates":{"'$DBNAME'":{"replicas":1}}}}}'
oc wait sts $DBNAME-galera --for=jsonpath='{.status.replicas}'=1
oc wait sts $DBNAME-galera --for=jsonpath='{.status.availableReplicas}'=1

  6. Disable the database traffic to this galera cluster during the restore:

    oc patch svc $DBNAME --type=merge -p '{"spec":{"selector":{"statefulset.kubernetes.io/pod-name": ""}}}'

  7. Optional, if you updated the table schemas or created additional databases since the backup was taken, you may want to drop all the existing Openstack databases prior to restoring the backup:

    oc exec -q -c galera $DBNAME-galera-0 -- bash -c "mysql -uroot -p\$DB_ROOT_PASSWORD -sN -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs -r -n1 mysqladmin -uroot -p\$DB_ROOT_PASSWORD -f drop"

  8. Restore the data:

    oc exec -q -i -c galera $DBNAME-galera-0 -- bash -c "mysql -uroot -p\$DB_ROOT_PASSWORD" < $BACKUP_FILE

  9. Verify that the databases are present in the galera pod:

    oc exec -q -it -c galera $DBNAME-galera-0 -- bash -c "mysql -uroot -p\$DB_ROOT_PASSWORD -e \"show databases;\""

  10. Enable the database traffic to this galera cluster:

oc patch svc $DBNAME --type=merge -p '{"spec":{"selector":{"statefulset.kubernetes.io/pod-name": "'$DBNAME-galera-0'"}}}'
  1. Scale the galera cluster back to 3 pods:
oc patch oscp $CONTROLPLANE --type merge  -p '{"spec":{"galera":{"templates":{"'$DBNAME'":{"replicas":3}}}}}'
oc wait sts $DBNAME-galera --for=jsonpath='{.status.availableReplicas}'=3

Restore OpenStack credentials in a galera cluster

You must perform a full password rotation for every Openstack service declared in the database to restore their OpenStack credentials. In RHOSO 18, there is one MariaDBAccount CR for the Openstack user defined in a galera cluster.

This is the high-level password rotation workflow for a galera cluster:

  1. Create a new MariaDBAccount CR for each Openstack service, to automatically create new credentials in the database, without deleting the existing ones.

  2. For each service defined in the control plane, reference the name of the new MariaDBAccount in the service’s CR. This automatically triggers a rolling restart of the Openstack service to prevent service disruption.

  3. When all services have been restarted, delete the old MariaDBAccount CRs.

For the exact steps to perform, see: This content is not included.This content is not included.https://issues.redhat.com/browse/RHOSSTRAT-1049

SBR
Product(s)
Components

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.