RHCS 6 - BLUESTORE_NO_PER_POOL_OMAP OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats

Solution Verified - Updated

Environment

  • Red Hat Ceph Storage (RHCS)
    • 6 (when updating from RHCS 5)
  • Red Hat Openshift Data Foundation (RHODF)
    • 4.13 (when updating from RHODF 4.12)

Issue

  • After upgrade from Red Hat Ceph Storage 5 to version 6, or from Red Hat Openshift Data Foundation 4.12 to version 4.13 the following errors are observed:

    $ ceph health detail
HEALTH_WARN 5 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats
[WRN] BLUESTORE_NO_PER_POOL_OMAP: 5 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats
osd.0 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.2 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.3 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.4 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.5 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics

  • In the Red Hat Openshift Data Foundation 4.13 web console the following error is shown:

    x OSD(s) reporting legacy (not per-pool) bluestore omap usage stats

Resolution

NOTE: The following procedure will NOT cause any data deletion

The "bluestore_fsck_quick_fix_on_mount=true" will trigger omap fix run to adjust to new version, the time it takes to finish depends on the size of the OMAP on the individual OSDs and its disk performance, if the OMAP size in GB it is recommended to also set noout flag in order to avoid OSDs to be set out after 600 seconds being in down state and trigger data rebalance.

For Red Hat Ceph Storage (RHCS) 6

  1. Set the following value:
$ ceph osd set noout
$ ceph config set osd bluestore_fsck_quick_fix_on_mount true
  1. Restart all OSDs - Note: one node at a time and always wait for all PGs to be active+clean before restarting OSDs on another node
$ ceph -s
  1. After restart of all OSDs, remove the configuration:
$ ceph config rm osd bluestore_fsck_quick_fix_on_mount
$ ceph osd unset noout

For Red Hat Openshift Data Foundation (RHODF) 4.13

  1. Enable the Rook-Ceph Toolbox by following the article Configuring the Rook-Ceph Toolbox in OpenShift Data Foundation 4.x and rsh connect to the Pod
$ oc -n openshift-storage rsh "$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)"
  1. Once connected to the Rook-Ceph Toolbox Pod, set the following option:
$ ceph osd set noout
$ ceph config set osd bluestore_fsck_quick_fix_on_mount true # This should be issued inside the Rook-Ceph Toolbox Pod
  1. Restart all OSDs by deleting their Pods. This will trigger a Pod restart - Note: one Pod at a time and always wait for all PGs to be active+clean before restarting any other OSDs from any other Pod/Node
$ oc delete pod/<OSD-PODNAME>
$ oc -n openshift-storage rsh "$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)"
$ ceph -s # This should be issued inside the Rook-Ceph Toolbox Pod
  1. Once it is fixed for all OSDs, remove the config it via the Rook-Ceph Toolbox
$ oc -n openshift-storage rsh "$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)"
$ ceph config rm osd bluestore_fsck_quick_fix_on_mount # This should be issued inside the Rook-Ceph Toolbox Pod
$ ceph osd unset noout

Root Cause

One or more OSDs have volumes that were created prior to the Octopus release. In Octopus and later releases, BlueStore tracks omap space utilization by pool.
If there are any BlueStore OSDs that do not have the new tracking enabled, the cluster will report an approximate value for per-pool omap usage based on the most recent deep scrub.
The OSDs can be updated to track by pool by stopping each OSD, running a repair operation, and then restarting the OSD.
Please check the What are the Red Hat Ceph Storage releases and corresponding Ceph package versions? article to track down Red Hat Container Storage vs Ceph Upstream versioning.

Diagnostic Steps

  • By issuing the command ceph -s the following errors appear:

    x OSD(s) reporting legacy (not per-pool) bluestore omap usage stats

  • By issuing the command ceph health detail the following appear:

    $ ceph health detail
HEALTH_WARN 5 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats
[WRN] BLUESTORE_NO_PER_POOL_OMAP: 5 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats
osd.0 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.2 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.3 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.4 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
osd.5 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.