Workaround to skip SELinux relabelling issues in Openshift Data Foundation / Openshift Container Storage

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Container Storage (RHOCS)
  • 4
• Red Hat OpenShift Data Foundation (RHODF)
  • 4

Issue

• When the number of files in these PVs grows largely (thousands/millions of objects, depending on each case and size), the pods get stuck in ContainerCreating or init status.

• OpenShift namespaces are by default, configured to have different Multi-Category Security (MCS) SELinux settings. A simple way to review this security context is by inspecting a namespace yaml:

     oc get project mds-test -o yaml | grep scc.mcs
        openshift.io/sa.scc.mcs: s0:c25,c20
              f:openshift.io/sa.scc.mcs: {}
  

• This SELinux context is inherited by all the pods running inside these namespaces. Because of this feature, all the files inside PVs bound to these pods need to be relabeled accordingly to ensure the SELinux context matches the security specs in the pods.

Resolution

NOTE: This solution will ONLY cover new volumes created using the Storage Class from the mitigation below. The new volume mitigation creates a Storage Class to address any future volumes that will be created using CephFS with that specific Storage Class. If the volumes have already been provisioned, please refer to pods using Persistent Volumes with high file counts fail to start or take an excessive amount of time in OpenShift solution to reconcile the issue. If assistance is needed to execute the referenced solution, please open a Red Hat Support case with the platform team (OCP/Shift).

Mitigation for new Volumes

Note: the sample yaml below has the kernelMountOptions: context="system_u:object_r:container_file_t:s0" defined. This is a universal option that will work for any namespace inheriting the already defined context in that namespace.

Create an additional Storage Class, with the security context set to the required values, using the variable kernelMountOptions. Sample yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ocs-storagecluster-cephfs-selinux-relabel
provisioner: openshift-storage.cephfs.csi.ceph.com
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  fsName: ocs-storagecluster-cephfilesystem
  kernelMountOptions: context="system_u:object_r:container_file_t:s0"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

New volumes will need to be created using the above Storage Class.

Root Cause

Refer to pods using Persistent Volumes with high file counts fail to start or take an excessive amount of time in OpenShift for additional information about the cause of the issue for this workaround.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.