Failed to configure NFS for the internal image-registry storage in OpenShift

Solution Unverified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • UPI install
  • VMware vSphere

Issue

  • When attempting to utilize an NFS share for image-registry storage, the PVC created by the image registry operator fails with an error such as:

    Failed to provision volume with StorageClass "thin": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce] are supported

Resolution

Based on the information in is NFS supported for OpenShift cluster internal components in Production, while it is supported to use NFS V4 as the storage provider, there are known issues and it is not recommended to use NFS as the storage provider for the internal image registry in OpenShift.

Workaround

To work around the invalid AccessMode issue, even if it is not recommended to use NFS as the storage provider for the internal image registry, the following procedure can be followed:

1. Delete the PV and PVC

timeout 2 oc delete pv pv-name
timeout 2 oc delete pvc image-registry-storage -n openshift-image-registry

2. Delete image-registry pods

oc get pods -n openshift-image-registry

# for each pod with a name beginning with image-registry- , run:
timeout 2 oc delete pod image-registry-some-random-numbers -n openshift-image-registry

3. Create a new storage class which does not have a configured access mode.

oc create -f - <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: non-dynamic 
provisioner: no-provisioning 
parameters:
EOF

4. Disable the "thin" storage class as the default storageclass

oc patch storageclass thin -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

5. Enable the "non-dynamic" storage class as the default storageclass

oc patch storageclass non-dynamic -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

6. Create a new PV which references the new storage class.

oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteMany
  nfs:
    path: /server
    server: 10.x.x.138
  persistentVolumeReclaimPolicy: Retain
  storageClassName: non-dynamic
EOF

7. Create a new image-registry PVC which references the custom storage class.

oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  name: image-registry-storage
  namespace: openshift-image-registry
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
EOF

Root Cause

On a vSphere cluster, the default storage class is thin which has an access mode of ReadWriteOnly. The PVC created by the image registry does not define a storage class and as a result, the default storage class is used. The image registry PVC is configured to have an access mode of ReadWriteMany.

Diagnostic Steps

Check the state of the image registry PVC by running:

oc describe pvc -A

The image-registry-storage PVC should be in a pending state with an output similar to the following.

Name:          image-registry-storage
Namespace:     openshift-image-registry
StorageClass:  thin
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type       Reason              Age                  From                         Message
  ----       ------              ----                 ----                         -------
  Warning    ProvisioningFailed  5s (x11 over 8m18s)  persistentvolume-controller  Failed to provision volume with StorageClass "thin": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce] are supported
Mounted By:  image-registry-7dcbf9cd8b-xdx6l
             image-registry-89b9688bd-h54sz
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.