Support for RGW Erasure Coding in Internal Mode - Developer preview OpenShift Data Foundation 4.16
Important: A developer preview feature is subject to Developer preview support limitations. Developer preview features are not intended to be run in production environments. The clusters deployed with the developer preview features are considered to be development clusters and are not supported through the Red Hat Customer Portal case management system. Development Preview features are meant for customers who are willing to evaluate new products or releases of products in an early stage of product development. If you need assistance with developer preview features, reach out to the ocs-devpreview@redhat.com mailing list and a member of the Red Hat Development Team will assist you as quickly as possible based on availability and work schedules. To know more about the support scope refer to the This content is not included.KCS
NOTE: This feature is tentatively planned to be GA'd in Content from redhat.atlassian.net is not included.ODF 4.22 subject to any change.
Support erasure coding deployment for RGW in OpenShift Data Foundation using CLI
Procedure
-
Decide the values for dataChunks (k) and codingChunks (m).
The supported values are:
i) k=8 m=3
ii) k=8 m=4
iii) k=4 m=2
Minimum number of worker nodes required will be the sum of dataChunks and codingChunks (k+m)
For more information, see Red Hat Ceph Storage: Supported configurations -
Create a new Cephobjectstore with datapool using ec spec by using the following
object-ec.yaml:
# Object-ec.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: ocs-storagecluster-cephobjectstore-ec
namespace: openshift-storage
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
dataPool:
failureDomain: host
erasureCoded:
dataChunks: 4 # update the value of k determined in step 1
codingChunks: 2 # update the value of m determined in step 1
preservePoolsOnDelete: true
gateway:
port: 80
instances: 1
Note: Metadata pool supports only replicated pool.
- Create the new RGW storageclass.
rgw-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-retain-bucket
provisioner: openshift-storage.ceph.rook.io/bucket
reclaimPolicy: Retain
parameters:
objectStoreName: ocs-storagecluster-cephobjectstore-ec
objectStoreNamespace: openshift-storage
-
Validate the cluster with newly created resources
i) Get thecephobjectstore.``` $ oc get cephobjectstore -nopenshift-storage NAME PHASE ocs-storagecluster-cephobjectstore-ec Ready ```ii) check the erasure code RGW pools.
``` $ Ceph osd pool ls ocs-storagecluster-cephobjectstore-ec.rgw.log ocs-storagecluster-cephobjectstore-ec.rgw.buckets.non-ec ocs-storagecluster-cephobjectstore-ec.rgw.otp ocs-storagecluster-cephobjectstore-ec.rgw.control ocs-storagecluster-cephobjectstore-ec.rgw.buckets.index ocs-storagecluster-cephobjectstore-ecrgw.meta ocs-storagecluster-cephobjectstore-ec.rgw.buckets.data ```Detail view of
ocs-storagecluster-cephobjectstore-ec.rgw.buckets.data``` pool 19 'ocs-storagecluster-cephobjectstore-ec.rgw.buckets.data' erasure profile ocs-storagecluster- cephobjectstore-ec.rgw.buckets.data_ecprofile size 6 min_size 5 crush_rule 36 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 172 lfor 0/0/165 flags hashpspool,ec_overwrites stripe_width 16384 compression_mode none application rook-ceph-rgw ```iii) Check the
ec rgw crush_rule.``` ocs-storagecluster-cephobjectstore-ec.rgw.buckets.data { "rule_id": 36, "rule_name": "ocs-storagecluster-cephobjectstore-ec.rgw.buckets.data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ```iv) Get the new storageclass.
``` $ oc get sc -nopenshift-storage rook-ceph-retain-bucket openshift-storage.ceph.rook.io/bucket Retain Immediate false 101m ``` -
Consume the Storage.
- Create the objectBucketClaim
# Rgw-obc.yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-retain-bucket
spec:
generateBucketName: ceph-bkt
storageClassName: rook-ceph-retain-bucket
$ oc get objectbucketclaim -nopenshift-storage
NAME STORAGE-CLASS PHASE AGE
ceph-retain-bucket rook-ceph-retain-bucket Bound 79m
This creates the ObjectBucket too.
$ kubectl get objectbucket -nopenshift-storage
NAME STORAGE-CLASS CLAIM-NAMESPACE CLAIM-NAME RECLAIM-POLICY PHASE AGE
obc-openshift-storage-ceph-retain-bucket rook-ceph-retain-bucket Retain Bound 52m
After this is completed, you can use this bucket with your application pod.
Verifying by adding data
You can use the guidelines provided in [Consume the Object Storage](https://rook.io/docs/rook/latest-release/Storage-Configuration/Object-Storage-RGW/object-storage/#consume-the-object-storage).
In this example, k=4 and m=2 for dataPool configuration.
- Store a 5 GB file in it and notice that only 8 Gb is needed to store it.
Before storing 5 GB file
sh-4.4$ s5cmd --endpoint-url http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore-ec.openshift-storage.svc:80 du s3://ceph-bkt-38dff5f2-a26c-46fc-9141-da5c94cd3d6f
33 bytes in 3 objects: s3://ceph-bkt-38dff5f2-a26c-46fc-9141-da5c94cd3d6f
sh-4.4$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 0.58612 - 600 GiB 10 GiB 10 GiB 0 B 424 MiB 590 GiB 1.75 1.00 - root default
-8 0.19537 - 200 GiB 3.5 GiB 3.4 GiB 0 B 140 MiB 197 GiB 1.75 1.00 - rack rack0
-7 0.09769 - 100 GiB 2.1 GiB 2.0 GiB 0 B 97 MiB 98 GiB 2.13 1.22 - host ocs-deviceset-1-data-0m2v4c
1 ssd 0.09769 1.00000 100 GiB 2.1 GiB 2.0 GiB 0 B 97 MiB 98 GiB 2.13 1.22 133 up osd.1
-19 0.09769 - 100 GiB 1.4 GiB 1.3 GiB 0 B 43 MiB 99 GiB 1.36 0.78 - host ocs-deviceset-1-data-1m2xbc
5 ssd 0.09769 1.00000 100 GiB 1.4 GiB 1.3 GiB 0 B 43 MiB 99 GiB 1.36 0.78 145 up osd.5
-4 0.19537 - 200 GiB 3.5 GiB 3.4 GiB 0 B 146 MiB 196 GiB 1.75 1.00 - rack rack1
-15 0.09769 - 100 GiB 2.1 GiB 2.0 GiB 0 B 54 MiB 98 GiB 2.06 1.18 - host ocs-deviceset-0-data-1658wk
4 ssd 0.09769 1.00000 100 GiB 2.1 GiB 2.0 GiB 0 B 54 MiB 98 GiB 2.06 1.18 142 up osd.4
-3 0.09769 - 100 GiB 1.4 GiB 1.4 GiB 0 B 93 MiB 99 GiB 1.44 0.82 - host ocs-deviceset-2-data-0v6x24
0 ssd 0.09769 1.00000 100 GiB 1.4 GiB 1.4 GiB 0 B 93 MiB 99 GiB 1.44 0.82 142 up osd.0
-12 0.19537 - 200 GiB 3.5 GiB 3.4 GiB 0 B 137 MiB 197 GiB 1.75 1.00 - rack rack2
-11 0.09769 - 100 GiB 1.3 GiB 1.3 GiB 0 B 78 MiB 99 GiB 1.33 0.76 - host ocs-deviceset-0-data-0tfv2m
2 ssd 0.09769 1.00000 100 GiB 1.3 GiB 1.3 GiB 0 B 78 MiB 99 GiB 1.33 0.76 144 up osd.2
-17 0.09769 - 100 GiB 2.2 GiB 2.1 GiB 0 B 59 MiB 98 GiB 2.17 1.24 - host ocs-deviceset-2-data-1cwfjp
3 ssd 0.09769 1.00000 100 GiB 2.2 GiB 2.1 GiB 0 B 59 MiB 98 GiB 2.17 1.24 137 up osd.3
TOTAL 600 GiB 10 GiB 10 GiB 0 B 424 MiB 590 GiB 1.75
MIN/MAX VAR: 0.76/1.24 STDDEV: 0.37
After storing a 5GB file,
sh-4.4$ s5cmd --endpoint-url http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore-ec.openshift-storage.svc:80 du s3://ceph-bkt-38dff5f2-a26c-46fc-9141-da5c94cd3d6f
5368709153 bytes in 4 objects: s3://ceph-bkt-38dff5f2-a26c-46fc-9141-da5c94cd3d6f
sh-4.4$
sh-4.4$
sh-4.4$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 0.58612 - 600 GiB 18 GiB 18 GiB 0 B 460 MiB 582 GiB 3.03 1.00 - root default
-8 0.19537 - 200 GiB 6.0 GiB 5.9 GiB 0 B 147 MiB 194 GiB 3.02 1.00 - rack rack0
-7 0.09769 - 100 GiB 3.4 GiB 3.3 GiB 0 B 101 MiB 97 GiB 3.41 1.13 - host ocs-deviceset-1-data-0m2v4c
1 ssd 0.09769 1.00000 100 GiB 3.4 GiB 3.3 GiB 0 B 101 MiB 97 GiB 3.41 1.13 133 up osd.1
-19 0.09769 - 100 GiB 2.6 GiB 2.6 GiB 0 B 46 MiB 97 GiB 2.63 0.87 - host ocs-deviceset-1-data-1m2xbc
5 ssd 0.09769 1.00000 100 GiB 2.6 GiB 2.6 GiB 0 B 46 MiB 97 GiB 2.63 0.87 145 up osd.5
-4 0.19537 - 200 GiB 6.1 GiB 5.9 GiB 0 B 154 MiB 194 GiB 3.03 1.00 - rack rack1
-15 0.09769 - 100 GiB 3.3 GiB 3.3 GiB 0 B 58 MiB 97 GiB 3.34 1.10 - host ocs-deviceset-0-data-1658wk
4 ssd 0.09769 1.00000 100 GiB 3.3 GiB 3.3 GiB 0 B 58 MiB 97 GiB 3.34 1.10 142 up osd.4
-3 0.09769 - 100 GiB 2.7 GiB 2.6 GiB 0 B 96 MiB 97 GiB 2.71 0.90 - host ocs-deviceset-2-data-0v6x24
0 ssd 0.09769 1.00000 100 GiB 2.7 GiB 2.6 GiB 0 B 96 MiB 97 GiB 2.71 0.90 142 up osd.0
-12 0.19537 - 200 GiB 6.1 GiB 5.9 GiB 0 B 160 MiB 194 GiB 3.03 1.00 - rack rack2
-11 0.09769 - 100 GiB 2.6 GiB 2.5 GiB 0 B 97 MiB 97 GiB 2.62 0.87 - host ocs-deviceset-0-data-0tfv2m
2 ssd 0.09769 1.00000 100 GiB 2.6 GiB 2.5 GiB 0 B 97 MiB 97 GiB 2.62 0.87 144 up osd.2
-17 0.09769 - 100 GiB 3.4 GiB 3.4 GiB 0 B 63 MiB 97 GiB 3.44 1.14 - host ocs-deviceset-2-data-1cwfjp
3 ssd 0.09769 1.00000 100 GiB 3.4 GiB 3.4 GiB 0 B 63 MiB 97 GiB 3.44 1.14 137 up osd.3
TOTAL 600 GiB 18 GiB 18 GiB 0 B 460 MiB 582 GiB 3.03
MIN/MAX VAR: 0.87/1.14 STDDEV: 0.37
Deletion of cephObjectstore
For more information, see Content from rook.io is not included.Deleting a CephObjectStore
If the objectbucket has data, you need to remove it
- Delete objects in bucket
s5cmd --endpoint-url $url rm $objectbucket/$object
- Delete bucket.
s5cmd --endpoint-url $url rb $objectbucket
- Delete the ObjecBucketClaim
$kubectl delete objectbucketclaim ceph-retain-bucket -nopenshift-storage
objectbucketclaim.objectbucket.io "ceph-retain-bucket" deleted
This also removes the k8s objectbucket
4. Remove the cephobjectstore
oc delete cephobjectstore ocs-storagecluster-cephobjectstore-ec -nopenshift-storage
cephobjectstore.ceph.rook.io "ocs-storagecluster-cephobjectstore-ec" deleted
- Delete the RGW
ec storageclasscreated if you intend to remove it.