DG 8 Persistence Volume details and how to read PV

Environment

• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Data Grid (RHDG)
  • 8.x

Issue

• How to read PV data in OCP 4?
• Issues about reading DG 8 PV Data in OCP 4?

Resolution

FAQ about Reading DG 8 PV Data::

Q0. What is PV? What does it mean? Does PV persist data after the pod restart?
A0. PV means persisted volume. It means persisted because it persists pod restarts. Everything in $RHDG_HOME/server/data that is meant to survive pod restarts. Therefore, deleting the pod does not mean the PV gets cleaned, although type of the PV is delete, which only matters when the PV gets unbounded.

Q1. Does the user create PV on DG 8?
A1. See Root cause for the complete details on PV/PVC interaction.

Q2. When we set persistence on the cache configuration for file-storage, which writes to the PV. If the pod drops, will the PV be persisted? - baseline question.
A2. Yes, given writing to the PV (persisted) so then PV will be persisted.

Q3. How to retrieve the data from the PV? (when persistence is enabled)? If the DG goes down?
A3. With the file-store, the data is stored in binary format and there aren't any 3rd party tools to read it. You would need to mount the PVC(s) in a Pod(s) to retrieve PV access.

Q4. How to retrieve the data itself?
A4. When the DG is restarted, the PV is mounted with the data and DG is able to read the data again.

For example, if you want to just recreate a cluster using the persistent data: if the PVCs remain, it's possible to create a new Infinispan CR with the same name as before and the same PVCs will be re-used. However, if you just want to extract the raw files for future use/debugging then you need to:

1. mount the PVC to a pod
2. and extract the data from the location you mount it using oc cp ... for extraction.

See Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4.

Q5. What is the format the data is stored? Is it ascii?
A5. Binary format for all cache store configurations except the SQL store, where is field is mapped to a table column for the SQL DB. However, nothing is stored in ascii though, nor human readable format.

For other questions, see the full article Data Grid Persistence volume FAQ.

Root Cause

PV vs PVC deep explanation

The operator never deals with PV directly, in the same way we use the Pod abstraction instead of interacting with Nodes directly.
The operator provides a PVC spec as part of the cluster StatefulSet. The StatefulSet controller then creates a PVC for each of the Pods in the cluster, using the configured amount of storage.
For details on PV manual provisioning see Deploy Red Hat Data Grid 8 in OpenShift 4 using Operator.In summary:

• For dynamic provisioning: It is the default method used. However, on the Infinispan CR user can set a custom storageClassName. The persistent volume claim uses the storage class that has the storageclass.kubernetes.io/is-default-class annotation set to true.
• For manual provisioning: the user creates a dg8storageclass and then the PV, which would then be picked up by the DG pods accordingly - which creates the claims. See solution Deploy Red Hat Data Grid 8 in OpenShift 4 using Operator.

Finally, the pods deployed by the DG Operator have RWO set for the PVC, so the DG pods need to be set to zero (on Infinispan CR set to zero), before injecting from another.

It's possible for the user to set the specific the amount of storage allocated to each PVC by providing a value for spec.service.container.storage in the Infinispan Custom Resource, larger than the default 1Gi.
See more details on allocating storage class services.

Issues

DG 8.3 has issues with persistence for FileStore as explained on the solution How an unshared CacheStore behave in a RHDG clustered environment? - preloading doesn't solve this issue. The only workaround is setting purge=true (or not using a persistent PV) that would stop the issue. However, the main implication is that the data will not survive a full cluster restart.

DG Operator will use the default Storage Class, i.e. the class with storageclass.kubernetes.io/is-default-class. To change the default storage, change that annotation therefore.

Diagnostic Steps

1. See pod logs for corruption of file:

16:18:24,543 ERROR (non-blocking-thread--p2-t1)
[org.infinispan.server.hotrod.BaseRequestProcessor] ISPN005003: Exception reported
org.infinispan.persistence.spi.PersistenceExcep8on: ISPN029019: Cannot load key
WrappedByteArray[03\>0A\5\0\0\0\0\2\5\6\9\7 (13 bytes)] from index.
at
org.infinispan.persistence.sifs.NonBlockingSoCIndexFileStore.lambda$load$17(NonBlockingSoCInde
xFileStore.java:605)
at
java.base/java.u8l.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:3
5)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1
486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateExcep8on: Error reading from 25:16511130
Page 4 of 7
at org.infinispan.persistence.sifs.EntryRecord.readEntryHeader(EntryRecord.java:93)
at org.infinispan.persistence.sifs.IndexNode$LeafNode.getHeaderAndKey(IndexNode.java:1033)
at org.infinispan.persistence.sifs.IndexNode$LeafNode.loadRecord(IndexNode.java:1063)
at org.infinispan.persistence.sifs.IndexNode$ReadOpera8on$1.apply(IndexNode.java:224)
at org.infinispan.persistence.sifs.IndexNode$ReadOpera8on$1.apply(IndexNode.java:221)
at org.infinispan.persistence.sifs.IndexNode.applyOnLeaf(IndexNode.java:295)
at org.infinispan.persistence.sifs.Index.getRecord(Index.java:120)
at org.infinispan.persistence.sifs.Index.getRecord(Index.java:106)
at
org.infinispan.persistence.sifs.NonBlockingSoCIndexFileStore.lambda$load$17(NonBlockingSoCInde
xFileStore.java:596)
... 6 mo

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.