Procedure

1. Create a my-nfd.yaml manifest file according to the following example:

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
     namespace: openshift-nfd
   spec:
     operand:
       image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.22
       imagePullPolicy: Always
       servicePort: 12000
     workerConfig:
       configData: |

   Copy code to clipboard

2. Create the NodeFeatureDiscovery CR:

   $ oc create -f my-nfd.yaml

   Copy code to clipboard

Procedure

1. Create a custom resource manifest named my-nodefeaturerule.yaml:

   apiVersion: nfd.openshift.io/v1alpha1
   kind: NodeFeatureRule
   metadata:
     name: consolidated-hardware-features
     namespace: openshift-nfd
   spec:
     rules:
       - name: "runtime.kata"
         labels:
           feature.node.kubernetes.io/runtime.kata: "true"
         matchAny:
           - matchFeatures:
               - feature: cpu.cpuid
                 matchExpressions:
                   SSE42: { op: Exists }
                   VMX: { op: Exists }
               - feature: kernel.loadedmodule
                 matchExpressions:
                   kvm: { op: Exists }
                   kvm_intel: { op: Exists }
           - matchFeatures:
               - feature: cpu.cpuid
                 matchExpressions:
                   SSE42: { op: Exists }
                   SVM: { op: Exists }
               - feature: kernel.loadedmodule
                 matchExpressions:
                   kvm: { op: Exists }
                   kvm_amd: { op: Exists }
       - name: "amd.sev-snp"
         labels:
           amd.feature.node.kubernetes.io/snp: "true"
         extendedResources:
           sev-snp.amd.com/esids: "@cpu.security.sev.encrypted_state_ids"
         matchFeatures:
           - feature: cpu.cpuid
             matchExpressions:
               SVM: { op: Exists }
           - feature: cpu.security
             matchExpressions:
               sev.snp.enabled: { op: Exists }
   
       - name: "intel.sgx"
         labels:
           intel.feature.node.kubernetes.io/sgx: "true"
         extendedResources:
           sgx.intel.com/epc: "@cpu.security.sgx.epc"
         matchFeatures:
           - feature: cpu.cpuid
             matchExpressions:
               SGX: { op: Exists }
               SGXLC: { op: Exists }
           - feature: cpu.security
             matchExpressions:
               sgx.enabled: { op: IsTrue }
           - feature: kernel.config
             matchExpressions:
               X86_SGX: { op: Exists }
   
       - name: "intel.tdx"
         labels:
           intel.feature.node.kubernetes.io/tdx: "true"
         extendedResources:
           tdx.intel.com/keys: "@cpu.security.tdx.total_keys"
         matchFeatures:
           - feature: cpu.cpuid
             matchExpressions:
               VMX: { op: Exists }
           - feature: cpu.security
             matchExpressions:
               tdx.enabled: { op: Exists }

   Copy code to clipboard

2. Create the NodeFeatureRule CR by running the following command:

   $ oc create -f my-nodefeaturerule.yaml

   Copy code to clipboard
   Note

   A relabeling delay of up to 1 minute might occur.

Verification

1. Verify that the NodeFeatureRule CR was created by running the following command:

   $ oc get nodefeaturerule -n openshift-nfd

   Copy code to clipboard

   Confirm that consolidated-hardware-features appears in the output.

2. Verify that the expected hardware feature labels were applied to your nodes by running the following command:

   $ oc get nodes --show-labels | grep feature.node.kubernetes.io

   Copy code to clipboard

Procedure

1. Create a storage-encrypted.yaml manifest file for the PersistentVolumeClaim object with the volumeMode parameter set to Block:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: storage-encrypted
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Block
     resources:
       requests:
         storage: <size>

   Copy code to clipboard

2. Create the PersistentVolumeClaim object by running the following command:

   $ oc create -f storage-encrypted.yaml

   Copy code to clipboard

3. Create an encrypted-pod.yaml manifest file with the complete pod specification:

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <init_data>
     name: storage-encrypted
     labels:
       app: storage-encrypted
   spec:
     runtimeClassName: kata-cc
     shareProcessNamespace: true
     initContainers:
     - name: format-disk
       image: quay.io/redhat-user-workloads/ose-osc-tenant/osc-storage-helper:on-pr-3fe822e41e1bd31cd2cbfd9468ab087abc58d9a1-linux-x86-64
       command: ["/usr/local/bin/luks-helper", "format-disk"]
       securityContext:
         privileged: true
       restartPolicy: Always
       env:
         - name: PASS
           valueFrom:
             secretKeyRef:
               name: <my_sealed_secret>
               key: <secret_key>
       volumeMounts:
         - name: storage-ipc
           mountPath: /dev/shm
       volumeDevices:
         - name: luks-block
           devicePath: /dev/block-device
     - name: check-ready
       image: quay.io/redhat-user-workloads/ose-osc-tenant/osc-storage-helper:on-pr-3fe822e41e1bd31cd2cbfd9468ab087abc58d9a1-linux-x86-64
       command: ["/usr/local/bin/luks-helper", "wait-ready"]
       securityContext:
         privileged: true
       volumeMounts:
         - name: storage-ipc
           mountPath: /dev/shm
     containers:
     - name: <container_name>
       image: <image_name>
       ports:
         - containerPort: 8888
       env:
         - name: DATA_DIR
           value: <mount_point>
       lifecycle:
         postStart:
           exec:
             command:
               - /bin/sh
               - -c
               - |
                 PID=$(cat /dev/shm/luks-helper.pid)
                 chmod ug+w "$(dirname "$DATA_DIR")"
                 ln -sfn "/proc/$PID/root/mnt/storage" "$DATA_DIR"
       securityContext:
         privileged: true
       volumeMounts:
         - name: storage-ipc
           mountPath: /dev/shm
     volumes:
     - name: luks-block
       persistentVolumeClaim:
         claimName: storage-encrypted
     - name: storage-ipc
       emptyDir:
         medium: Memory

   Copy code to clipboard

   where:

   <init_data>

   Specifies the initdata for the runtime configuration.

   <my_sealed_secret>

   Specifies the name of the sealed secret that contains the LUKS encryption passphrase.

   <secret_key>

   Specifies the key within the sealed secret that contains the encryption passphrase.

   <container_name>

   Specifies the container name for your application container.

   <image_name>

   Specifies the image name.

   <mount_point>

   Specifies the mount point for encrypted storage inside your application container.

4. Create the pod by running the following command:

   $ oc create -f encrypted-pod.yaml

   Copy code to clipboard

Verification

1. Verify that the pod is running by running the following command:

   $ oc get pod storage-encrypted

   Copy code to clipboard

   NAME                READY   STATUS    RESTARTS   AGE
   storage-encrypted   1/1     Running   0          30s

   Copy code to clipboard

2. Confirm that the LUKS formatting completed without errors by checking the pod logs:

   $ oc logs storage-encrypted -c format-disk

   Copy code to clipboard

   Confirm that the LUKS formatting completed without errors.

Procedure

1. Create a gpu-machine-config.yaml manifest file according to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 100-iommu-kernel-args
   spec:
     config:
       ignition:
         version: 3.2.0
     kernelArguments:
     - amd_iommu=on
     - intel_iommu=on

   Copy code to clipboard
   Note

   If using Single Node OpenShift (SNO), replace worker with master in the machineconfiguration.openshift.io/role label.

   The nodes will reboot after applying this configuration.

2. Create the config map by running the following command:

   $ oc create -f gpu-machine-config.yaml

   Copy code to clipboard

Procedure

1. Install the Node Feature Discovery (NFD) Operator by following the This content is not included.OpenShift Container Platform documentation.

Procedure

1. Create a NodeFeatureRule custom resource to match NVIDIA kernel modules by running the following command:

   apiVersion: nfd.openshift.io/v1alpha1
   kind: NodeFeatureRule
   metadata:
     name: nvidia-kernel-modules
   spec:
     rules:
       - name: kernel-module-gdrdrv
         labels:
           nvidia.com/gdrcopy.capable: "true"
         matchFeatures:
           - feature: kernel.loadedmodule
             matchExpressions:
               gdrdrv:
                 op: Exists
       - name: kernel-module-nvidia_fs
         labels:
           nvidia.com/gds.capable: "true"
         matchFeatures:
           - feature: kernel.loadedmodule
             matchExpressions:
               nvidia_fs:
                 op: Exists
       - name: kernel-module-nvidia_peermem
         labels:
           nvidia.com/peermem.capable: "true"
         matchFeatures:
           - feature: kernel.loadedmodule
             matchExpressions:
               nvidia_peermem:
                 op: Exists

   Copy code to clipboard

2. Create the NodeFeatureRule CR by running the following command:

   $ oc create -f my-nfd-gpu.yaml

   Copy code to clipboard

Verification

1. Verify that the NodeFeatureRule CR is present by running the following command:

   $ oc get nodefeaturerule -n openshift-nfd

   Copy code to clipboard

   Confirm that nvidia-kernel-modules appears in the output.

2. Verify that GPU-related node labels appear on your worker nodes by running the following command:

   $ oc get nodes --show-labels | grep nvidia

   Copy code to clipboard

Verification

1. Verify that the GPU Operator pods are running by running the following command:

   $ oc get pods -n nvidia-gpu-operator

   Copy code to clipboard

   NAME                                       READY   STATUS    RESTARTS   AGE
   gpu-operator-1234567890-abcde              1/1     Running   0          10m

   Copy code to clipboard

Procedure

1. Create a my-cluster-policy-gpu.yaml manifest file according to the following example:

   apiVersion: nvidia.com/v1
   kind: ClusterPolicy
   metadata:
     name: gpu-cluster-policy
   spec:
     ccManager:
       defaultMode: "on"
       enabled: true
     cdi:
       default: false
       enabled: true
       nriPluginEnabled: false
     daemonsets:
       rollingUpdate:
         maxUnavailable: '1'
       updateStrategy: RollingUpdate
     dcgm:
       enabled: false
     dcgmExporter:
       config:
         name: ''
       enabled: false
       serviceMonitor:
         enabled: true
     devicePlugin:
       config:
         default: ''
         name: ''
       enabled: false
       mps:
         root: /run/nvidia/mps
     driver:
       certConfig:
         name: ''
       enabled: false
       kernelModuleConfig:
         name: ''
       kernelModuleType: auto
       licensingConfig:
         configMapName: ''
         nlsEnabled: true
       repoConfig:
         configMapName: ''
       upgradePolicy:
         autoUpgrade: true
         drain:
           deleteEmptyDir: false
           enable: false
           force: false
           timeoutSeconds: 300
         maxParallelUpgrades: 1
         maxUnavailable: 25%
         podDeletion:
           deleteEmptyDir: false
           force: false
           timeoutSeconds: 300
         waitForCompletion:
           timeoutSeconds: 0
       useNvidiaDriverCRD: false
       useOpenKernelModules: false
       virtualTopology:
         config: ''
     gdrcopy:
       enabled: false
     gds:
       enabled: false
     gfd:
       enabled: true
     kataManager:
       enabled: false
     mig:
       strategy: single
     migManager:
       enabled: false
     nodeStatusExporter:
       enabled: true
     operator:
       defaultRuntime: crio
       initContainer: {}
       runtimeClass: nvidia
       use_ocp_driver_toolkit: true
     kataSandboxDevicePlugin:
       enabled: true
       env:
         - name: P_GPU_ALIAS
           value: pgpu
         - name: NVSWITCH_ALIAS
           value: nvswitch
     sandboxWorkloads:
       defaultWorkload: vm-passthrough
       enabled: true
       mode: kata
     toolkit:
       enabled: false
       installDir: /usr/local/nvidia
     validator:
       plugin:
         env:
           - name: WITH_WORKLOAD
             value: 'false'
     vfioManager:
       enabled: true
       env:
         - name: BIND_NVSWITCHES
           value: 'true'
     vgpuDeviceManager:
       enabled: false
     vgpuManager:
       enabled: false

   Copy code to clipboard

2. Create the ClusterPolicy CR by running the following command:

   $ oc create -f my-cluster-policy-gpu.yaml

   Copy code to clipboard

Verification

1. Verify that the required labels are present on your worker nodes by running the following command:

   $ oc get nodes -o json | jq '.items[].metadata.labels | with_entries(select(.key | startswith("nvidia.com")))'

   Copy code to clipboard

   "nvidia.com/cc.mode.state"
   "nvidia.com/cc.ready.state"
   "nvidia.com/gpu.deploy.cc-manager"
   "nvidia.com/gpu.deploy.kata-manager"
   "nvidia.com/gpu.deploy.kata-sandbox-device-plugin"
   "nvidia.com/gpu.deploy.sandbox-validator"
   "nvidia.com/gpu.deploy.vfio-manager"
   "nvidia.com/gpu.present"

   Copy code to clipboard

2. Verify the GPU Operator setup by running the following command:

   $ oc get pods -n nvidia-gpu-operator

   Copy code to clipboard

   NAME                                                READY   STATUS    RESTARTS      AGE
   gpu-operator-cb99f5757-djl7k                        1/1     Running   2             16h
   nvidia-cc-manager-hjd6t                             1/1     Running   5 (42m ago)   16h
   nvidia-kata-sandbox-device-plugin-daemonset-wn6bc   1/1     Running   2             16h
   nvidia-sandbox-validator-7cvx5                      1/1     Running   0             70m
   nvidia-vfio-manager-zsmqn                           1/1     Running   2             16h

   Copy code to clipboard

3. Verify the nvidia-cc-manager daemon set by running the following command:

   $ oc get daemonset -n nvidia-gpu-operator | grep cc-manager

   Copy code to clipboard

   nvidia-cc-manager     1         1         1       1            1           nvidia.com/gpu.deploy.cc-manager=true   7m43s

   Copy code to clipboard

Procedure

1. Create a sample-gpu-pod.yaml manifest file with the following content:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sample-gpu-pod
     annotations:
       io.katacontainers.config.hypervisor.default_memory: "32768"
       io.katacontainers.config.hypervisor.cc_init_data: "H4sIAAAAAAAAA6WUS2/TQBCA7/4Vli85xQ1IoKpSDyUJUIkQy07JoYqiyXpir7IPszsONb++YyIQINauxMGHnfk875kzOi+tiW/jZJa+SmdJBKqyTlKte5mv4fWbt0n0WALBLkoaq6ToUoeVTVg/mUyiBsQJKoz5M7S/AFFU4hFaRfFdWd7l2WeUVX2wzuf4tUVP8c1tTK7F37HiGzQh9VxZjwWV0gQJ23TvpcKg3iEQzq0hkAbdMFaAKQ/2KQQtWOhsN0Itn1Bkzgr0waw/IK2QnBRDxHq9Wp65tEGkly6Q81JBM5+kp3tD6I7A8QxRuW0pTKxQf7TUqLZ613Fuh2C510ZxmefZA/8RYjJo/XhHslape83DFQJyhLIg7lvQUY7ansc9XbCCQOEX6Ujaoy9qcLiyraFgSXL0iGXOs2D1As8DWKvHYyiQLv3kKdxIHUyauezHpgUBWRlQI/PHuToaj6mnNg6ENNUAQ/4FlmwzYmhD3bbfci+/B5N/aMqX7PIFWzY1anSghtt4gX/tyDA2vCRbkDRS+C0fWPz30PYnNQFIyWr188Q+kj2h2QtrjrLyu7/e6enAstap/mDXRM3N1RWLph7dWQpM2awnxKltuAxk3dR3/NY317NrPve9/SgRZf2nx9NB7CIDGnujQuz5nfyXj2f3zZjpaQYAAA=="
   spec:
     runtimeClassName: kata-cc-nvidia-gpu
     restartPolicy: OnFailure
     containers:
       - name: gpu-cc-verifier
         image: quay.io/openshift_sandboxed_containers/gpu-verifier:ubi9
         imagePullPolicy: IfNotPresent
         command: ["/bin/bash"]
         args:
           - -c
           - |
             /opt/cuda-samples/Samples/0_Introduction/vectorAdd/build/vectorAdd
             sleep 36000
         resources:
           limits:
             nvidia.com/pgpu: 1
         securityContext:
           privileged: false

   Copy code to clipboard
   Note

   The io.katacontainers.config.hypervisor.cc_init_data annotation includes a permissive kata-agent policy for verification purposes. The embedded kata-agent policy disables the exec and log APIs. This configuration does not include a Key Broker Service (KBS) URL, which prevents issues in customer environments where the KBS URL might not align to the actual deployment.

2. Apply the manifest by running the following command:

   $ oc apply -f sample-gpu-pod.yaml

   Copy code to clipboard

Verification

1. Verify that the sample pod is running by running the following command:

   $ oc get pods

   Copy code to clipboard

   NAME               READY   STATUS    RESTARTS   AGE
   sample-gpu-pod     1/1     Running   0          2m

   Copy code to clipboard

2. Check the pod logs to verify GPU functionality by running the following command:

   $ oc logs sample-gpu-pod

   Copy code to clipboard

   [Vector addition of 50000 elements]
   Copy input data from the host memory to the CUDA device
   CUDA kernel launch with 196 blocks of 256 threads
   Copy output data from the CUDA device to the host memory
   Test PASSED
   Done

   Copy code to clipboard

Procedure

1. Create a dgx-b200-cluster-policy.yaml manifest file according to the following example:

   apiVersion: nvidia.com/v1
   kind: ClusterPolicy
   metadata:
     name: gpu-cluster-policy
   spec:
     ccManager:
       defaultMode: "on"
       enabled: true
     cdi:
       default: false
       enabled: true
       nriPluginEnabled: false
     daemonsets:
       rollingUpdate:
         maxUnavailable: '1'
       updateStrategy: RollingUpdate
     dcgm:
       enabled: false
     dcgmExporter:
       config:
         name: ''
       enabled: false
       serviceMonitor:
         enabled: true
     devicePlugin:
       config:
         default: ''
         name: ''
       enabled: false
       mps:
         root: /run/nvidia/mps
     driver:
       certConfig:
         name: ''
       enabled: false
       kernelModuleConfig:
         name: ''
       kernelModuleType: auto
       licensingConfig:
         configMapName: ''
         nlsEnabled: true
       repoConfig:
         configMapName: ''
       upgradePolicy:
         autoUpgrade: true
         drain:
           deleteEmptyDir: false
           enable: false
           force: false
           timeoutSeconds: 300
         maxParallelUpgrades: 1
         maxUnavailable: 25%
         podDeletion:
           deleteEmptyDir: false
           force: false
           timeoutSeconds: 300
         waitForCompletion:
           timeoutSeconds: 0
       useNvidiaDriverCRD: false
       useOpenKernelModules: false
       virtualTopology:
         config: ''
     gdrcopy:
       enabled: false
     gds:
       enabled: false
     gfd:
       enabled: true
     kataManager:
       enabled: false
     mig:
       strategy: single
     migManager:
       enabled: false
     nodeStatusExporter:
       enabled: true
     operator:
       defaultRuntime: crio
       initContainer: {}
       runtimeClass: nvidia
       use_ocp_driver_toolkit: true
     kataSandboxDevicePlugin:
       enabled: true
       env:
         - name: P_GPU_ALIAS
           value: pgpu
         - name: NVSWITCH_ALIAS
           value: nvswitch
     sandboxWorkloads:
       defaultWorkload: vm-passthrough
       enabled: true
       mode: kata
     toolkit:
       enabled: false
       installDir: /usr/local/nvidia
     validator:
       plugin:
         env:
           - name: WITH_WORKLOAD
             value: 'false'
     vfioManager:
       enabled: true
       env:
         - name: BIND_NVSWITCHES
           value: 'true'
     vgpuDeviceManager:
       enabled: false
     vgpuManager:
       enabled: false

   Copy code to clipboard
   Note

   The kataSandboxDevicePlugin section registers GPUs under the nvidia.com/pgpu resource name.

2. Create the ClusterPolicy CR by running the following command:

   $ oc create -f dgx-b200-cluster-policy.yaml

   Copy code to clipboard

Verification

1. Verify that the ClusterPolicy status is ready by running the following command:

   $ oc get clusterpolicy gpu-cluster-policy -o jsonpath='{.status.state}'

   Copy code to clipboard

   ready

   Copy code to clipboard

2. Verify the allocatable GPU resources on the NVIDIA DGX B200 node by running the following command:

   $ oc get node <node_name> -o jsonpath='{.status.allocatable}' | jq

   Copy code to clipboard

   The output must include nvidia.com/pgpu entries that correspond to the number of GPUs available on the node.

3. Verify that the CC mode labels are applied to the node by running the following command:

   $ oc get node <node_name> -o json | jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com/cc")))'

   Copy code to clipboard

   {
     "nvidia.com/cc.mode.state": "on",
     "nvidia.com/cc.ready.state": "true"
   }

   Copy code to clipboard

Procedure

1. Open a debug session on the NVIDIA DGX B200 worker node by running the following command:

   $ oc debug node/<node_name>

   Copy code to clipboard

2. Change to the host root file system by running the following command:

   sh-5.1# chroot /host

   Copy code to clipboard

3. Determine the NVIDIA driver version from the kata initrd filename by running the following command:

   # ls /usr/share/kata-containers/osbuilder-images/6.12.0-*/kata*nvidia*

   Copy code to clipboard

   kata-cc-nvidia-gpu-595.58.03.initrd

   Copy code to clipboard

   The file name includes the driver version. In this example, the driver version is 595.58.03.

4. Unlock Red Hat Enterprise Linux CoreOS (RHCOS) for hotfix mode by running the following command:

   # rpm-ostree usroverlay

   Copy code to clipboard

5. Find the Fabric Manager and NVLSM RPM packages that match your driver version in the Content from developer.download.nvidia.com is not included.NVIDIA CUDA repository for RHEL 9.

   Note

   The RPM package naming format can vary between driver versions. Search for nvidia-fabricmanager and nvidia-nvlsm packages that match the driver version identified in the previous step.

6. Install the Fabric Manager RPM by running the following command:

   # rpm -ivh --nodeps <fabricmanager_rpm_url>

   Copy code to clipboard

   Replace <fabricmanager_rpm_url> with the URL of the Fabric Manager RPM package from the NVIDIA CUDA repository.

7. Install the NVLSM RPM by running the following command:

   # rpm -ivh --nodeps <nvlsm_rpm_url>

   Copy code to clipboard

   Replace <nvlsm_rpm_url> with the URL of the NVLSM RPM package from the NVIDIA CUDA repository.

   Important

   The hotfix layer created by rpm-ostree usroverlay is non-persistent. Any RPM packages installed with this method are lost after a node reboot or an OpenShift Container Platform upgrade. You must repeat this procedure after each reboot.

   For a persistent installation, consider using a MachineConfig or rpm-ostree layering.

Next steps

1. Install ibstat dependencies for Fabric Manager.
2. Configure and start Fabric Manager.

Procedure

1. Find the ibstat binary in the container overlay directory by running the following command:

   # find /var/lib/containers/storage/overlay -path '*/usr/sbin/ibstat' | head -1 | sed 's|/usr/sbin/ibstat||'

   Copy code to clipboard

2. Copy the ibstat binary to /usr/sbin/ by running the following command:

   # cp <overlay_path>/usr/sbin/ibstat /usr/sbin/

   Copy code to clipboard

   Replace <overlay_path> with the path returned in the previous step.

3. Copy the libibmad library by running the following command:

   # cp <overlay_path>/usr/lib64/libibmad.so.5.* /usr/lib64/

   Copy code to clipboard

4. Copy the libibnetdisc library by running the following command:

   # cp <overlay_path>/usr/lib64/libibnetdisc.so.5.* /usr/lib64/

   Copy code to clipboard

5. Copy the libibumad library by running the following command:

   # cp <overlay_path>/usr/lib64/libibumad.so.3.* /usr/lib64/

   Copy code to clipboard

6. Create the libibmad symlink by running the following command:

   # ln -sf /usr/lib64/libibmad.so.5.* /usr/lib64/libibmad.so.5

   Copy code to clipboard

7. Create the libibnetdisc symlink by running the following command:

   # ln -sf /usr/lib64/libibnetdisc.so.5.* /usr/lib64/libibnetdisc.so.5

   Copy code to clipboard

8. Create the libibumad symlink by running the following command:

   # ln -sf /usr/lib64/libibumad.so.3.* /usr/lib64/libibumad.so.3

   Copy code to clipboard

9. Update the shared library cache by running the following command:

   # ldconfig

   Copy code to clipboard

Procedure

1. Set the partition rail policy to symmetric by running the following command:

   $ sed -i 's/PARTITION_RAIL_POLICY=greedy/PARTITION_RAIL_POLICY=symmetric/' /usr/share/nvidia/nvswitch/fabricmanager.cfg

   Copy code to clipboard

2. Enable the Fabric Manager service by running the following command:

   # systemctl enable nvidia-fabricmanager

   Copy code to clipboard

3. Start the Fabric Manager service by running the following command:

   # systemctl start nvidia-fabricmanager

   Copy code to clipboard

Procedure

1. Check the kernel messages for TDX module initialization by running the following command:

   # dmesg | grep -i tdx

   Copy code to clipboard

   The output must confirm the TDX module is initialized.

2. Verify that TDX is enabled in the kvm_intel kernel module parameters by running the following command:

   # cat /sys/module/kvm_intel/parameters/tdx

   Copy code to clipboard

   Y

   Copy code to clipboard
   Note

   If the TDX module is not initialized, the Platform Secure Encrypted Architecture Loader (P-SEAMLDR) might have silently rejected an incompatible module version. Verify that the firmware supports TDX and that the TDX kernel parameters are correctly configured in the MachineConfig.

Procedure

1. Create a kubelet-timeout.yaml manifest file according to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-kubelet-timeout
   spec:
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/master: ""
     kubeletConfig:
       runtimeRequestTimeout: "10m"

   Copy code to clipboard

2. Create the KubeletConfig CR by running the following command:

   $ oc create -f kubelet-timeout.yaml

   Copy code to clipboard

3. Monitor the update status of MachineConfigPool by running the following command:

   $ oc get mcp --watch

   Copy code to clipboard

   When the UPDATED column displays True and the UPDATING column displays False for the relevant pool, the update is complete.

   Important

   After the MachineConfigPool update completes, the nodes reboot. You must reinstall the hotfix RPM packages (Fabric Manager, NVLink Switch Manager (NVLSM), and ibstat dependencies) because the hotfix layer does not persist across reboots.

4. Add the container_create_timeout = 600 setting to the [crio.runtime.runtimes.kata-tdx-nvidia-gpu] section in the /etc/crio/crio.conf.d/50-kata-tdx-nvidia-gpu file on the NVIDIA DGX B200 worker node:

   [crio.runtime.runtimes.kata-tdx-nvidia-gpu]
     runtime_path = "/usr/bin/containerd-shim-kata-v2"
     runtime_type = "vm"
     runtime_root = "/run/vc"
     runtime_config_path = "/etc/kata-containers/kata-tdx-nvidia-gpu/configuration.toml"
     privileged_without_host_devices = true
     runtime_pull_image = true
     container_create_timeout = 600
     allowed_annotations = [
       "io.kubernetes.cri-o.Devices",
     ]

   Copy code to clipboard

5. Delete all GPU workload pods by running the following command:

   $ oc delete pods -n <namespace> --all

   Copy code to clipboard

   Replace <namespace> with the namespace that contains your GPU workload pods.

6. Restart CRI-O on the NVIDIA DGX B200 worker node by running the following command:

   # systemctl restart crio

   Copy code to clipboard
   Important

   The CRI-O timeout configuration does not persist across node reboots. You must reapply this configuration after each reboot.

Procedure

1. To disable CC mode:

   1. Delete all GPU workload pods by running the following command:

      $ oc delete pods -n <namespace> --all

      Copy code to clipboard

      Replace <namespace> with the namespace that contains your GPU workload pods.

   2. Stop the Fabric Manager service on the NVIDIA DGX B200 worker node by running the following command:

      # systemctl stop nvidia-fabricmanager

      Copy code to clipboard

   3. Patch the ClusterPolicy CR to disable CC mode by running the following command:

      $ oc patch clusterpolicy gpu-cluster-policy --type merge -p '{"spec":{"ccManager":{"defaultMode":"off"}}}'

      Copy code to clipboard

   4. Verify that the CC mode labels are updated on the node by running the following command:

      $ oc get node <node_name> -o json | jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com/cc")))'

      Copy code to clipboard

      {
        "nvidia.com/cc.mode.state": "off",
        "nvidia.com/cc.ready.state": "false"
      }

      Copy code to clipboard

   5. Restart the Fabric Manager service by running the following command:

      # systemctl start nvidia-fabricmanager

      Copy code to clipboard

      Fabric Manager is required for CUDA operations in non-CC mode. You must restart it after switching CC mode off.

   6. Deploy your workloads using the kata-nvidia-gpu runtime class for non-CC GPU workloads.

2. To enable CC mode:

   1. Delete all GPU workload pods by running the following command:

      $ oc delete pods -n <namespace> --all

      Copy code to clipboard

      Replace <namespace> with the namespace that contains your GPU workload pods.

   2. Patch the ClusterPolicy CR to enable CC mode by running the following command:

      $ oc patch clusterpolicy gpu-cluster-policy --type merge -p '{"spec":{"ccManager":{"defaultMode":"on"}}}'

      Copy code to clipboard

   3. Verify that the CC mode labels are updated on the node by running the following command:

      $ oc get node <node_name> -o json | jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com/cc")))'

      Copy code to clipboard

      {
        "nvidia.com/cc.mode.state": "on",
        "nvidia.com/cc.ready.state": "true"
      }

      Copy code to clipboard

   4. Start the Fabric Manager service by running the following command:

      # systemctl start nvidia-fabricmanager

      Copy code to clipboard

   5. Deploy your workloads using the kata-cc-nvidia-gpu runtime class for confidential containers GPU workloads.

      Note

      Fabric Manager is required for multi-GPU workloads in both CC and non-CC modes. Single-GPU workloads in CC mode do not require Fabric Manager.

1. Install Fabric Manager and NVLSM.
2. Install ibstat dependencies for Fabric Manager.
3. Configure and start Fabric Manager.
4. Configure timeouts for multi-GPU confidential containers workloads (CRI-O timeout only; the KubeletConfig CR persists).