1. You create an Azure managed identity with the required permissions to manage peer pod virtual machines.
2. You create a federated identity credential that links the OpenShift sandboxed containers Operator service account to the managed identity.
3. When installing the OpenShift sandboxed containers Operator, you provide the managed identity credentials (CLIENTID, TENANTID, SUBSCRIPTIONID) as environment variables in the Operator subscription.
4. The OpenShift sandboxed containers Operator uses these credentials to create credential configuration secrets.
5. Microsoft Entra ID exchanges short-lived Kubernetes service account tokens for Azure access tokens.
6. The OpenShift sandboxed containers component uses these access tokens to authenticate Azure API requests for peer pod operations.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

   Copy code to clipboard

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy code to clipboard

4. Create the Operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

   Copy code to clipboard

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.13.1

   Copy code to clipboard

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

Verification

1. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

2. Watch the installation progress by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   NAME                             DISPLAY                                  VERSION         PHASE
   sandboxed-containers-operator.v1.13.1   OpenShift sandboxed containers Operator   1.13.1          Succeeded

   Copy code to clipboard

   The installation is complete when the PHASE column shows Succeeded.

1. Configure the mirror registry.
2. Add registry credentials to both clusters.
3. Copy the pull secret to workload namespaces.
4. Mirror Operator images to the private registry.
5. Mirror workload images to the private registry.
6. Plan non-overlapping network CIDRs for both clusters.
7. Create VNet peering for cross-cluster connectivity.

Procedure

1. Set the Azure resource group variable:

   $ export AZURE_RESOURCE_GROUP="<resource_group_name>"

   Copy code to clipboard

   where:

   <resource_group_name>

   Specifies the name of the resource group that contains your OpenShift Container Platform cluster.

2. Retrieve the network security group (NSG) name:

   $ NSG_NAME=$(az network nsg list -g "${AZURE_RESOURCE_GROUP}" \
     --query "[].name" -o tsv)

   Copy code to clipboard
   Note

   This command assumes that only one NSG exists in the resource group. If you have multiple NSGs, manually identify which NSG your worker subnet uses and set NSG_NAME explicitly.

3. Enable port 15150 for communication between the Kata shim and agent:

   $ az network nsg rule create -g "${AZURE_RESOURCE_GROUP}" \
     --nsg-name "${NSG_NAME}" -n "AllowKataShimAgent" \
     --priority 200 --direction Inbound --access Allow \
     --protocol Tcp --destination-port-ranges 15150 \
     --source-address-prefixes "VirtualNetwork" \
     --destination-address-prefixes "VirtualNetwork"

   Copy code to clipboard

4. Enable port 9000 for Virtual Extensible LAN (VXLAN) tunnel traffic:

   $ az network nsg rule create -g "${AZURE_RESOURCE_GROUP}" \
     --nsg-name "${NSG_NAME}" -n "AllowVXLAN" \
     --priority 201 --direction Inbound --access Allow \
     --protocol Tcp --destination-port-ranges 9000 \
     --source-address-prefixes "VirtualNetwork" \
     --destination-address-prefixes "VirtualNetwork"

   Copy code to clipboard

Procedure

1. Create a my-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"

   Copy code to clipboard

2. Create the config map by running the following command:

   $ oc create -f my-feature-gate.yaml

   Copy code to clipboard

Verification

1. Verify that the config map exists by running the following command:

   $ oc get configmap osc-feature-gates -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   NAME                DATA   AGE
   osc-feature-gates   1      10s

   Copy code to clipboard

2. Confirm that confidential: "true" is present in the config map data by running the following command:

   $ oc get configmap osc-feature-gates -n openshift-sandboxed-containers-operator -o jsonpath='{.data}'

   Copy code to clipboard

   {"confidential":"true"}

   Copy code to clipboard

Procedure

1. Obtain the Red Hat build of Trustee uniform resource locator (URL) by running the following command:

   $ TRUSTEE_URL=$(oc get route kbs-service \
     -n trustee-operator-system -o jsonpath='{.spec.host}') \
     && echo $TRUSTEE_URL

   Copy code to clipboard

2. Create the initdata.toml file:

   algorithm = sha256
   version = "0.1.0"
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   
   url = '<trustee_url>'
   
   [token_configs.kbs]
   url = '<trustee_url>'
   '''
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<trustee_url>'
   kbs_cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   [image]
   image_security_policy_uri = 'kbs:///default/<secret_policy_name>/<key>'
   '''
   
   "policy.rego" = '''
   package agent_policy
   
   import future.keywords.in
   import future.keywords.if
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true
   default CreateContainerRequest := true
   default ReadStreamRequest := false
   default CopyFileRequest := false
   default SetPolicyRequest := false
   default ExecProcessRequest := false
   
   CopyFileRequest if {
       print("CopyFileRequest: input =", input)
       allow_copy_file
       print("CopyFileRequest: true")
   }
   
   allow_copy_file if {
       print("allow_copy_file regular")
       input.file_type == "Regular"
       allow_copy_file_path(input.path, "")
       print("allow_copy_file regular: true")
   }
   
   allow_copy_file if {
       print("allow_copy_file directory")
       input.file_type == "Directory"
       allow_copy_file_path(input.path, "")
       print("allow_copy_file directory: true")
   }
   
   allow_copy_file if {
       print("allow_copy_file symlink")
       input.file_type == "Symlink"
       allow_copy_file_path(input.path, ".*/.+")
       check_directory_traversal(input.symlink_target)
       not startswith(input.symlink_target, "/")
       print("allow_copy_file symlink: true")
   }
   
   allow_copy_file_path(path, regex_suffix) if {
       check_directory_traversal(path)
       some regex1 in policy_data.request_defaults.CopyFileRequest
       regex2 := replace(regex1, "$(sfprefix)", policy_data.common.sfprefix)
       regex3 := replace(regex2, "$(cpath)", policy_data.common.cpath)
       regex4 := replace(regex3, "$(bundle-id)", "[a-z0-9]{64}")
       regex5 := concat("", [regex4, regex_suffix])
       print("allow_copy_file_path: regex5 =", regex5)
       regex.match(regex5, path)
   }
   
   check_directory_traversal(i_path) if {
       not regex.match("(^|/)\\.\\.($|/)", i_path)
   }
   
   policy_data := {
       "common": {
           "cpath": "/run/kata-containers/shared/containers(?:/passthrough)?",
           "sfprefix": "^$(cpath)/(watchable/)?$(bundle-id)-[a-z0-9]{16}-"
       },
       "request_defaults": {
           "CopyFileRequest": [
               "$(sfprefix)"
           ]
       }
   }
   '''

   Copy code to clipboard

   URL

   Specify Red Hat build of Trustee URL. If you configure Red Hat build of Trustee with insecure_http for testing purposes, use HTTP. Otherwise, use HTTPS. For production systems, avoid using insecure_http unless you configure your environment to handle Transport Layer Security (TLS) externally, for example, with a proxy.

   kbs_certificate

   Specify the Base64-encoded TLS certificate for the attestation agent.

   kbs_cert

   See the prerequisite above regarding kbs_cert and insecure_http.

   image_security_policy_uri

   Optional, only if you enabled the container image signature verification policy. Replace <secret_policy_name> with the name of the secret that contains the policy and <key> with the key within that secret.

3. Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Copy code to clipboard

   Record this string to use in the peer pods config map or the peer pod manifest.

4. Calculate the hash of the initdata.toml file and assign its value to the hash variable by running the command that corresponds to the algorithm value you set in the initdata.toml file:

   For sha256:

   +

   $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)

   Copy code to clipboard

   For sha384:

   +

   $ hash=$(sha384sum initdata.toml | cut -d' ' -f1)

   Copy code to clipboard

   For sha512:

   +

   $ hash=$(sha512sum initdata.toml | cut -d' ' -f1)

   Copy code to clipboard

5. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

   $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

   Copy code to clipboard

6. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

   $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH

   Copy code to clipboard

   Record the PCR8_HASH value for the RVPS config map.

Verification

1. Verify that the initdata.txt file exists and is not empty by running the following command:

   $ ls -lh initdata.txt && cat initdata.txt

   Copy code to clipboard

   Confirm that the file exists and has a Base64-encoded string.

2. Verify that the PCR8_HASH variable was set by running the following command:

   $ echo $PCR8_HASH

   Copy code to clipboard

   Confirm that the output is a non-empty hash value.

Procedure

1. Retrieve and record the Azure resource group by running the following command:

   $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
     -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') \
     && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

   Copy code to clipboard

2. Retrieve and record the Azure VNet name:

   $ AZURE_VNET_NAME=$(az network vnet list \
     --resource-group ${AZURE_RESOURCE_GROUP} \
     --query "[].{Name:name}" --output tsv)

   Copy code to clipboard

   Use this value to retrieve the Azure subnet ID.

3. Retrieve and record the Azure subnet ID:

   $ AZURE_SUBNET_ID=$(az network vnet subnet list \
     --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME \
     --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) \
      && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

   Copy code to clipboard

4. Retrieve and record the Azure network security group (NSG) ID:

   $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} \
     --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

   Copy code to clipboard

5. Retrieve and record the Azure region:

   $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} \
     --query "{Location:location}" --output tsv) \
     && echo "AZURE_REGION: \"$AZURE_REGION\""

   Copy code to clipboard

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "8m"
     AZURE_INSTANCE_SIZE: "Standard_DC2as_v5"
     AZURE_INSTANCE_SIZES: "Standard_DC2as_v5,Standard_DC4as_v5,Standard_DC8as_v5"
     AZURE_SUBNET_ID: "<azure_subnet_id>"
     AZURE_NSG_ID: "<azure_nsg_id>"
     AZURE_IMAGE_ID: ""
     AZURE_REGION: "<azure_region>"
     AZURE_RESOURCE_GROUP: "<azure_resource_group>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "10"
     DISABLECVM: "false"
     INITDATA: "<initdata_string>"

   Copy code to clipboard

   AZURE_INSTANCE_SIZE

   Defines the default instance size that is used if the instance size is not defined in the workload object. "Standard_DC2as_v5" is for AMD SEV-SNP. If your trusted execution environment (TEE) is Intel® Trust Domain Extensions (TDX), specify Standard_EC4eds_v5.

   AZURE_IMAGE_ID

   Leave this value empty. When you install the Operator, a Job is scheduled to download the default pod virtual machine (VM) image from the Red Hat Ecosystem Catalog and upload it to the Azure Image Gallery within the same Azure Resource Group as the OpenShift Container Platform cluster. This image provides root disk integrity protection (dm-verity) and encrypted container storage. See This content is not included.Confidential VMs: The core of confidential containers for details.

   AZURE_INSTANCE_SIZES

   Specify the allowed instance sizes, without spaces, for creating the pod. You can define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.

   TAGS

   You can configure custom tags as key=value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   Maximum number of peer pods per node. The default value is 10.

   ROOT_VOLUME_SIZE

   Root volume size in gigabytes for the pod VM. The minimum size is 10 GB.

   INITDATA

   Specify the initdata string to create a default configuration for all peer pods. If you add initdata to a peer pod manifest, that setting overrides this global configuration.

2. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy code to clipboard

Verification

1. Verify that the peer pods config map exists by running the following command:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   NAME           DATA   AGE
   peer-pods-cm   10     10s

   Copy code to clipboard

Procedure

1. Add the initdata string to the pod manifest and save the file as my-pod.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy code to clipboard

   where

   <initdata_string>

   Specify the gzipped, Base64-encoded initdata value in a pod annotation to override the global INITDATA setting in the peer pods config map.

   <container_name>

   Specify a container name.

2. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy code to clipboard

Verification

1. Verify that the pod is running by running the following command:

   $ oc get pod ocp-cc-pod

   Copy code to clipboard

   NAME          READY   STATUS    RESTARTS   AGE
   ocp-cc-pod    1/1     Running   0          30s

   Copy code to clipboard

2. Verify that the pod is using the expected runtime class by running the following command:

   $ oc get pod <pod_name> -o jsonpath='{.spec.runtimeClassName}'

   Copy code to clipboard

   Confirm the output matches the expected runtime class, for example, kata-remote.

Procedure

1. Optional: If the custom pod VM image is in a private registry, configure a pull secret:

   1. Set the NS variable to the namespace where you deploy your peer pods:

      $ NS=<namespace>

      Copy code to clipboard

   2. Copy the pull secret to the peer pod namespace:

      $ oc get secret pull-secret -n openshift-config -o yaml \
        | sed "s/namespace: openshift-config/namespace: ${NS}/" \
        | oc apply -n "${NS}" -f -

      Copy code to clipboard

      You can use the cluster pull secret, as in this example, or a custom pull secret.

   3. Optional: Link the pull secret to the default service account:

      $ oc secrets link default pull-secret --for=pull -n ${NS}

      Copy code to clipboard

2. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy code to clipboard

3. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy code to clipboard

Verification

1. Verify that the pod is running with the custom peer pod VM image by running the following command:

   $ oc get pod my-pod-manifest -o jsonpath='{.metadata.annotations.io\.katacontainers\.config\.hypervisor\.image}'

   Copy code to clipboard
2. Confirm that the output matches the custom image ID you specified.

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Optional: Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

Procedure

1. Create a test-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: "<initdata_string>"
   spec:
     runtimeClassName: kata-remote
     containers:
       - name: skr-openshift
         image: registry.access.redhat.com/ubi9/ubi:latest
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault

   Copy code to clipboard

   where:

   io.katacontainers.config.hypervisor.cc_init_data

   Optional: Specifies initdata in a pod annotation, which overrides the global INITDATA setting in the peer pods config map.

2. Create the pod by running the following command:

   $ oc create -f test-pod.yaml

   Copy code to clipboard

3. Log in to the pod by running the following command:

   $ oc exec -it ocp-cc-pod -- bash

   Copy code to clipboard

Verification

1. Fetch the Red Hat build of Trustee resource to confirm that attestation succeeded:

   $ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status

   Copy code to clipboard

   success #/

   Copy code to clipboard

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy code to clipboard

2. Optional: To override the default instance size, add the following annotation with an instance size that is defined in the peer pods config map:

   apiVersion: v1
   kind: <object>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.machine_type: <instance>
   # ...

   Copy code to clipboard

3. Optional: To use an automatic instance size, add the following annotations:

   apiVersion: v1
   kind: <Pod>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
       io.katacontainers.config.hypervisor.default_memory: <memory>
   # ...

   Copy code to clipboard

   The workload runs on an automatic instance size based on the amount of memory available.

4. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy code to clipboard

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.13.1

   Copy code to clipboard

2. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

Verification

1. Verify that the Operator upgrade is complete by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

2. Watch the upgrade progress by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   NAME                             DISPLAY                                  VERSION   REPLACES    PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.13.1    1.13.0      Succeeded

   Copy code to clipboard

   The upgrade is complete when the PHASE column shows Succeeded for the new version.

Procedure

1. Delete the KataConfig CR by running the following command:

   $ oc delete kataconfig example-kataconfig

   Copy code to clipboard

   The OpenShift sandboxed containers Operator removes all resources that were initially created to enable the runtime on your cluster.

   Important

   When you delete the KataConfig CR, the command-line interface (CLI) stops responding until all worker nodes reboot. You must wait for the deletion process to complete before performing the verification.

Verification

1. Confirm that the KataConfig CR no longer exists by running the following command:

   $ oc get kataconfig example-kataconfig

   Copy code to clipboard

   Error from server (NotFound): kataconfigs.kataconfiguration.openshift.io "example-kataconfig" not found

   Copy code to clipboard

Procedure

1. Obtain the value of the AZURE_IMAGE_ID in the peer pods config map by running the following command:

   $ oc get configmap -n openshift-sandboxed-containers-operator peer-pods-cm -o jsonpath="{.data.AZURE_IMAGE_ID}"

   Copy code to clipboard

2. If the value is not empty, update the value and patch the config map by running the following command:

   $ oc patch configmap peer-pods-cm -n openshift-sandboxed-containers-operator -p '{"data":{"AZURE_IMAGE_ID":""}}'

   Copy code to clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Optional: Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

Procedure

1. Search for the pods by running the following command:

   $ oc get pods -A -o json | jq -r '.items[] |
     select(.spec.runtimeClassName == "kata-remote").metadata.name'

   Copy code to clipboard

2. Delete each pod by running the following command:

   $ oc delete pod <pod>

   Copy code to clipboard

Verification

1. Verify that the pods using the kata-remote runtime class are no longer running by running the following command:

   $ oc get pods -A -o json | jq -r '.items[] | select(.spec.runtimeClassName == "kata-remote").metadata.name'

   Copy code to clipboard

   Confirm that the command returns no output.

Procedure

1. Delete the KataConfig CR by running the following command:

   $ oc delete kataconfig example-kataconfig

   Copy code to clipboard

   The OpenShift sandboxed containers Operator removes all resources that were initially created to enable the runtime on your cluster.

   Important

   When you delete the KataConfig CR, the command-line interface (CLI) stops responding until all worker nodes reboot. You must wait for the deletion process to complete before performing the verification.

Verification

1. Confirm that the KataConfig CR no longer exists by running the following command:

   $ oc get kataconfig example-kataconfig

   Copy code to clipboard

   Error from server (NotFound): kataconfigs.kataconfiguration.openshift.io "example-kataconfig" not found

   Copy code to clipboard

Procedure

1. Delete the subscription by running the following command:

   $ oc delete subscription sandboxed-containers-operator -n openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Delete the namespace by running the following command:

   $ oc delete namespace openshift-sandboxed-containers-operator

   Copy code to clipboard

Procedure

1. Delete the KataConfig CRD by running the following command:

   $ oc delete crd kataconfigs.kataconfiguration.openshift.io

   Copy code to clipboard

Verification

1. Confirm that the KataConfig CRD no longer exists by running the following command:

   $ oc get crd kataconfigs.kataconfiguration.openshift.io

   Copy code to clipboard

   Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "kataconfigs.kataconfiguration.openshift.io" not found

   Copy code to clipboard

Procedure

1. In the OpenShift Container Platform web console, navigate to Observe → Metrics.

2. In the input field, enter the query for the metric you want to observe.

   All kata-related metrics begin with kata. Typing kata displays a list of all available kata metrics.

   The metrics from your query are visualized on the page.

Procedure

1. Change the logLevel field in your existing KataConfig CR to debug:

   $ oc patch kataconfig <kataconfig> --type merge --patch '{"spec":{"logLevel":"debug"}}'

   Copy code to clipboard

2. Monitor the kata-oc machine config pool until the value of UPDATED is True, indicating that all worker nodes are updated:

   $ oc get mcp kata-oc

   Copy code to clipboard

   Example output

   NAME     CONFIG                 UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
   kata-oc  rendered-kata-oc-169   False    True      False     3             1                  1                    0                     9h

   Copy code to clipboard

Verification

1. Start a debug session with a node in the machine config pool:

   $ oc debug node/<node_name>

   Copy code to clipboard

2. Change the root directory to /host:

   # chroot /host

   Copy code to clipboard

3. Verify the changes in the crio.conf file:

   # crio config | egrep 'log_level

   Copy code to clipboard

   Example output

   log_level = "debug"

   Copy code to clipboard