Procedure

1. Log in to the OpenShift web console as a cluster administrator.

2. Go to the Installed Operators page. The navigation path depends on your OpenShift version:

   • On OpenShift 4.20 and later, click Ecosystem → Installed Operators.
   • On OpenShift 4.19, click Operators → Installed Operators.
3. Click the Red Hat OpenShift AI Operator to open its details.
4. Click the Data Science Cluster tab.
5. On the DataScienceClusters page, click the default-dsc object.

6. Click the YAML tab.

   An embedded YAML editor opens, displaying the configuration for the DataScienceCluster custom resource.

7. In the YAML editor, locate the spec.components section. If the ogx field does not exist, add it. Then, set the managementState field to Managed:

   spec:
     components:
       ogx:
         managementState: Managed

   Copy code to clipboard
8. Click Save to apply your changes.

1. In the OpenShift web console, click Workloads → Pods.
2. From the Project list, select the redhat-ods-applications namespace.
3. Confirm that a pod with the label name=ogx-k8s-operator is displayed and has a status of Running.

Procedure

1. In the OpenShift web console, select Administrator → Quick Create ( ) → Import YAML, and create a CR similar to the following example ogx-custom-server.yaml file:

   Example ogx-custom-server.yaml

   apiVersion: ogx.io/v1beta1
   kind: OGXServer
   metadata:
     name: ogx-custom-server
     namespace: <project-name> # Replace with your OpenShift project
   spec:
     distribution:
       name: rh-dev
     workload:
       replicas: 1
       overrides:
         env:
           - name: VLLM_URL
             value: 'https://llama32-3b.ogx.svc.cluster.local/v1'
           - name: INFERENCE_MODEL
             value: llama32-3b
           - name: VLLM_TLS_VERIFY
             value: 'false'
           - name: POSTGRES_HOST
             value: <postgres-host>
           - name: POSTGRES_PORT
             value: '<postgres-port>' # Default PostgreSQL port is 5432
           - name: POSTGRES_DB
             value: ogx
           - name: POSTGRES_USER
             value: ogx
           - name: POSTGRES_PASSWORD
             valueFrom:
               secretKeyRef:
                 key: password
                 name: postgres-secret 1
         name: ogx
         port: 8321
       distribution:
         name: 'rh-dev'
       storage:
         size: 20Gi
         mountPath: <custom-mount-path> ## Defaults to /opt/app-root/src/.ogx/distributions/rh/

   Copy code to clipboard

   1

   Create the secret in the same namespace as the OGXServer resource. Avoid placing passwords directly on the command line, as they can be stored in shell history. Instead, create a file that contains only the database password and use that file to create the secret, or create the secret by using the OpenShift web console.

   For example:

   $ oc create secret generic postgres-secret --from-file=password=pg-password.txt -n <project-name>
   $ rm -f pg-password.txt

   Copy code to clipboard

   For more information about creating and managing Secrets, see Providing sensitive data to pods by using secrets.

   Ensure that the file pg-password.txt contains only the database password and is deleted after the secret is created.

   OGX automatically creates the metadata database specified by the POSTGRES_DB environment variable if it does not already exist, provided that the PostgreSQL user has sufficient privileges.

Verification

1. Check that the custom resource was created with the following command:

   $ oc get ogxserver -n ogx

   Copy code to clipboard

2. Check the running pods with the following command:

   $ oc get pods -n ogx | grep ogx-custom-server

   Copy code to clipboard

3. Check the logs with the following command:

   $ oc logs -n ogx -l app=ogx

   Copy code to clipboard

   Example output

   INFO: Started server process
   INFO: Waiting for application startup.
   INFO: Application startup complete.
   INFO: Uvicorn running on http://['::', '0.0.0.0']:8321

   Copy code to clipboard

1. In the OpenShift AI dashboard, navigate to the project details page and click the Deployments tab.
2. In the Model serving platform tile, click Select model.

3. Click the Deploy model button.

   The Deploy model dialog opens.

4. Configure the deployment properties for your model:

   1. In the Model deployment name field, enter a unique name for your deployment.
   2. In the Serving runtime field, select vLLM NVIDIA GPU serving runtime for KServe from the drop-down list.
   3. In the Deployment mode field, select KServe RawDeployment from the drop-down list.
   4. Set Number of model server replicas to deploy to 1.

   5. In the Model server size field, select Custom from the drop-down list.

      • Set CPUs requested to 1 core.
      • Set Memory requested to 10 GiB.
      • Set CPU limit to 2 core.
      • Set Memory limit to 14 GiB.
      • Set Accelerator to NVIDIA GPUs.
      • Set Accelerator count to 1.
   6. From the Connection type, select a relevant data connection from the drop-down list.

5. In the Additional serving runtime arguments field, specify the following recommended arguments:

   --dtype=half
   --max-model-len=20000
   --gpu-memory-utilization=0.95
   --enable-chunked-prefill
   --enable-auto-tool-choice
   --tool-call-parser=llama3_json
   --chat-template=/app/data/template/tool_chat_template_llama3.2_json.jinja

   Copy code to clipboard

   1. Click Deploy.

      Note

      Model deployment can take several minutes, especially for the first model that is deployed on the cluster. Initial deployment may take more than 10 minutes while the relevant images download.

Verification

1. Verify that the kserve-controller-manager and odh-model-controller pods are running:

   1. Open a new terminal window.
   2. Log in to your OpenShift cluster from the CLI:
   3. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   4. After you have logged in, click Display token.

   5. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy code to clipboard

   6. Enter the following command to verify that the kserve-controller-manager and odh-model-controller pods are running:

      $ oc get pods -n redhat-ods-applications | grep -E 'kserve-controller-manager|odh-model-controller'

      Copy code to clipboard

   7. Confirm that you see output similar to the following example:

      kserve-controller-manager-7c865c9c9f-xyz12   1/1     Running   0          4m21s
      odh-model-controller-7b7d5fd9cc-wxy34        1/1     Running   0          3m55s

      Copy code to clipboard

   8. If you do not see either of the kserve-controller-manager and odh-model-controller pods, there could be a problem with your deployment. In addition, if the pods appear in the list, but their Status is not set to Running, check the pod logs for errors:

      $ oc logs <pod-name> -n redhat-ods-applications

      Copy code to clipboard

   9. Check the status of the inference service:

      $ oc get inferenceservice -n ogx
      $ oc get pods -n <project name> | grep llama

      Copy code to clipboard

      • The deployment automatically creates the following resources:

        • A ServingRuntime resource.
        • An InferenceService resource, a Deployment, a pod, and a service pointing to the pod.

      • Verify that the server is running. For example:

        $ oc logs llama-32-3b-instruct-predictor-77f6574f76-8nl4r  -n <project name>

        Copy code to clipboard

        Check for output similar to the following example log:

        INFO     2025-05-15 11:23:52,750 __main__:498 server: Listening on ['::', '0.0.0.0']:8321
        INFO:     Started server process [1]
        INFO:     Waiting for application startup.
        INFO     2025-05-15 11:23:52,765 __main__:151 server: Starting up
        INFO:     Application startup complete.
        INFO:     Uvicorn running on http://['::', '0.0.0.0']:8321 (Press CTRL+C to quit)

        Copy code to clipboard
      • The deployed model displays in the Deployments tab on the project details page for the project it was deployed under.

2. If you see a ConvertTritonGPUToLLVM error in the pod logs when querying the /v1/chat/completions API, and the vLLM server restarts or returns a 500 Internal Server error, apply the following workaround:

   Before deploying the model, remove the --enable-chunked-prefill argument from the Additional serving runtime arguments field in the deployment dialog.

   The error is displayed similar to the following:

   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: error: Failures have been detected while processing an MLIR pass pipeline
   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: note: Pipeline failed while executing [`ConvertTritonGPUToLLVM` on 'builtin.module' operation]: reproducer generated at `std::errs, please share the reproducer above with Triton project.`
   INFO:     10.129.2.8:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

   Copy code to clipboard

Procedure

1. Open a new terminal window.

   1. Log in to your OpenShift cluster from the CLI:
   2. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   3. After you have logged in, click Display token.

   4. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy code to clipboard

2. If you enabled Require token authentication during model deployment, retrieve your token:

   $ export MODEL_TOKEN=$(oc get secret default-name-llama-32-3b-instruct-sa -n <project name> --template={{ .data.token }} | base64 -d)

   Copy code to clipboard

3. Obtain your model endpoint URL:

   • If you enabled Make deployed models available through an external route during model deployment, click Endpoint details on the Deployments page in the OpenShift AI dashboard to obtain your model endpoint URL.

   • In addition, if you did not enable Require token authentication during model deployment, you can also enter the following command to retrieve the endpoint URL:

     $ export MODEL_ENDPOINT="https://$(oc get route llama-32-3b-instruct -n <project name> --template={{ .spec.host }})"

     Copy code to clipboard

4. Test the endpoint with a sample chat completion request:

   • If you did not enable Require token authentication during model deployment, enter a chat completion request. For example:

     $ curl -X POST $MODEL_ENDPOINT/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
      "model": "llama-32-3b-instruct",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        }
      ]
     }'

     Copy code to clipboard

   • If you enabled Require token authentication during model deployment, include a token in your request. For example:

     curl -s -k $MODEL_ENDPOINT/v1/chat/completions \
     --header "Authorization: Bearer $MODEL_TOKEN" \
     --header 'Content-Type: application/json' \
     -d '{
       "model": "llama-32-3b-instruct",
       "messages": [
         {
           "role": "user",
           "content": "can you tell me a funny joke?"
         }
       ]
     }' | jq .

     Copy code to clipboard
     Note

     The -k flag disables SSL verification and should only be used in test environments or with self-signed certificates.

Procedure

1. To configure OGX to use role-based access control (RBAC) for model access, view and verify the OAuth provider token structure.

   1. Generate a Keycloak test token by running the following command:

      $ curl -d client_id=ogx -d client_secret=YOUR_CLIENT_SECRET -d username=user1 -d password=user-password -d grant_type=password https://my-keycloak-server.com/realms/ogx-demo/protocol/openid-connect/token | jq -r .access_token > test.token

      Copy code to clipboard

   2. View the token claims by running the following command:

      $ cat test.token | cut -d . -f 2 | base64 -d 2>/dev/null | jq .

      Copy code to clipboard

   Example token structure from Keycloak

   {
     "iss": "https://my-keycloak-server.com/realms/ogx-demo",
     "aud": "account",
     "sub": "761cdc99-80e5-4506-9b9e-26a67a8566f7",
     "preferred_username": "user1",
     "ogx_roles": [
       "inference_max"
     ]
   }

   Copy code to clipboard

2. Update your existing run.yaml file to add the OAuth parameters.

   Example OAuth parameters in the run.yaml file

   server:
     port: 8321
     auth:
       provider_config:
         type: "oauth2_token"
         jwks:
           uri: "https://my-keycloak-server.com/realms/ogx-demo/protocol/openid-connect/certs" 1
           key_recheck_period: 3600
         issuer: "https://my-keycloak-server.com/realms/ogx-demo" 2
         audience: "account"
         verify_tls: true
         claims_mapping:
           ogx_roles: "roles" 3
       access_policy:
         - permit: 4
             actions: [read]
             resource: model::vllm-inference/llama-3-2-3b
           description: Allow all authenticated users to access the Llama 3.2 model
         - permit: 5
             actions: [read]
             resource: model::openai/gpt-4o-mini
           when: user with inference_max in roles
           description: Allow only users with the inference_max role to access OpenAI models

   Copy code to clipboard

   1 2

   Specify your Keycloak host and realm in the URL.

   3

   Maps the ogx_roles claim from the token to the roles field.

   4

   Allows all authenticated users to access vLLM models.

   5

   Restricts OpenAI models to users with the inference_max role.

3. Create a ConfigMap that uses the updated run.yaml configuration by running the following command:

   $ oc create configmap ogx-custom-config --from-file=run.yaml=run.yaml -n redhat-ods-operator

   Copy code to clipboard

4. Create a ogx-server.yaml file with the following content:

   apiVersion: ogx.io/v1beta1
   kind: OGXServer
   metadata:
     name: ogx-server
     namespace: redhat-ods-operator
   spec:
     distribution:
       name: rh-dev
     workload:
       replicas: 1
       overrides:
         env:
           # vLLM provider configuration
           - name: VLLM_URL
             value: "https://your-vllm-service:8000/v1"
           - name: VLLM_API_TOKEN
             value: "your-vllm-token"
           - name: VLLM_TLS_VERIFY
             value: "false"
           # OpenAI provider configuration
           - name: OPENAI_API_KEY
             value: "your-openai-api-key"
           - name: OPENAI_BASE_URL
             value: "https://api.openai.com/v1"
     userConfig:
       configMapName: ogx-custom-config
       configMapNamespace: redhat-ods-operator

   Copy code to clipboard

5. Apply the distribution by running the following command:

   $ oc apply -f ogx-server.yaml

   Copy code to clipboard

6. Wait for the distribution to be ready by running the following command:

   $ oc wait --for=jsonpath='{.status.phase}'=Ready ogxserver/ogx-server -n redhat-ods-operator --timeout=300s

   Copy code to clipboard

7. Generate OAuth tokens for each user account to authenticate API requests.

   1. To request a basic access token and save it to a user1.token file, run the following command:

      $ curl -d client_id=ogx \
        -d client_secret=YOUR_CLIENT_SECRET \
        -d username=user1 \
        -d password=user1-password \
        -d grant_type=password \
        https://my-keycloak-server.com/realms/ogx-demo/protocol/openid-connect/token \
        | jq -r .access_token > user1.token

      Copy code to clipboard

   2. To request a token for the privileged user and save it to a user2.token file, run the following command:

      $ curl -d client_id=ogx \
        -d client_secret=YOUR_CLIENT_SECRET \
        -d username=user2 \
        -d password=user2-password \
        -d grant_type=password \
        https://my-keycloak-server.com/realms/ogx-demo/protocol/openid-connect/token \
        | jq -r .access_token > user2.token

      Copy code to clipboard

   3. Verify the token claims by running the following command:

      $ cat user2.token | cut -d . -f 2 | base64 -d 2>/dev/null | jq .

      Copy code to clipboard

Verification

1. Set the OGX service URL:

   $ export OGX_HOST="http://<ogx-host>:8321"

   Copy code to clipboard

2. Verify basic access for user1, who has no privileged roles.

   Load the token:

   $ USER1_TOKEN=$(cat user1.token)

   Copy code to clipboard

   Confirm that user1 can access the vLLM-served model:

   $ curl -s -o /dev/null -w "%{http_code}\n" \
     -X POST "${OGX_HOST}/v1/openai/chat/completions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer ${USER1_TOKEN}" \
     -d '{"model":"vllm-inference/llama-3-2-3b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}'

   Copy code to clipboard

   Expected result: HTTP 200.

   Confirm that user1 is denied access to the restricted OpenAI model:

   $ curl -s -o /dev/null -w "%{http_code}\n" \
     -X POST "${OGX_HOST}/v1/openai/chat/completions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer ${USER1_TOKEN}" \
     -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}'

   Copy code to clipboard

   Expected result: HTTP 403.

3. Verify privileged access for user2, who is assigned the inference_max role.

   Load the token:

   $ USER2_TOKEN=$(cat user2.token)

   Copy code to clipboard

   Confirm that user2 can access both models:

   $ curl -s -o /dev/null -w "%{http_code}\n" \
     -X POST "${OGX_HOST}/v1/openai/chat/completions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer ${USER2_TOKEN}" \
     -d '{"model":"vllm-inference/llama-3-2-3b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}'

   Copy code to clipboard

   $ curl -s -o /dev/null -w "%{http_code}\n" \
     -X POST "${OGX_HOST}/v1/openai/chat/completions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer ${USER2_TOKEN}" \
     -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}'

   Copy code to clipboard

   Expected result: HTTP 200 for both requests.

4. Verify that requests without a Bearer token are denied.

   $ curl -s -o /dev/null -w "%{http_code}\n" \
     -X POST "${OGX_HOST}/v1/openai/chat/completions" \
     -H "Content-Type: application/json" \
     -d '{"model":"vllm-inference/llama-3-2-3b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}'

   Copy code to clipboard

   Expected result: HTTP 401.

Procedure

1. The AUTH_* parameters also need to be set in the OGXServer custom resource. For example:

   Example OGXServer CR

   spec:
     replicas: 1
     server:
       containerSpec:
         env:
   ...
         - name: AUTH_ISSUER
           value: https://keycloak-redhat-ods-applications.apps.rosa.<user-cluster>.gm8d.p3.openshiftapps.com/realms/ogx-demo
         - name: AUTH_JWKS_URI
           value: http://keycloak:8080/realms/ogx-demo/protocol/openid-connect/certs

   Copy code to clipboard

   The server.auth section of the config.yaml file includes the authentication environment variables, these specifications uses OAuth2 token validation:

   server:
     auth:
       provider_config:
         type: ${env.AUTH_ISSUER:+oauth2_token}
         audience: ${env.AUTH_AUDIENCE:=ogx}
         issuer: ${env.AUTH_ISSUER:=}
         jwks:
           uri: ${env.AUTH_JWKS_URI:=}
           key_recheck_period: ${env.AUTH_JWKS_RECHECK_PERIOD:=3600}
         verify_tls: ${env.AUTH_VERIFY_TLS:=true}

   Copy code to clipboard

   Table 10.1. Environment variables reference

   Variable	Description	Default
   AUTH_ISSUER	OpenID connect (OIDC) issuer URL. If unset, authentication is disabled	None
   AUTH_AUDIENCE	Expected token audience	ogx
   AUTH_JWKS_URI	JSON Web key set (JWKS) endpoint for token validation	None
   AUTH_JWKS_RECHECK_PERIOD	How often, in seconds, to refresh JWKS keys	3600
   AUTH_VERIFY_TLS	Verify TLS when fetching JWKS	true

2. The client user must include a valid JWT bearer token in requests, for example:

   $ curl -H "Authorization: Bearer <token>" \
     https://ogx.example.com/v1/models

   Copy code to clipboard

3. The OGX distribution ships with a default access policy:

   access_policy:
     - permit:
         actions: [read]
       when: resource is unowned
       description: "All users can read system resources"
     - permit:
         actions: [create]
       description: "Authenticated users can create resources"
     - permit:
         actions: [read, update, delete]
       when: user is owner
       description: "Owners can manage their own resources"

   Copy code to clipboard

   You can change these policies and create custom permissions for resource allocation.

   The default policy describes the following behaviors for users:

   • System resources are readable by all - Resources without an owner. Models, shields, benchmarks registered in configuration are readable by any authenticated user.
   • Any authenticated user can create resources - Users can create their own vector databases, files, datasets, conversations, etc.
   • Users can only manage their own resources - Read, update, and delete operations on owned resources are restricted to the resource owner.

   This access policy applies to user-created resources including: Vector databases, Files, Datasets, Conversations, Responses, Agents. While system resources registered in the config.yaml file do not have an owner and are accessible by all user types.

Procedure

1. To enable high availability for your OGX server, add the following parameters to your OGXServer CR:

   spec:
     replicas: 2 1
     server:
       podDisruptionBudget:
         maxUnavailable: 1 2
       topologySpreadConstraints: 3
         - maxSkew: 1 4
           topologyKey: topology.kubernetes.io/zone 5
           whenUnsatisfiable: ScheduleAnyway 6
           labelSelector:
             matchLabels:
               app.kubernetes.io/instance: ogxserver-sample 7

   Copy code to clipboard

   1

   Runs two OGX pods for high availability.

   2

   Specifies voluntary disruption tolerance. This configuration keeps at least one server pod available during voluntary disruptions.

   3

   Specifies how matching pods are spread across the cluster topology.

   4

   Instructs the scheduler to minimize replica imbalance across zones. With two replicas, the scheduler attempts to place one pod per zone.

   5

   Uses the node zone label as the failure domain for pod spreading.

   6

   Allows scheduling to proceed even if spread constraints cannot be fully satisfied.

   7

   Ensures that only pods from the same application instance are considered when calculating spread.

2. To enable autoscaling for your OGX server, add the following parameters to your OGXServer CR:

   spec:
     server:
       autoscaling: 1
         minReplicas: 1 2
         maxReplicas: 5 3
         targetCPUUtilizationPercentage: 75 4
         targetMemoryUtilizationPercentage: 70 5

   Copy code to clipboard

   1

   Configures a HorizontalPodAutoscaler (HPA) for the server pods.

   2

   Specifies the minimum number of replicas maintained by the HPA.

   3

   Specifies the maximum number of replicas maintained by the HPA.

   4

   Enables CPU-based scaling.

   5

   Enables memory-based scaling.