How do I upgrade Red Hat Connectivity Link (RHCL) when AuthPolicy custom resources use fields not available in all served versions of the CRD?

Solution Verified - Updated

Environment

  • Product: Red Hat Connectivity Link (RHCL)
  • Versions Affected: Upgrades where AuthConfig CRD v1beta2 and v1beta3 are both served but v1beta2 lacks schema definitions for fields added in v1beta3
    • Specifically tested: RHCL 1.3.3 → 1.3.4
    • May affect other version transitions
  • Component: Authorino Operator, AuthPolicy/AuthConfig CRDs
  • Installation Method: Operator Lifecycle Manager (OLM)

Issue

KCS Solution: Workaround for CONNLINK-1131

When upgrading RHCL via OLM, the upgrade fails with a validation error if AuthPolicy custom resources use certain fields that were introduced in AuthConfig v1beta3.

Example error message:

error validating existing CRs against new CRD's schema for "authconfigs.authorino.kuadrant.io":
error validating authorino.kuadrant.io/v1beta2, Kind=AuthConfig:
"spec.authentication.*.when[0]" must validate one and only one schema (oneOf). Found none valid

Symptoms:

  • InstallPlan enters Failed state
  • Authorino operator CSV shows Pending phase
  • Other operators (DNS, Limitador) may show UpgradePending
  • Upgrade does not complete

When does this occur?

  • During OLM-managed upgrades between versions where:
    • AuthConfig CRD serves both v1beta2 and v1beta3
    • v1beta2 schema lacks definitions for certain fields added in v1beta3
    • Existing AuthPolicy CRs use any of the affected v1beta3-only fields (see Root Cause section for complete list)

Resolution

Use this method BEFORE starting the upgrade to avoid failure.

  1. Before approving the upgrade InstallPlan, temporarily disable v1beta2 serving on the AuthConfig CRD:

    kubectl patch crd authconfigs.authorino.kuadrant.io --type=json -p='[
  {"op": "replace", "path": "/spec/versions/0/served", "value": false}
]'

  2. Verify the patch was applied:

    kubectl get crd authconfigs.authorino.kuadrant.io -o jsonpath='{range .spec.versions[*]}{.name}{" served="}{.served}{"\n"}{end}'

    Expected output:

    v1beta2 served=false
v1beta3 served=true

  3. Proceed with the upgrade by approving the InstallPlan via your preferred method (OpenShift Console, CLI, GitOps, etc.)

  4. Wait for the upgrade to complete. Monitor CSV status:

    kubectl get csv -n kuadrant-system | grep -E "authorino|rhcl|dns|limitador"

  5. Verify all operators show Succeeded phase.

Result: The CRD patch is automatically restored by OLM after the upgrade completes. No manual cleanup is required.

Why this works:

  • OLM validates existing CRs against ALL served CRD versions during upgrades
  • Temporarily disabling v1beta2 forces OLM to validate only against v1beta3
  • v1beta3 includes the schema definitions for these fields, so validation passes
  • The upgrade succeeds and OLM restores v1beta2 serving from the new CSV

Alternative: Reactive Recovery (After Upgrade Failure)

Use this method ONLY if the upgrade has already failed and cannot be rolled back.

⚠️ Warning: This procedure is complex and requires active monitoring. Use the proactive workaround whenever possible.

Prerequisites

  • Failed InstallPlan with validation error
  • Access to kuadrant-system namespace
  • Manual approval workflow enabled

Steps

  1. Apply the CRD patch to disable v1beta2 serving:

    kubectl patch crd authconfigs.authorino.kuadrant.io --type=json -p='[
  {"op": "replace", "path": "/spec/versions/0/served", "value": false}
]'

  2. Delete the RHCL subscription:

    kubectl delete subscription rhcl-operator -n kuadrant-system

  3. Delete all CSVs from the failed upgrade:

    kubectl delete csv -n kuadrant-system \
  $(kubectl get csv -n kuadrant-system -o name | grep -E "rhcl-operator|authorino-operator|dns-operator|limitador-operator")

  4. Delete dependency operator subscriptions:

    kubectl delete subscription -n kuadrant-system \
  authorino-operator-stable-redhat-operators-openshift-marketplace \
  dns-operator-stable-redhat-operators-openshift-marketplace \
  limitador-operator-stable-redhat-operators-openshift-marketplace

  5. Recreate the RHCL subscription with the target version specified:

    kubectl apply -f - <<'EOFINNER'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: rhcl-operator
  namespace: kuadrant-system
spec:
  channel: stable
  installPlanApproval: Manual
  name: rhcl-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: rhcl-operator.v1.3.4  # Replace with target version
EOFINNER

    Important: Replace v1.3.4 with your intended target version. Without startingCSV, OLM will select the latest available version.

  6. Wait for the new InstallPlan to be created (approximately 30 seconds):

    kubectl get installplan -n kuadrant-system

  7. Approve the new InstallPlan:

    INSTALLPLAN=$(kubectl get installplan -n kuadrant-system --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
kubectl patch installplan $INSTALLPLAN -n kuadrant-system --type=json -p='[
  {"op": "replace", "path": "/spec/approved", "value": true}
]'

  8. Critical: Monitor the upgrade and re-apply the CRD patch if needed.

    The Authorino CSV may restore v1beta2 serving when it installs. If this happens before the RHCL CSV completes, the validation error will recur.

    Monitor CSV status:

    watch kubectl get csv -n kuadrant-system

    If you observe Authorino in Succeeded phase but RHCL still in Installing or showing errors, re-apply the patch:

    kubectl patch crd authconfigs.authorino.kuadrant.io --type=json -p='[
  {"op": "replace", "path": "/spec/versions/0/served", "value": false}
]'

    OLM will automatically retry and should succeed within 60 seconds.

  9. Verify all operators reached Succeeded phase:

    kubectl get csv -n kuadrant-system -o custom-columns='NAME:.metadata.name,VERSION:.spec.version,PHASE:.status.phase' | grep -E "NAME|authorino|rhcl|dns|limitador"

  10. Verify user resources are preserved:

    kubectl get authpolicy -A

Root Cause

During OLM-managed operator upgrades, the Operator Lifecycle Manager validates all existing custom resources against the schemas defined in ALL served versions of the associated CRD.

When upgrading RHCL:

  1. The current AuthConfig CRD serves both v1beta2 and v1beta3
  2. Several fields were added in v1beta3 that do not exist in the v1beta2 schema
  3. Existing AuthPolicy CRs that use these v1beta3-only fields create AuthConfig resources with these fields
  4. During the upgrade, OLM attempts to validate these AuthConfigs against both v1beta2 and v1beta3 schemas
  5. Validation against v1beta2 fails because the schema does not include definitions for these fields
  6. OLM marks the InstallPlan as Failed and halts the upgrade

The workaround temporarily disables v1beta2 serving, forcing OLM to validate only against v1beta3, which includes all field definitions.

Note: Most of these fields support Common Expression Language (CEL), but the issue is caused by the schema incompatibility between v1beta2 and v1beta3, not by the use of CEL itself.

Affected AuthPolicy Fields

The following fields were introduced in AuthConfig v1beta3 and will trigger this validation error if used in AuthPolicy CRs:

Predicates (when conditions):

  • spec.rules.when.predicate
  • spec.rules.authentication.when.predicate
  • spec.rules.metadata.when.predicate
  • spec.rules.authorization.when.predicate
  • spec.rules.authorization.patternMatching.patterns.predicate
  • spec.rules.response.success.headers.when.predicate
  • spec.rules.response.success.dynamicMetadata.when.predicate

URL expressions:

  • spec.rules.metadata.userInfo.userInfoUrl
  • spec.rules.metadata.http.urlExpression
  • spec.rules.authorization.opa.externalPolicy.urlExpression
  • spec.rules.callbacks.http.urlExpression

Other expressions:

  • spec.rules.authentication.plain.expression

Upstream Issue: Content from github.com is not included.https://github.com/Kuadrant/authorino/issues/627

Diagnostic Steps

To confirm this is the issue affecting your upgrade:

  1. Check if the InstallPlan is in Failed state:

    kubectl get installplan -n kuadrant-system

  2. Examine the InstallPlan for validation errors:

    kubectl get installplan <installplan-name> -n kuadrant-system -o jsonpath='{.status.message}'

    Look for:

    • References to authconfigs.authorino.kuadrant.io
    • Mentions of v1beta2
    • Validation errors containing when[0] or oneOf

  3. Check if your AuthPolicy CRs use any of the affected v1beta3-only fields:

    kubectl get authpolicy -A -o yaml | grep -E "predicate:|\.expression:|urlExpression:|userInfoUrl:"

    If you see any of these field names, this workaround applies.

    Alternative check for specific patterns:

    kubectl get authpolicy -A -o yaml | grep -A 2 -E "when:|plain:|urlExpression:|userInfoUrl:|patterns:"

  4. Verify which CRD versions are served:

    kubectl get crd authconfigs.authorino.kuadrant.io -o jsonpath='{range .spec.versions[*]}{.name}{" served="} 
{.served}{"\n"}{end}'

Diagnostic Steps

To confirm this is the issue affecting your upgrade:

  1. Check if the InstallPlan is in Failed state:

    kubectl get installplan -n kuadrant-system

  2. Examine the InstallPlan for validation errors:

    kubectl get installplan <installplan-name> -n kuadrant-system -o jsonpath='{.status.message}'

    Look for:

    • References to authconfigs.authorino.kuadrant.io
    • Mentions of v1beta2
    • Validation errors containing when[0] or oneOf

  3. Check if your AuthPolicy CRs use any of the affected v1beta3-only fields:

    kubectl get authpolicy -A -o yaml | grep -E "predicate:|\.expression:|urlExpression:|userInfoUrl:"

    If you see any of these field names, this workaround applies.

    Alternative check for specific patterns:

    kubectl get authpolicy -A -o yaml | grep -A 2 -E "when:|plain:|urlExpression:|userInfoUrl:|patterns:"

  4. Verify which CRD versions are served:

    kubectl get crd authconfigs.authorino.kuadrant.io -o jsonpath='{range .spec.versions[*]}{.name}{" served="}    
{.served}{"\n"}{end}'

Additional Information

Best Practices:

  • Always use the proactive workaround when planning upgrades
  • Test upgrades in non-production environments first
  • Ensure you have recent backups of all custom resources
  • Monitor operator pod logs during upgrades for early warning signs

Limitations:

  • The CRD patch must be monitored during reactive recovery
  • Reactive recovery requires subscription recreation, which may cause brief service interruption
  • Without specifying startingCSV, OLM will select the latest version, not the originally intended version

Future Versions:
A fix for the underlying validation schema issue is being tracked. Once released, this workaround will no longer be necessary for upgrades to/from versions containing the fix.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.