Error "invalid flag: plugins-order, error: list must only contain currently" in OpenShift 4 console pods

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• OpenShift Console

Issue

• The OpenShift 4 console pods are in CrashLoopBackOff with the following error:

  invalid flag: plugins-order, error: list must only contain currently
  

• Upgrade is stuck with a control plane node not able to be drained:

  Node [node_name] is reporting: "failed to drain node: [node_name] after
  

• The machine-config-controller shows a node failing to drain due to the console pods:

  node [node_name]: Drain failed. Drain has been failing for more than 10 minutes. Waiting 5 minutes then retrying. Error message from drain: error when evicting pods/"console-xxxxxxxxxx-xxxxx" -n "openshift-console": global timeout reached: 1m30s
  

Resolution

Follow the Diagnostic Steps for troubleshooting the issue, and if there is a missing consoleplugins, remove the reference in the spec.plugins field from the console.operator resource.

Root Cause

A plugin configured in the console.operator resource but not available is causing the console Pods to fail, preventing remaining running pod to be evicted, and causing an upgrade to got stuck.

Diagnostic Steps

• Some Cluster Operators shows that there are unavailable instances.

• Console pods are in CrashLoopBackOff, and only data in logs is error "invalid flag: plugins-order, error: list must only contain currently":

      $ oc get pods -n openshift-console
      NAME                         READY   STATUS             RESTARTS
      console-xxxxxxxxxx-xxxxx     0/1     CrashLoopBackOff   65
      [...]
  
      $ oc logs -n openshift-console console-xxxxxxxxxx-xxxxx
      2025-01-01T00:00:00.00000Z F0101 00:00:00.00000       1 validate.go:76] invalid flag: plugins-order, error: list must only contain currently enabled plugins
  

• Check the console-operator logs:

  $ oc logs -n openshift-console-operator console-operator-yyyyyyyyyy-yyyyy
  [...]
  2025-01-01T00:00:00.00000Z E0101 00:00:00.00000       1 sync_v400.go:726] failed to get "xxxxxxxxxx" plugin: consoleplugin.console.openshift.io "xxxxxxxxxx" not found
  2025-01-01T00:00:00.00000Z E0101 00:00:00.00000       1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.y.z, 1 replicas available
  [...]
  

• Check the plugins configured in the console.operator resource, and the available consoleplugins:

      $ oc get console.operator -o yaml
      [...]
          plugins:
          - networking-console-plugin
          - monitoring-plugin
          - nmstate-console-plugin
          - kubevirt-plugin
      [...]
  
      $ oc get consoleplugins
      [...]
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.