How to invalidate system admin kubeconfig

Solution Verified - Updated

Environment

  • OpenShift Container Platform 4

Issue

  • How to invalidate a kubeconfig that was created for system:admin user during the installation?
  • How to invalidate a client certificate for system:admin user in case of data leak?
  • Is there option to create a new kubeconfig with new client certificate to replace the kubeconfig from the installation?
  • How to prevent unauthorized access with the admin kubeconfig to the cluster?
  • How to rotate the kubeconfigs on regular basis?

Resolution

Important Notes!

  • Make sure to always maintain the access to the cluster through the OAuth user, like kubeadmin or through the Identity Provider with proper cluster-admin permissions.

  • The following process can cause an access loss to the cluster if not followed properly! Without the kubeconfig as system:admin or OAuth user with cluster-admin, there is no option to gain access to the cluster and cluster needs to be reinstalled!

  • Read the solution article first before attempting the changes, and check the commands and their purpose. The steps described in the solution article require proper knowledge of the OpenShift Container Platform and advance skills in linux administration.

  • In case the support is needed from Red Hat, create new case with the Red Hat Support team.

Prerequisites

  • Having the cluster-admin permissions with system:admin or any OAuth user (like kubeadmin).
  • Steps below require access to the cluster using the oc client tools. Configuration through Web Console is not possible.
  • Working cluster without any Degraded cluster operators, nodes in Ready state and no Pending CSRs.
  • OAuth user with username/password that can be used in case of a failure (like kubeadmin).
  • Installed jq command on the bastion host.

Backup the original kubeconfig

Make sure to always backup the original kubeconfig or any other object mentioned in the article. In case of review of configuration or rollback of the changes. Never delete any data without backup!

  • Make a backup of the original kubeconfig.

    $ cp -v ~/.kube/config ~/.kube/config-original.backup

  • Make a backup of the node kubeconfigs created by the operator. Make sure to verify that the file exists and contains the correct data.

      $ mkdir -v $PWD/new-kubeconfig-workdir

  $ oc get secret -n openshift-kube-apiserver node-kubeconfigs -o json >$PWD/new-kubeconfig-workdir/node-kubeconfigs-secret.json

  $ cat $PWD/new-kubeconfig-workdir/node-kubeconfigs-secret.json | jq -r '.data[]' | less

  • (Optional) - Additionally, you can extract the node kubeconfigs to have them ready in case of an issue.

      $ oc extract -n openshift-kube-apiserver secret/node-kubeconfigs --to=$PWD/new-kubeconfig-workdir/

  $ ls -l $PWD/new-kubeconfig-workdir/

Generate a new kubeconfig

There is a built-in command into the oc binary client tools that allow creation of a new kubeconfig for a system:admin with client certificates.
This command also creates a new signing CA and updates the ConfigMap which Kube Apiserver trusts to allow the client certificate traffic.
The expiration of the client certificates are set to 10 years.

Note - that everytime you run the command below, a new CA is generated and added to the ca trust in ConfigMap.

  • To generate new admin kubeconfig, please follow the steps in the article below.
    How to regenerate the admin kubeconfig file in OpenShift 4

  • Check that the kubeconfig was generated with the correct client certificate. Verify that the subject contains group OU=system:masters and user CN=system:admin. Make note of the new CA signer.

      $ cat $PWD/new-kubeconfig-workdir/new-admin-kubeconfig | grep client-certificate | awk '{print $2}' | base64 -d | openssl x509 -noout -issuer -subject -dates

  ///  Example
  issuer=CN=openshift-config_admin.kubeconfig-signer@1782071313
  subject=O=system:masters, CN=system:admin 
  notBefore=Jun 21 19:48:33 2026 GMT
  notAfter=Jun 18 19:48:34 2036 GMT

  • Verify that you can use the new kubeconfig and you are able to access the cluster.

    $ export KUBECONFIG=$PWD/new-kubeconfig-workdir/new-admin-kubeconfig

  • Check the user and basic commands.

    $ oc whoami 

system:admin

Invalidate the original CA

In this section the original CA, that was used to sign the client certificate, will be invalidated. Make sure to follow the process correctly as in those steps a failure can happen and the access to the cluster might be lost!

  • Take a backup of the previous ConfigMaps

      $ oc get cm -n openshift-config admin-kubeconfig-client-ca -o yaml >$PWD/new-kubeconfig-workdir/admin-kubeconfig-client-ca.backup

  $ oc get cm -n openshift-kube-apiserver client-ca -o yaml >$PWD/new-kubeconfig-workdir/openshift-kube-apiserver-client-ca.backup

  $ oc get cm -n openshift-kube-controller-manager client-ca -o yaml >$PWD/new-kubeconfig-workdir/openshift-kube-controller-manager-client-ca.backup

  $ ls -l $PWD/new-kubeconfig-workdir/*.backup

  • Verify that the files were created and contain the correct data.

    $ cat $PWD/new-kubeconfig-workdir/*.backup | less

  • Export the ca-bundle.crt into your working directory

    $ oc get cm -n openshift-config admin-kubeconfig-client-ca -o json | jq -r '.data."ca-bundle.crt"' >$PWD/new-kubeconfig-workdir/working-ca-bundle.crt

  • Make a copy of that file, so the original file is still there.

    $ cp -v $PWD/new-kubeconfig-workdir/working-ca-bundle.crt $PWD/new-kubeconfig-workdir/ca-bundle.crt.backup

  • Edit the file and remove the certificate that should be invalidated. Usually the certificate will be the first in the list, as the newly created CA will be appended to the end of the file. Use the command below to check which certificate is which - the subject will show you the order in the file.

      $ cat $PWD/new-kubeconfig-workdir/new-admin-kubeconfig | grep client-certificate | awk '{print $2}' | base64 -d | openssl x509 -noout -issuer

  $ cat $PWD/new-kubeconfig-workdir/ca-bundle.crt.backup | openssl storeutl -noout -text -certs /dev/stdin | grep -e "Subject: "

  $ vi $PWD/new-kubeconfig-workdir/working-ca-bundle.crt

    Note that the certificate you want to remove depend on how many new kubeconfigs were created. If you wish to use only one kubeconfig that was created as part of process within this Solution, keep just the signer CA that was used to sign the client certificate in the kubeconfig. Check the step above to see which issuer should be kept.

  • If the file is correctly edit, update the ConfigMap

    $ oc set data cm -n openshift-config admin-kubeconfig-client-ca --from-file=ca-bundle.crt=$PWD/new-kubeconfig-workdir/working-ca-bundle.crt

    Note, as soon as the configmap is updated, the kube-apiserver will update the CA bundles locally. If the access to the cluster is lost at this point, even when using the new kubeconfig (make sure to use KUBECONFIG variable or verify that it was copied correctly in ~/.kube/config file), then there could be an issue with the cluster. Use the OAuth user to login the cluster and verify the ConfigMap contains the correct CAs with the command below and check the overall health of the cluster.

  • (Optional) - Make sure to use the new kubeconfig!

    $ export KUBECONFIG=$PWD/new-kubeconfig-workdir/new-admin-kubeconfig

  • Verify that the ConfigMap is correctly created and contains the correct CA signer(s).

    $ oc get cm -n openshift-config admin-kubeconfig-client-ca -o json | jq -r '.data."ca-bundle.crt"' | openssl storeutl -noout -text -certs /dev/stdin | grep -e "Subject: "

  • Check if that matches the CA used in the kubeconfig

    $ cat $PWD/new-kubeconfig-workdir/new-admin-kubeconfig | grep client-certificate | awk '{print $2}' | base64 -d | openssl x509 -noout -issuer -subject -dates

If the CA inside of the ConfigMap doesn't match the CA with the newly created kubeconfig, please open a case with Red Hat Support team and reference the solution article in the description.

Next Steps

The kubeconfig should trigger the rollout of the new revision of the control plane components. Monitor the situation with the cluster operators.

  • Check if the old kubeconfig is not working.

    $ oc get nodes --kubeconfig=~/.kube/config

    If you get "Unauthorized" result, this was successfull. If not, please reach out to the Red Hat Support team.

  • Move the new admin kubeconfig to local store.

    $ cp -v $PWD/new-kubeconfig-workdir/new-admin-kubeconfig ~/.kube/config

  • Check the Cluster Operators state, specifically the kube-apiserver and kube-controller-manager.

    $ oc get co

  • New CSR for the nodes might be requested, to approve any Pending CSR, run the command below.

    $ oc get csr | grep -i pending | awk '{print $1}' | xargs oc adm certificate approve

    Make sure to run the command periodically, as new CSRs could be requested at different time.

Rollback

In case the access to the cluster is lost or the new kubeconfig not working properly, the previous objects can be recreated from backup. Follow the process below.

  • Use the node kubeconfig lb-ext.kubeconfig to access the cluster through external load balancer.

    $ oc get nodes --kubeconfig=$PWD/new-kubeconfig-workdir/lb-ext.kubeconfig

  • If the command above works and you can gain access, follow with the steps to export the variable to point to the correct kubeconfig and check if you can view objects. If not, try to use the lb-int.kubeconfig. 

      $ export KUBECONFIG=$PWD/new-kubeconfig-workdir/lb-ext.kubeconfig

  $ oc whoami; oc get nodes

  • Revert the changes by adding the CA back.

    $ oc set data cm -n openshift-config admin-kubeconfig-client-ca --from-file=ca-bundle.crt=$PWD/new-kubeconfig-workdir/ca-bundle.crt.backup

  • Check with the old kubeconfig if you can gain access.

    $ export KUBECONFIG=~/.kube/config-original.backup

  • Check the CA configmap and verify if the old CA was recovered.

      $ oc get cm -n openshift-config admin-kubeconfig-client-ca -o json | jq -r '.data."ca-bundle.crt"' | openssl storeutl -noout -text -certs /dev/stdin | grep -e "Subject: "

  $ cat ~/.kube/config | grep client-certificate | awk '{print $2}' | base64 -d | openssl x509 -noout -issuer -subject -dates

Root Cause

The kubeconfig that was created during the installation of the OpenShift cluster contains the client certificate and key used for authentication against the API.
Because there is a client certificate signed by CA that the cluster trusts, there is no token that can be invalidated in case of data leak. The client certificate is only limited by the expiration date or by the signed CA.

The only way to invalidate or rotate the kubeconfig, the admin needs to invalidate the CA within the cluster, so the component will reject the request that the user with invalid client certificate is doing.

Diagnostic Steps

Reference to similar articles.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.