Manual podman pull fails within OCP 4 nodes

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • Trying to pull images with podman pull fails within OpenShift 4 nodes:

    sh-4.4# podman pull registry.redhat.io/openshift4/ose-logging-fluentd
Failed to pull image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:xxxxxxx": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials.

    sh-4.4# podman pull image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator:latest
[...]
Trying to pull image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator:latest...
Error: Error initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator:latest: Error reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator: unauthorized: authentication required

  • After a successful podman login, podman pull fails with unauthorized error.

Resolution

For running podman pull withing an OpenShift 4 node, it is needed to include the credentials for the registry with --authfile parameter. The registry credentials configured in the cluster via the global pull secret can be used with --authfile /var/lib/kubelet/config.json.

An example to pull an image from a node (change the [image_name] with the name of the required image):

$ oc get nodes
[...]
$ oc debug node/[node_name]
[...]

sh-4.4# chroot /host bash

[root@node_name /]# podman pull --authfile /var/lib/kubelet/config.json [image_name]

For additional troubleshooting, it is possible to set the log level to debug the following way:

# podman --log-level debug pull --authfile /var/lib/kubelet/config.json [image_name]

If still failing with credentials error when using the above credentials, configure a valid global pull secret as per how to change the global pull secret in OCP 4.

Root Cause

By default, podman login does not store the credentials, as they will be in clear text, causing podman pull to fail even after a successful podman login if not authentication is used. Including the credentials for the registry with --authfile parameter will be needed.

Diagnostic Steps

  • Connect to an OCP 4 node:

        $ oc get nodes
    [...]

    $ oc debug node/[node_name]
    Creating debug namespace/openshift-debug-node-xxxxx ...
    [...]
    If you don't see a command prompt, try pressing enter.
    sh-4.4# chroot /host bash

  • Check that after a successful podman login withing a OCP 4 node, the pull fails:

    [root@node_name /]# podman login
Login Succeeded!
[root@node_name /]# podman pull registry.redhat.io/openshift4/ose-cluster-logging-operator
[...]
Failed to pull image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:ed07723ee730fee6e4758410c65271dac0b035f89c14595ed4a92db4fb83f037": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials.
[...]

  • Try the podman pull with --authfile /var/lib/kubelet/config.json

    [root@node_name /]# podman pull --authfile /var/lib/kubelet/config.json registry.redhat.io/openshift4/ose-cluster-logging-operator
Trying to pull registry.redhat.io/openshift4/ose-cluster-logging-operator...
Getting image source signatures
Copying blob xxxxxxxxxxxx skipped: already exists
[...]
Copying blob xxxxxxxxxxxx done
[...]
Writing manifest to image destination
Storing signatures
[...]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.