Troubleshooting must-gather data collection for Technical Supportability Review with AI in OpenShift Container Platform (OCP)

Solution Verified - Updated 21 May 2026

Environment

Failure to generate or upload data for Technical Supportability Review with AI.
Partial or inaccurate results in the generated AI report.
Errors during the execution of the custom oc adm must-gather script.
Data collection size exceeds the 25 GB upload limit.

Issue

Failure to generate or upload data for Technical Supportability Review with AI.
Partial or inaccurate results in the generated AI report.
Errors during the execution of the custom oc adm must-gather script.
Data collection size exceeds the 25 GB upload limit.

Resolution

1. Execute the Data Collection Script

Use the following script to collect the required must-gather data. This script automatically identifies installed operators and includes their specific diagnostic data:

must_gather_cmd="oc adm must-gather \
 --image-stream=openshift/must-gather \
 --image=quay.io/pg.next/pg-must-gather"; \
csvJson=$(oc get clusterserviceversions.operators.coreos.com -A -o json); \
while IFS= read -r image ; \
do \
 must_gather_cmd="$must_gather_cmd \
 $image"; \
done < <(echo "$csvJson" | jq -r '.items[] | select(.status.phase == "Succeeded") | select(.spec.relatedImages != null) | .spec.relatedImages | map(select(.image |test("must-?gather"; "i"))) | group_by(.image | split("@")[0] | split("/")[0:3] | join("/")) | map(last) | .[] | "--image=" + .image ' | sort -u); \
while IFS= read -r image ; \
do \
 must_gather_cmd="$must_gather_cmd \
 $image"; \
done < <(echo "$csvJson" | jq -r '.items[] | select(.status.phase == "Succeeded") | select(.metadata.name | contains("cluster-logging")) | select(.spec.install.spec.deployments[]?.name == "cluster-logging-operator") | .spec.install.spec.deployments[].spec.template.spec.containers[].image | "--image=" + .' | sort -u); \
echo -e "Generated command :\n $must_gather_cmd \
 | tee -a must-gather-console.log"

2. Verify Execution Logs

The primary file for troubleshooting is must-gather-console.log. It records which must-gather images were executed and their results. For instance:

    Generated command :
	 oc adm must-gather \
	  --image-stream=openshift/must-gather \
	  --image=quay.io/pg.next/pg-must-gather \
	  --image=registry.redhat.io/odf4/odf-must-gather-rhel9@sha256:16f27c8673487f2a3671319fbf01357726e6e416aa8b26300cccd88302400289 \
	  --image=registry.redhat.io/openshift4/ose-local-storage-mustgather-rhel9@sha256:c9fb09d588c0e3ff46b94581bd65048d596b8d5c2a84555a3486ff645ba11c6a \
	  --image=registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:ca448c48fee983cb3df77d959b1e0cb4b938fa22621ea30445a29d73f18dde73 \
	 | tee -a must-gather-console.log

In this example, this script captures:

A standard OCP must-gather.
A PG must-gather.
An ODF must-gather.
An LSO must-gather.
A Logging must-gather.

Each collection is assigned a unique ID (e.g., must-gather-m9kmd) which can be used to filter specific logs:

must-gather-console.log:[must-gather-m9kmd] POD 2026-03-17T17:52:33.814169648Z must-gather is using image: registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:ca448c48fee983cb3df77d959b1e0cb4b938fa22621ea30445a29d73f18dde73

3. Requirements and Constraints

Authentication: The user executing the script must have cluster-admin privileges.
Disconnected Environments: For air-gapped clusters, mirror the required images to the internal OCP registry before execution. Refer to KCS 4647561 for mirroring steps.
Timing: Wait at least 24 hours after a cluster upgrade or major maintenance activity before running the collection to ensure diagnostic data is stable and accurate.
Data Privacy: Using must-gather-clean for obfuscation may result in report inaccuracies, particularly regarding cluster IP-based rules. Refer to KCS 6345311.
Operator Status: The script only collects data from operators in the Succeeded phase. Data from operators in Pending or Replacing phases will not be captured.
Data Type: The tool only processes must-gather data. Screenshots, custom text files, or manual command outputs will not generate a report.

4. Common Execution Errors

The following errors may occur during script execution based on environment constraints or configuration issues:

Error: Fatal glibc error: CPU does not support x86-64-v2
- The pg-must-gather image is based on RHEL 9 and requires a CPU supporting the x86-64-v2 instruction set.
- Solution: Use the RHEL 8-based image by appending :rhel8 to the image flag: --image=quay.io/pg.next/pg-must-gather:rhel8.
Error: FIPS mode is enabled, but the required OpenSSL backend is unavailable
- This occurs when the oc binary is executed on a RHEL 8 or 9 system with FIPS enabled, as the binary expects an older OpenSSL version.
- Solution: Use the oc binary from version 4.16 or later to connect to the cluster. Refer to the KCS article 7046917 - OpenSSL backend is unavailable error in oc commands in OpenShift 4 with FIPS enabled and the oc binary version policy for additional details.
Error: ImagePullBackOff: Back-off pulling image "quay.io/pg.next/pg-must-gather"
- This indicates the cluster cannot reach the external registry.
- Solution: Verify internet connectivity or, in disconnected environments, mirror the image to the local repository.
Error: [must-gather-<id>] POD <timestamp> Error from server (Forbidden): namespaces "openshift-cluster-version" is forbidden: User "<user-name>" cannot get resource "namespaces" in API group "" in the namespace "openshift-cluster-version"
- The user executing the command lacks cluster-admin privileges.
- Solution: Verify permissions by running: oc get clusterrolebindings -o json | jq '.items[] | select (.subjects[].name == "'$(oc whoami)'") | .metadata.name'.
Error: error extracting tar at destination directory: unexpected EOF
- This is typically caused by slow execution or network timeouts.
- Solution: Re-run the command with an increased timeout, for example: --timeout=30m.
Error: fatal error: runtime: out of memory
- The local oc process has exhausted available system memory.
- Solution: Follow standard procedures for managing high CPU/memory consumption during must-gather operations. For additional information, refer to the KCS article 7128538 - High CPU consumption when running oc adm must-gather.

5. Must-Gather Exit Codes

If the oc adm must-gather command fails, check the exit status to identify the root cause:

Exit Status 5 (EIO): Indicates an I/O error. This is a generic hardware or filesystem issue preventing a read/write operation.
Exit Status 10 (ECHILD): Indicates no child processes. This occurs if the must-gather container is killed, often due to being out of memory (OOM) or a timeout.
Exit Status 23 (ENFILE): Indicates a file table overflow. The system-wide limit for open files has been reached.
Exit Code 137: Indicates the container was terminated because it exceeded its memory limit (OOM Kill).
- Solution: Start a new collection using a longer timeout or limit the data volume using the --since flag.

For additional information regarding this topic, take a look at the Red Hat Blog post This content is not included.Bash command line exit codes demystified.

6. Managing Large Data Collections

The maximum upload size is 25 GB. If the collection exceeds this limit:

Time Window: Use the --since option (available in OCP 4.16+) to limit log duration. See KCS 7030209.
MetalLB: If MetalLB is installed, refer to KCS 7068652 to address oversized collections.
ODF: For large OpenShift Data Foundation collections, follow KCS 7137684.
Maintenance: Delete completed or failed pods and jobs, specifically in openshift-* namespaces, to reduce the overall metadata size.

7. What is the Must-Gather Data Quality Gate, and why was my report refused?

To prevent misleading analysis, the validation framework includes an automated data quality gate that checks the integrity and completeness of your uploaded archive before generating a report.

If your must-gather archive is incomplete, corrupted, or contains no actionable items, the system will refuse report generation and display one of the following diagnostic messages:

No Actionable Issues Found
- Message: "No actionable issues were found in this must-gather archive." OR "This assessment is exclusively related to our predefined checks. The assessment was successful and no issues were found."
- Reason: The archive is technically valid and was processed successfully, but the cluster is completely healthy and contains no FAIL or WARNING statuses to generate recommendations for.
Collection Failure Detected
- Message: "This must-gather archive may be corrupted due to errors during data collection. Please re-collect the must-gather and try again."
- Reason: The tool detected major failures (such as the creation of inspect.local directories) during the initial oc adm must-gather collection process, meaning the data cannot be trusted for a reliable review.
Mandatory Must-Gather Missing
- Message: "This must-gather archive is missing required data and cannot be analyzed. Please re-collect the must-gather with the required must-gather images and try again."
- Reason: Core required diagnostic images (such as openshift/must-gather or pg-next-pg-must-gather) are missing from the bundle, depriving the engine of the foundation data required for analysis.
Too Many Rules Skipped
- Message: "This must-gather archive appears to be incomplete. Please verify the archive and try again."
- Reason: Too much data is missing from the archive, causing the majority of the internal supportability rules to skip. If fewer than 245 rules execute successfully, the archive is rejected as incomplete.

In cases where the archive is corrupted or missing mandatory files, you are advised to re-collect the diagnostic data by following the step-by-step instructions detailed in Section 1: Execute the Data Collection Script of this document.

NOTE: For further assistance regarding data collection for Technical Supportability Review with AI, please contact your local Red Hat Account team or Ask Red Hat to get answers from our library of support resources

Category

Troubleshoot

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.