Obfuscating must-gather using must-gather-clean

Updated 5 Dec 2024

Introduction

Privacy and security-aware customers of OpenShift want to ensure they don’t leak confidential information from their must-gather when reporting bugs. On the other hand, it is vital for Red Hat to get accurate enough information that still allows engineers to debug and solve reported bugs. It has been very difficult for customers to properly remove confidential information from must-gather, as they are usually big in size and number of files and sometimes confidential information is hidden in unexpected places.

We developed a community-supported command line tool called must-gather-clean to help those customers obfuscate and omit files that contain confidential information. Note: this tool is not shipped and not supported by Red Hat.

Key Features

must-gather-clean is designed to be fast and extensible, yet simple to use. It has the following key features:

Obfuscation for many common types of confidential information, for example IPs, MACs, DNS
Granular control over what files should be shared
Replace confidential information consistently to preserve debuggability
Ability to parse and understand Kubernetes and OpenShift resources
Concise and feature rich tool configuration

Installation

Please follow the Content from github.com is not included.installation instructions on the GitHub project. We support Linux, Mac and Windows with AMD64+ARM64.

To test the installed tool, the following command will print the version:

$ must-gather-clean version

Usage

must-gather-clean should be pointed to the root folder of an already generated must-gather. Provided a must-gather generated by the command:

$ oc adm must-gather --dest-dir=must-gather-output

Then cleaning can be done by running:

$ must-gather-clean -c config.yaml -i must-gather-output -o must-gather-output-cleaned

The cleaned must-gather can then be found in the must-gather-output-cleaned folder, indicated by the -o argument.

Configuration

To clean, you need to provide a configuration through the -c argument. A very basic default configuration for OpenShift can be found at Content from raw.githubusercontent.com is not included.examples/openshift_default.yaml:

config:
  obfuscate:
    - type: IP
      replacementType: Consistent
      target: All
    - type: MAC
      replacementType: Consistent
      target: All
    - type: Domain
      replacementType: Consistent
      target: All
      domainNames:
        - "rhcloud.com"
        - "dev.rhcloud.com"
  omit:
    - type: Kubernetes
      kubernetesResource:
        kind: "Secret"
    - type: Kubernetes
      kubernetesResource:
        kind: "ConfigMap"
    - type: Kubernetes
      kubernetesResource:
        kind: CertificateSigningRequest
        apiVersion: certificates.k8s.io/v1
    - type: Kubernetes
      kubernetesResource:
        kind: CertificateSigningRequestList
        apiVersion: certificates.k8s.io/v1

Which is obfuscating IPs, MACs and domain names, the latter can be customized to the deployed DNS names under domainNames. Additionally, this configuration omits files and resources that contain confidential information (for example Secrets and ConfigMaps).

The whole configuration is very flexible and has several customization options explained in the Content from github.com is not included.project's README.

Pipe Support

The tool also supports piping content on your shell:

$ echo "some ip 192.168.2.1" | must-gather-clean 
some ip x-ipv4-0000000001-x

By default, this will obfuscate IPs and MAC addresses. You can still pass configuration options which further define what needs to be obfuscated. Omissions are not supported when supplying content by pipes.

Reporting

After cleaning, you'll find a report of all the found information and their replacements in a report.yaml in the current working directory. This is helpful to understand whether all your confidential information was caught and how they were replaced.

Please do not share this report with Red Hat, as this allows anybody to reverse the obfuscation process.