Standalone NooBaa Multicloud Object Gateway (MCG) Performance Tuning Guide (without StorageCluster CR) - Quay

Solution Verified - Updated

Environment

Red Hat Quay (RHQ) v3.x
Red Hat OpenShift Container Platform (RCOCP) v4.x
Red Hat OpenShift Data Foundation (RHODF) v4.x

Issue

  • noobaa-endpoint OOMKilled.
  • noobaa-core-0 and noobaa-db-pg-x consumed more CPU than the limits set.
  • NooBaa PV-Pool backingstore pods OOMKilled.

IMPORTANT:

This is a legacy deployment that was configured using section 3.2.2.1.1. Create A Standalone Object Gateway of the Quay documentation. This configuration is no longer supported, and will fail the ODF v4.19 upgrade (won't transition to High Availability (HA) NooBaa DB). See resolution to transition from this configuration and properly tune NooBaa.

Resolution

  1. Migrate to a storagecluster managed MCG using the Migrating from a Quay Standalone Multicloud Object Gateway (MCG) to an ODF StorageCluster ODF MCG - NooBaa solution.

  2. Once fully transitioned, and the storagecluster.yaml is in phase Ready, follow the Performance tuning guide for Multicloud Object Gateway solution to tune MCG (NooBaa) properly.

Diagnostic Steps

Below means noobaa-core and noobaa-db need CPU resources Increase:

117839 d6f   02/01 19:24:22 181694 retryable error [44038], retry = 0, sleep for 1 second(s)
117839 b12   02/01 19:24:23 ###### CURL error, CURLcode= 28, Timeout was reached
<omitted>CCloudFSwithPaging::ListDirectoryInternalPageWise(712) - GetFileList failed for[quay-enterprise-<omitted>2/01 19:06:00 181694 504 Gateway Timeout. <--------

Below indicates noobaa-endpoint HPA needs to be scaled (more pods) to prevent OOMKills:

apiVersion: v1
kind: Pod
<ommited-for-space>
  labels:
    app: noobaa
    noobaa-s3: noobaa
<ommited-for-space>
    name: noobaa-endpoint-6c854fffd6
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-02-07T20:33:54Z"
    status: "True"
    type: Initialized
<omitted-for-space>
  containerStatuses:
  - containerID: cri-o://7ec25d9075ccd3c4a17db08ca82d6d1e8847b340c723ffdbb2cf3f776d593729
    image: registry.redhat.io/ocs4/mcg-core-rhel8@sha256:3083c2de68baa0fabf4a0e92742762af756e22206af4647331e0889ac1d6c90c
    imageID: registry.redhat.io/ocs4/mcg-core-rhel8@sha256:3083c2de68baa0fabf4a0e92742762af756e22206af4647331e0889ac1d6c90c
    lastState: {}
    name: endpoint
<omitted-for-space>
        exitCode: 137 <----------------------------- OOMKill
        reason: Error
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.