High I/O wait time observed on Red Hat Enterprise Linux

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 10
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7

Issue

  • The system is experiencing poor I/O performance.
    • High I/O wait percentages are observed in performance monitoring tools (e.g., %wa in top or avgqu-sz in iostat).
    • Basic operations, such as copying a large file (e.g., 4GB) to local or virtual storage, take significantly longer than expected.

Resolution

  1. Identify the offending process: Use iotop to identify which processes are consuming the most I/O bandwidth.

    # iotop -o

  2. Analyze disk latency: Use iostat to check for high disk utilization and wait times.

    # iostat -xz 1 10

    Note: High await values (e.g., >10-20ms consistently) indicate the storage subsystem is struggling to keep up with requests.

  3. Check for kernel errors: Review system logs for I/O errors or resets.

    # dmesg | grep -iE "io|error|reset|scsi"
# journalctl -xe

  4. Virtualization Context: If the system is a virtual machine (e.g., VMware), high I/O wait in the guest often indicates congestion at the hypervisor or physical storage level.

    • Verify the health and performance of the underlying host.
    • Check for storage snapshots or migrations (vMotion) occurring during the performance drop.

  5. Vendor Engagement: If no local issues are found in RHEL but disk latency remains high, contact the storage or virtualization vendor (e.g., VMware) to investigate the backend storage performance.

Root Cause

  • High I/O wait occurs when the CPU is idle but there are outstanding disk I/O requests. This is typically caused by:
    • Saturated physical storage bandwidth.
    • Latency in the storage area network (SAN) or network-attached storage (NAS).
    • Resource contention at the hypervisor level in virtualized environments.
    • Hardware failure or failing disk sectors.

Diagnostic Steps

  • Install the sysstat package to enable performance monitoring tools:

    # yum install sysstat

  • Capture performance data while the issue is occurring:

    1. Run these commands in separate terminal sessions to capture a snapshot of system activity:

      # mkdir /tmp/capture
# cd /tmp/capture
# top -n 5 -b > top.out
# vmstat 1 50 > vm.out
# iostat -x 2 10 > io.out
# sar -A 1 50 > sar.out
# cd ~

    2. Archive the gathered logs:

      # tar -cvzf /tmp/perf_data_$(date +%F).tar.gz /tmp/capture

    3. Attach the resulting /tmp/perf_data_*.tar.gz file to your Red Hat support case.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.