FFU is failing on Ansible task special treatment for OpenvSwitch

Environment

• Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1

Issue

• openstack overcloud upgrade run fails during execution of task "Special treatment for OpenvSwitch"

Resolution

1. Ssh into the overcloud node that failed the upgrade and

2. Detect which openvswitch packages are installed on the host that failed the ugprade:

    [tripleo-admin@overcloud-novacompute-0 ~]$ rpm -qa | grep ^openvswitch
    openvswitch2.15-test-2.15.0-142.el8fdp.noarch
    openvswitch2.15-2.15.0-142.el8fdp.x86_64
    [tripleo-admin@overcloud-novacompute-0 ~]$
   

3. In this example, an extra "test" package is installed, but the issue would also happen with the "devel" version of openvswitch2.15

4. Remove the extra, non-core openvswitch package package:

    [tripleo-admin@overcloud-novacompute-0 ~]$ sudo dnf -y remove openvswitch2.15-test-2.15.0-142.el8fdp.noarch
   

Root Cause

• The tripleo-ansible task that takes care of the special treatment upgrade of openvswitch is not correctly detecting extra (optional) packages that might be installed
• Extra openvswitch packages are installed on some overcloud nodes for example when troubleshooting network issues, and they can be removed afterwards
• The following BugZilla has been created to address this error: This content is not included.This content is not included.https://bugzilla.redhat.com/show_bug.cgi?id=2320138

Diagnostic Steps

• In the Ansible logs, we see error such as this:

    2024-02-24 17:25:19.201414 | 001c2891-506d-6fc4-0d5e-00000001ff44 |      FATAL | Special treatment for OpenvSwitch | cpt-g-001 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", l
    ine 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_mo
    dule_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_tripleo_ovs_upgrade_payload_mbqbl7xs/ansible_tripleo_ovs_upgrade_payload.zip/ansible/modules/trip
    leo_ovs_upgrade.py\", line 337, in <module>\n  File \"/tmp/ansible_tripleo_ovs_upgrade_payload_mbqbl7xs/ansible_tripleo_ovs_upgrade_payload.zip/ansible/modules/tripleo_ovs_upgrade.py\", line 326, in main\n  File \"/tmp/ansible_tripleo_ovs_upgrade_payload_mbqbl7xs/ansible_
    tripleo_ovs_upgrade_payload.zip/ansible/modules/tripleo_ovs_upgrade.py\", line 202, in layer_product_upgrade\nIndexError: list index out of range\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.