Update from 16.2 to 17.1 failed on migrate existing introspection data in the undercloud

Solution Verified - Updated

Environment

  • Red Hat OpenStack Platform 17.1

Issue

  • TripleO failed on migrate existing introspection data step during second try of undercloud upgrade for 16.2->17.1 FFU because of lost connection to mysql.
  • After failed undercloud upgrade stopped mysql, next try of undercloud upgrade failed due for some reason.

Resolution

  • Restart tripleo_mysql.service since it was stopped during the last upgrade attempt & upgrade got failed.

Root Cause

  • mysql is stopped during last run and upgrade got failed.
   2023-07-19 07:24:57.191964 | 525400cf-381f-98fd-f6ac-00000000095c |       TASK | Stop the current mysql container
  • Re-run of undercloud upgrade failed with following message.
   2023-07-19 08:15:18.869778 | 525400cf-381f-5864-38c7-00000000073e |      FATAL | migrate existing introspection data | undercloud | error={"changed": true, "cmd": "podman exec -u root ironic_inspector ironic-inspector-migrate-data --from swift --to database --config-file /etc/ironic-inspector/inspector.conf\n", "delta": "0:01:42.467282", "end": "2023-07-19 08:15:18.830981", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2023-07-19 08:13:36.363699", "stderr": "b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\"", "stderr_lines":["b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\""], "stdout": "", "stdout_lines": []}

Diagnostic Steps

  • Re-run of undercloud upgrade failed with following message.
   2023-07-19 08:15:18.869778 | 525400cf-381f-5864-38c7-00000000073e |      FATAL | migrate existing introspection data | undercloud | error={"changed": true, "cmd": "podman exec -u root ironic_inspector ironic-inspector-migrate-data --from swift --to database --config-file /etc/ironic-inspector/inspector.conf\n", "delta": "0:01:42.467282", "end": "2023-07-19 08:15:18.830981", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2023-07-19 08:13:36.363699", "stderr": "b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\"", "stderr_lines": ["b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\""], "stdout": "", "stdout_lines": []}
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.