Red Hat Satellite 6: hammer ping fails with Katello::Errors::CandlepinNotRunning

Environment

• Red Hat Satellite 6

Issue

• hammer ping fails for candlepin, candlepin_auth, and candlepin_events with the following errors:

  candlepin:        
      Status:          FAIL
      Server Response: Message: 404 Not Found
  candlepin_auth:   
      Status:          FAIL
      Server Response: Message: Katello::Errors::CandlepinNotRunning
  candlepin_events: 
      Status:          FAIL
      message:         Not running
      Server Response: Duration: 1ms
  

Resolution

• Begin by reviewing the Diagnostic Steps. If the issue matches the described scenario, proceed with the resolution steps outlined below.

• Stop the fapolicyd service and restart the tomcat service:

  # systemctl stop fapolicyd
  # systemctl restart tomcat
  

• Allow a few minutes for the services to stabilize, then verify the status using hammer ping. If the command returns no errors, start the fapolicyd service again.

  # systemctl start fapolicyd
  

Reach out to Red Hat Technical Support in case this does not resolve the issue.

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

• A rule in fapolicyd is configured to trust the Candlepin file. When fapolicyd blocks access to this file, it returns an "Operation not permitted" error, which leads to the Candlepin failure. This occurs due to presence of xml file in the custom trust files which were present at /etc/fapolicyd/fapolicyd.trust.

Diagnostic Steps

• Check the journalctl output for errors related to XML configuration file access and look for entries similar to the following:

  # journalctl |grep -i error
   ...
  Apr 06 09:50:22 satellite.example.com server[1441]: 09:50:22,848 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
  Apr 06 09:50:22 satellite.example.com server[1441]: 09:50:22,850 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/logback.xml]
  Apr 06 09:50:22 satellite.example.com server[1441]: 09:50:22,889 |-ERROR in ch.qos.logback.core.joran.event.SaxEventRecorder@6b250a96 - I/O error occurred while parsing xml file java.io.FileNotFoundException: /var/lib/tomcat/webapps/candlepin/WEB-INF/classes/logback.xml (Operation not permitted)
  

• The key indicator is the "Operation not permitted" error, which confirms that the application is unable to access or read the required XML configuration file due to a permission restriction.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.