Configured tomcat6 as a Highly Available clustered service but rgmanager can not start it in RHEL 6

Environment

• Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
• resource-agents prior to release 3.9.2-21.el6

Issue

• Would like to configure Tomcat6 as a Highly Available clustered service in Red Hat Cluster Suite, but rgmanager will not start it.
• When starting a service with a tomcat resource it fails and the following errors are in the logs:

Jan 13 20:10:57 rhel6-node1 rgmanager[5733]: Starting disabled service service:Tomcat
Jan 13 20:10:57 rhel6-node1 rgmanager[17767]: [ip] Adding IPv4 address 10.13.211.133/24 to eth0
Jan 13 20:10:59 rhel6-node1 avahi-daemon[1982]: Registering new address record for 10.13.211.133 on eth0.IPv4.
Jan 13 20:11:01 rhel6-node1 rgmanager[17917]: [tomcat-6] Starting Service tomcat-6:tomcat
Jan 13 20:11:01 rhel6-node1 rgmanager[5733]: Service service:Tomcat started
Jan 13 20:12:07 rhel6-node1 rgmanager[18694]: [tomcat-6] Checking Existence Of File /var/run/cluster/tomcat-6/tomcat-6:tomcat.pid [tomcat-6:tomcat] > Failed
Jan 13 20:12:07 rhel6-node1 rgmanager[18716]: [tomcat-6] Monitoring Service tomcat-6:tomcat > Service Is Not Running
Jan 13 20:12:07 rhel6-node1 rgmanager[5733]: status on tomcat-6 "tomcat" returned 1 (generic error)
Jan 13 20:12:07 rhel6-node1 rgmanager[5733]: Stopping service service:Tomcat
Jan 13 20:12:07 rhel6-node1 rgmanager[18812]: [tomcat-6] Stopping Service tomcat-6:tomcat
Jan 13 20:12:07 rhel6-node1 rgmanager[18834]: [tomcat-6] Checking Existence Of File /var/run/cluster/tomcat-6/tomcat-6:tomcat.pid [tomcat-6:tomcat] > Failed - File Doesn't 
Jan 13 20:12:07 rhel6-node1 rgmanager[18856]: [tomcat-6] Stopping Service tomcat-6:tomcat > Succeed
Jan 13 20:12:07 rhel6-node1 rgmanager[18909]: [ip] Removing IPv4 address 10.13.211.133/24 from eth0
Jan 13 20:12:17 rhel6-node1 rgmanager[5733]: Service service:Tomcat is recovering
Jan 13 20:12:18 rhel6-node1 rgmanager[5733]: Recovering failed service service:Tomcat
Jan 13 20:12:18 rhel6-node1 rgmanager[19003]: [ip] Adding IPv4 address 10.13.211.133/24 to eth0
Jan 13 20:12:21 rhel6-node1 rgmanager[19153]: [tomcat-6] Starting Service tomcat-6:tomcat
Jan 13 20:12:22 rhel6-node1 rgmanager[5733]: Service service:Tomcat started

Resolution

Update to resource-agents-3.9.2-21.el6 or later

Root Cause

The reason that the resource tomcat-6 failed was because the binary /usr/sbin/tomcat6 failed to start correctly because the variables that were needed for it to start were not sourced in. In order for /usr/sbin/tomcat6 to start it requires that all the variables in the file $TOMCAT_CFG are sourced in which they were not.

Diagnostic Steps

• Fully configured tomcat-6 in my RHEL 6 cluster as a HA service
• When rgmanager tries to start the service, it fails
• When using rgtest to test the service, the following error message can be seen in the debug output:

  line 30: /logs/catalina.out: No such file or directory

• Line 30 of the file /usr/sbin/tomcat6 is the following:

  org.apache.catalina.startup.Bootstrap start >> ${CATALINA_BASE}/logs/catalina.out

• When echoing $CATALINA_BASE on the termial, nothing comes back which means this variable is not defined.
• In the configuration file for tomcat6 /etc/tomcat6/tomcat6.conf, it is however defined.
• Going back to the /usr/sbin/tomcat6 file, the following is commented out
  (note I added the line numbers, they are not in the actual file):

1	# Get the tomcat config (use this for environment specific settings)
2	#if [ -z "${TOMCAT_CFG}" ]; then
3	#  TOMCAT_CFG="/etc/tomcat6/tomcat6.conf"
4	#fi

5	#if [ -r "$TOMCAT_CFG" ]; then
6	#  . $TOMCAT_CFG
7	#fi

• Uncomment lines 2-7, then save the file and start up the service.
• That being said, you will have to have the tomcat6 service started on every node in the cluster that
  you want to provide the service, and uncomment those lines in /usr/sbin/tomcat6.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.